Why bother using the shell?

12 Sep 2023
By Sung Kim

Learning to fully utilize the shell will make you a better tech professional.

Do not do manually what you can command the computer to do.

$ Shell = Code on demand

Shells¹ allow you to write/run code on the fly. Shell languages are optimized for typing convenience, suited for coding one-off actions.

Shell languages don’t require you to:

Write excessive boilerplate like Java²
Make sure parentheses match up in nested function calls
Balance curly braces
import/require/include libraries
Preface function calls with their namespaces³

Since shell languages avoid these incoveniences, they are better-suited for short, one-off commands than traditional programming languages. Having a tight syntax like this makes writing individual commands more convenient but this is only the tip of the iceberg.

$ The Pipe

The shell comes alive when you chain sequences of commands together. You may have previously used the pipe operator | without recognizing its expressive power. On the surface, it simply streams the output of the previous command into the input of the next. This semantic is analogous to chaining functions together in other programming languages.

# Python
    "two  ".strip().upper().replace('W', 'O')
#   "two  "  "two"   "TWO"    "TOO"

# Bash
    echo "two  " | xargs | tr 'a-z' 'A-Z' | sed 's/W/O/'
#        "two  "   "two"   "TWO"            "TOO"

Now, this may not look impressive until you consider the fact that the bash commands you see here are not methods. Python only allows you to call string objects to call string methods (there are ~47 string methods based on my count). In shells, we have no such restriction. In the bash code above, we’re chaining entirely unrelated CLI tools together. Just for analyzing/manipulating text alone, there is a plethora of CLIs and they form a combinatorial explosion of sequences.

To emphasize the difference, let me give another example. You couldn’t write Python like this:

# INVALID PYTHON CODE
with open("file.txt") as f:
    print(f.read() # Read file.txt
           .filter(lambda x: x not in string.punctuation) # remove punctuation
           .split() # Put words in a list
           .set() # Keep only unique words
           .length() # Count unique words
    )

Instead, here is one way to do this with valid Python:

from string import punctuation

with open("file.txt") as f:
    words = f.read()
    wordsWithoutPunctuation = ''.join(filter(lambda x: x not in punctuation, words))
    wordList = wordsWithoutPunctuation.split()
    uniqueWords = set(wordLists)
    print(len(uniqueWords))

This code above is not something you would want to write if you had a one-off need to count how many unique words there were in a file. Here is what it would look like in bash:

cat file.txt | tr -d '[:punct:]' | xargs | sort | uniq | wc -w

Here is the command with comments:

cat file.txt \ # output text of file.txt
| tr -d '[:punct:]' \ # remove punctuations
| xargs \ # Replace newline and whitespace sequences with a single space
| sort \ # Sort words alphabetically
| uniq \ # Remove repeated words (requires input to be sorted)
| wc -w  # Output number of words

$ Examples of using the shell

This is just a taste of what the shell can do with text data. It can do so much more with all different kinds of files:

CSV files
Log files
Assembly files
Code files
Configuration files
Symbol tables
Pretty much any file…

Even outside of the context of reading files, here are random examples of things you can do from the shell:

Kill all running Python programs

ps -eo pid,cmd                          | awk '$2 ~ /^python$/ { print $1; }' | xargs kill
# List all running processes w/ PID     Output PIDs of python processes         kill python processes

Find all test files that use a certain feature flag

find . -name *test* -type f         | xargs grep -l FEATURE_FLAG_A
# List all files w/ "test" in name   Among them, list all that mention the given feature flag

Find all the configuration files that are different between two mostly identical directories

diff -rq aws1/ aws2/                                    | grep '\.conf'
# Find all files with differences between the two dirs   List the results related to .conf files

Find all files that mention both foo and bar

grep -rl foo                     | xargs grep -l bar
# Find all files mentioning foo   From them find all mentioning bar

Print only the second column of the output of a Python script
```
python script.py | awk '{ print $2; }'
```

The fact that shell commands can integrate seamlessly with other scripts and programs of other languages creates another world of use cases that go beyond just your typical shell commands.

That’s not all! I’ve only talked about the use of the pipe operator, but shells have much more to offer:

functions/aliases
subshells
file operations
streams
customizations
- Personalizing your shell prompt
- Shell plugins
- Using a fuzzy-finder for searching previous commands
- Vi-mode
- Git integrations
- And more!

Learning to make full use of the shell is a timeless skill that will enhance your technical capabilities for your whole life.

Appendix: The only commands you need to be familiar with

To be clear, I am NOT saying you need to go and memorize a hundred CLIs to consider yourself “proficient” with the shell. You actually only need to familiarize youself with a handful:

File management
- ls: list directory contents
- cd: change the current directory
- cp: copy files and directories
- mv: move (rename) files
- mkdir: make directories
- rm: remove files or directories
Text manipulation/processing
- echo: display a line of text
- cat: concatenate files and print on the standard output
- grep or equivalent tools: print lines that match patterns
- awk or sed: stream editor for filtering and transforming text
- Any one of nano, vi, vim, nvim, emacs, …: A tool to edit text files
Command utilities
- xargs: build and execute command lines from standard input
Documentation
- man: an interface to the system reference manuals

People who have used the terminal for more than a few times probably are familiar with quite a few of these commands. It just takes a little effort to learn the rest. Again, you really don’t need to have a lot of CLIs memorized. What’s more important is being able to find online the CLIs you need and learning how to use them as you go. Learning to read man pages is essential.

Examples: bash, zsh, fish, etc ↩
Ex: public static void main(String[] args) ↩
Ex: math.log(10) ↩