Linking commands, pipes, and input/output

This section is probably one of the most important in the book because it describes a fundamental and powerful feature on Linux and Unix: the ability to use pipes and redirect input or output. By themselves, pipes are a fairly trivial feature - commands and scripts can redirect their output to files or commands. So what? This could be considered a massive understatement in the Bash scripting world, because pipes and redirection allow you to enhance commands with the functionality of other commands or features. 

Let's look into this with an example using commands called tail and grep. In this example, the user, Bob, wants to look at his logs in real time (live), but he only wants to find the entries related to the wireless interface. The name of Bob's wireless device can be found using the iwconfig command:

$ iwconfig
wlp3s0 IEEE 802.11abgn ESSID:"127.0.0.1-2.4ghz" 
          Mode:Managed Frequency:2.412 GHz Access Point: 18:D6:C7:FA:26:B1 
          Bit Rate=144.4 Mb/s Tx-Power=22 dBm 
          Retry short limit:7 RTS thr:off Fragment thr:off
          Power Management:on
          Link Quality=58/70 Signal level=-52 dBm 
          Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
          Tx excessive retries:0 Invalid misc:90 Missed beacon:0

The iwconfig command is deprecated now. The following commands also will give you wireless interface information:

$ iw dev                # This will give list of wireless interfaces
$ iw dev wlp3s0 link    # This will give detailed information about particular wireless interface

Now that Bob knows his wireless card's identifying name (wlp3s0), Bob can search his system's logs. It is usually found within /var/log/messages. Using the tail command and the -F flag, which allows continuously outputting the logs to the console, Bob can now see all the logs for his system. Unfortunately, he would like to filter the logs using grep, such that only logs with the keyword wlp3s0 are visible.

Bob is faced with a choice: does he search the file continuously, or can he combine tail and grep together to get the results he desires? The answer is yes—using pipes!

$ tail -F /var/log/messages | grep wlp3s0
Nov 10 11:57:13 moon kernel: wlp3s0: authenticate with 18:d6:c7:fa:26:b1
Nov 10 11:57:13 moon kernel: wlp3s0: send auth to 18:d6:c7:fa:26:b1 (try 1/3)
Nov 10 11:57:13 moon kernel: wlp3s0: send auth to 18:d6:c7:fa:26:b1 (try 2/3)
...

As new logs come in, Bob can now monitor them in real time and can stop the console output using Ctrl+C.

The usage and flexibility of pipes should be relatively straightforward, but what about directing the input and output of commands? This requires the introduction of three commands to get information from one place to another:

• stdin (standard in)
• stdout (standard out)
• stderr (standard error)

If we are thinking about a single program, stdin is anything that can be provided to it, usually either as a parameter or a user input, using read for example. Stdout and stderr are two streams where output can be sent. Usually, output for both is sent to the console for display, but what if you only want the errors within the stderr stream to go to a file?

$ ls /filethatdoesntexist.txt 2> err.txt
$ ls ~/ > stdout.txt
$ ls ~/ > everything.txt 2>&1 # Gets stderr and stdout
$ ls ~/ >> everything.txt 2>&1 # Gets stderr and stdout
$ cat err.txt
ls: cannot access '/filethatdoesntexist.txt': No such file or directory
$ cat stdout.txt
.... # A whole bunch of files in your home directory

When we cat err.txt, we can see the error output from the stderr stream. This is useful when you only want to record errors and not everything being output to the console. The key feature to observe from the snippet is the usage of >, 2>, and 2>&1. With the arrows we can redirect the output to any file or even to other programs!

Now that we know the basics of one of the most powerful features available in Bash, let's try an example—redirection and pipes bonzanza.