Pipes, Redirection and Chaining
Please read about Standard-output and Standard-input first.
Even though most people consider the command-line a thing of the past, there are some really interesting things you can do with the terminal that enables it to be more powerful than the gui interface.
Two advantages are the use of pipes, chaining, and redirection. With pipes, chaining, and redirection, you can “chain” multiple programs and functions to become extremely powerful commands. Most programs on the command-line accept different modes of operation. Many can read and write to files for data, and most can accept standard input or output. This means that you can direct the output of one program as input to another program. You can then take the output of the second program and redirect it as input to yet another program, or redirect the output to a file.
In fact, most programs have two modes of output: standard output (stdout for short) and standard error (stderr). These are both different streams of output that are distinct from each other, which can allow you to redirect the stderr of a program to the same “stream” as the standard input (stdin), or redirect them to different places.
Redirection
For example, to redirect the output of the ps program to a file, use:
- $ ps ax > output.txt
Instead of ps ax displaying on the screen, all that information has been sent to the output.txt file.
But lets say you already have an output.txt file and you don’t want to over-write that file, you’d simply like to append new data to that file. In that case we would
- $ ps ax >> output.txt
Now the output of ps ax is added to the bottom of the already existing output.txt.
You can use >> to merge two files together as well in conjunction with the cat command for instance
- $ cat file1.txt file2.txt >> outputfile.txt
The contents of file1.txt and file2.txt are now within outputfile.txt.
A single > will redirect output to a file, overwriting the contents of the file if it exists. A double >> will redirect output to a file, but will append the output to the file if it exists, and create a new file if it doesn’t.
By default, redirection works on stdout, while stderr will typically go to the screen. If you wish to redirect stderr to the same “stream” as stdout, use:
$ ls -l /usr/bin/user2 > output.txt 2>&1
/usr/bin/user2 should not exist, but whether if it does or does not the stdout and stderr from the “ls -l” on that directory will both go to the output.txt file. So, any error messages should be in that file rather than display on the screen along with the output of the command.
You can split where the stdout and stderr go to as well. For instance:
ps ax 1>out.txt 2>err.txt
This line would see what processes are running. The standard-out put (1) would go to the out.txt file, the standard error (2) would go to the err.txt file.
You can also use files as input as well. For instance
program < file
The file.txt would provide the input for the program being run.
To review:
- > - redirects the standard out to a file
- >> - redirects the standard out to be appended to a file
- < - denotes you want to grab input from a file and redirect it into a program
- 1> - states you want the standard out redirected to a file
- 2> - states you want the standard error redirected to a file
- 2>&1 specifying this would stated you want both standard out and standard-error going to a file
Pipes
Piping is a bit more simple. The act of piping is sending one command into another command. This can be used with any two commands, but for the example we’ll use ls and grep. Grep searches through a list of data or from a file and prints the lines to the screen. For instance.
user@hostname ~> ls -l | grep file.txt
The ls -l in this case would normally print on the screen the files listed in the directory we’re running it in. Using the “|” tells the shell to take that output and “pipe” it to grep so grep can search for file.txt. If there is a file.txt, the command will print that file one the screen.
Piping and Redirection can work in conjunction with each other. For instance:
ls -l | grep file.txt > grepfile.txt
this would pipe the results of ls -l to grep so it may find file.txt, and place those results in the grepfile.txt file.
Chaining
Chaining commands simply tells the shell to run one command after another using “&&”. For example:
mkdir dir1 && cd dir1 && touch file1.txt
This line would first make the dir1 directory, then move us to that directory, then touch would create the file file1.txt. If any of these commands error out the chain will be broken. For example, if “cd dir1″ did not work for some reason, we would not move to that directory and touch would not execute.