4. Text Processing

Unix includes very powerful standard tools to process text. I present the functionality and handling of a few of them.

4.1. cat

Concatenates files. Basically it sequentially outputs all files given as parameters. It is practical to output small text files because you don't need to open a pager (for example less). It's also very practical if you have two or more files and want to combine them to one file. Just cat them together and redirect the output to one file.

4.2. less

less is a so-called pager. This means you can page through large text files. With less you can go forward and backwards in a file. Use the cursor keys for this. Sometimes the cursor keys don't work for movement - it depends on your system configuration. It's then possible to use the keys 'j' and 'k' to move down and up respectively. With space you can move by pages.

4.3. grep

grep searches for patterns in files or standard input. The first argument is a pattern to look for and all additional arguments are file names. If no file is given, it uses the standard input. grep outputs all lines that contain the given pattern.

4.4. tail & head

tail and head output the last and the first lines of a file (or standard input) respectively. tail can also “follow” a file so that you immediately see new lines as they are appended to the file.

$ tail -f logfile
$ tail logfile
$ locate '*.txt' | head -3

The first example follows the file logfile. The second example prints the last ten lines (that's the default for tail and head) of logfile and the last example finds all files with the extension “.txt” and prints the top three lines of the result.

4.5. sort

The name of this utility is pretty self-explaining. It sorts the standard input (or a file) and prints the result. Use the parameter '-n' to sort numerically.

$ du -k -s * | sort -n

du calculates the file space usage. With this parameters it calculates the size of all directories given on the command line and prints it in kilobytes. With piping that result to sort, the largest directories get sorted to the bottom of the list. This command can be executed by root in /home to get the users who have the largest home directories.

4.6. sed

sed is a stream editor. This means that you can do some basic text transformations over an input stream - a file or the standard input. It can be used to do some search and replace work in large files and also in an endless number of files.

A few examples.

sed '1d' input.txt
sed 's/foo/bar/g' input.txt

The first example deletes the first line of the input file while the second example replaces all occurrences of “foo” with “bar”.

4.7. awk

awk is a programming language to work on structured text files. It can be used for some fairly complex tasks. For example I have written a complete address application with awk.

awk is very good if you have a file with some records that you want to process. The default configuration takes every line as one record and treats any column (separated by at least one whitespace) as a data field. I give you an example for a very simple script. Suppose you want to calculate the size of all files owned by you[1] in the /tmp directory. `ls -lk' gives you all directories and files in the current directory and the size is printed in kilobytes.

$ ls -lk /tmp
-rw-r--r-- 1 patrice patrice     474 Mar 14 09:24 baz
-rw-r--r-- 1  tester  tester      44 Mar 14 09:24 foo
drwxr-xr-x 2 patrice patrice    4096 Mar 14 09:24 foobar
-rw------- 1 patrice patrice    1261 Mar 14 09:24 quotes

If you want to calculate the size of all files owned by “patrice”, you can use awk. The following script does that for you:

$ ls -lk /tmp | awk '/^-/ && $3=="patrice" {
    total +=$5
}
END {
    print total "k"
}'

How does it work? Every column of the ls command is stored in a variable. The variable is a dollar sign followed by the number of the column. The first line of the awk script matches all lines starting with a dash (this are files) and where the third column is equal to “patrice”. As the third column contains the owner name this gets all files owned by patrice. The block inside the curly braces gets executed for each match. This code sums up the size of this file to a total.

“END” is a special match, the block gets executed after the input stream has ended. So this prints out the sum of the matched files followed by the letter “k”.

Notes

[1]

In this example I use the user name “patrice” as this is my login.