Unix includes very powerful standard tools to process text. I present the functionality and handling of a few of them.
Concatenates files. Basically it sequentially outputs all files given as parameters. It is practical to output small text files because you don't need to open a pager (for example less). It's also very practical if you have two or more files and want to combine them to one file. Just cat them together and redirect the output to one file.
less is a so-called pager. This means you can page through large text files. With less you can go forward and backwards in a file. Use the cursor keys for this. Sometimes the cursor keys don't work for movement - it depends on your system configuration. It's then possible to use the keys 'j' and 'k' to move down and up respectively. With space you can move by pages.
grep searches for patterns in files or standard input. The first argument is a pattern to look for and all additional arguments are file names. If no file is given, it uses the standard input. grep outputs all lines that contain the given pattern.
grep expects a regular expressions. Regular expressions are a very powerful syntax to express search patterns. They are used in many Unix tools. grep(1) provides some basic information about them. perlre(1) has more extensive documentation and should be available if you have Perl installed. “Mastering Regular Expressions” is a great book about regular expressions.
tail and head output the last and the first lines of a file (or standard input) respectively. tail can also “follow” a file so that you immediately see new lines as they are appended to the file.
$ tail -f logfile $ tail logfile $ locate '*.txt' | head -3 |
The first example follows the file logfile. The second example prints the last ten lines (that's the default for tail and head) of logfile and the last example finds all files with the extension “.txt” and prints the top three lines of the result.
The name of this utility is pretty self-explaining. It sorts the standard input (or a file) and prints the result. Use the parameter '-n' to sort numerically.
$ du -k -s * | sort -n |
du calculates the file space usage. With this parameters it calculates the size of all directories given on the command line and prints it in kilobytes. With piping that result to sort, the largest directories get sorted to the bottom of the list. This command can be executed by root in /home to get the users who have the largest home directories.
sed is a stream editor. This means that you can do some basic text transformations over an input stream - a file or the standard input. It can be used to do some search and replace work in large files and also in an endless number of files.
A few examples.
sed '1d' input.txt sed 's/foo/bar/g' input.txt |
The first example deletes the first line of the input file while the second example replaces all occurrences of “foo” with “bar”.
awk is a programming language to work on structured text files. It can be used for some fairly complex tasks. For example I have written a complete address application with awk.
awk is very good if you have a file with some records that you want to process. The default configuration takes every line as one record and treats any column (separated by at least one whitespace) as a data field. I give you an example for a very simple script. Suppose you want to calculate the size of all files owned by you[1] in the /tmp directory. `ls -lk' gives you all directories and files in the current directory and the size is printed in kilobytes.
$ ls -lk /tmp -rw-r--r-- 1 patrice patrice 474 Mar 14 09:24 baz -rw-r--r-- 1 tester tester 44 Mar 14 09:24 foo drwxr-xr-x 2 patrice patrice 4096 Mar 14 09:24 foobar -rw------- 1 patrice patrice 1261 Mar 14 09:24 quotes |
If you want to calculate the size of all files owned by “patrice”, you can use awk. The following script does that for you:
$ ls -lk /tmp | awk '/^-/ && $3=="patrice" { total +=$5 } END { print total "k" }' |
How does it work? Every column of the ls command is stored in a variable. The variable is a dollar sign followed by the number of the column. The first line of the awk script matches all lines starting with a dash (this are files) and where the third column is equal to “patrice”. As the third column contains the owner name this gets all files owned by patrice. The block inside the curly braces gets executed for each match. This code sums up the size of this file to a total.
“END” is a special match, the block gets executed after the input stream has ended. So this prints out the sum of the matched files followed by the letter “k”.
[1] | In this example I use the user name “patrice” as this is my login. |