Use The 'tail' Command To Monitor Everything

2021-04-08 - By Robert Elder

Introduction

     This article is about how to use the 'tail' command.  Here is an overview of how you can use the 'head' and 'tail' commands to output different parts of a file:

head -n 3 business-tips.txt  #  (POSIX Compatible)
#  OR
head -n +3 business-tips.txt #  (Not POSIX Compatible)

 1)  Buy low, sell high.
 2)  Invest with a plan.
 3)  Follow your heart.

 4)  Find your passion.
 5)  Believe in yourself.
 6)  Take risks, be bold.
 7)  Play it safe, and be considerate.
 8)  Realize that there are no limits.
 9)  Realize that you have limits.
10)  You can rest when you're dead.
11)  Take time to care for yourself.
12)  Ignore what negative people say.
13)  Listen to constructive criticism.

tail -n 3 business-tips.txt   #  (POSIX Compatible)
#  OR
tail -n -3 business-tips.txt  #  (POSIX Compatible)

 1)  Buy low, sell high.
 2)  Invest with a plan.
 3)  Follow your heart.
 4)  Find your passion.
 5)  Believe in yourself.
 6)  Take risks, be bold.
 7)  Play it safe, and be considerate.
 8)  Realize that there are no limits.
 9)  Realize that you have limits.
10)  You can rest when you're dead.

11)  Take time to care for yourself.
12)  Ignore what negative people say.
13)  Listen to constructive criticism.

head -n -3 business-tips.txt # (Not POSIX Compatible)

 1)  Buy low, sell high.
 2)  Invest with a plan.
 3)  Follow your heart.
 4)  Find your passion.
 5)  Believe in yourself.
 6)  Take risks, be bold.
 7)  Play it safe, and be considerate.
 8)  Realize that there are no limits.
 9)  Realize that you have limits.
10)  You can rest when you're dead.

11)  Take time to care for yourself.
12)  Ignore what negative people say.
13)  Listen to constructive criticism.

tail -n +3 business-tips.txt  # (POSIX Compatible)

 1)  Buy low, sell high.
 2)  Invest with a plan.

 3)  Follow your heart.
 4)  Find your passion.
 5)  Believe in yourself.
 6)  Take risks, be bold.
 7)  Play it safe, and be considerate.
 8)  Realize that there are no limits.
 9)  Realize that you have limits.
10)  You can rest when you're dead.
11)  Take time to care for yourself.
12)  Ignore what negative people say.
13)  Listen to constructive criticism.

     The table above shows the different ways that you can use the 'head' and 'tail' commands to output the beginning and end of a file.  You can get different results by passing a positive or negative number to these commands, although not all combinations are POSIX compliant.  Therefore, they may not be supported in all implementations of head/tail.  In the examples shown above, the green colour text is what would get printed, and the gray text is the part that would be omitted.

The Simplest Use Of The 'tail' Command

     You can use the 'tail' command to print out the 'tail-end' of a file or stream. If we start with the following file 'business-tips.txt':

 1)  Buy low, sell high.
 2)  Invest with a plan.
 3)  Follow your heart.
 4)  Find your passion.
 5)  Believe in yourself.
 6)  Take risks, be bold.
 7)  Play it safe, and be considerate.
 8)  Realize that there are no limits.
 9)  Realize that you have limits.
10)  You can rest when you're dead.
11)  Take time to care for yourself.
12)  Ignore what negative people say.
13)  Listen to constructive criticism.

     and run the tail command like this:

tail business-tips.txt

     We'll get the following output:

 4)  Find your passion.
 5)  Believe in yourself.
 6)  Take risks, be bold.
 7)  Play it safe, and be considerate.
 8)  Realize that there are no limits.
 9)  Realize that you have limits.
10)  You can rest when you're dead.
11)  Take time to care for yourself.
12)  Ignore what negative people say.
13)  Listen to constructive criticism.

     which shows the last 10 lines in the file.  The default number of lines to display is 10, but you can use the '-n' flag to specify a different number.  For example, this command:

tail -n 5 business-tips.txt

     will output the following:

 9)  Realize that you have limits.
10)  You can rest when you're dead.
11)  Take time to care for yourself.
12)  Ignore what negative people say.
13)  Listen to constructive criticism.

     which shows the last 5 lines in the file.

Tail's 'Follow' Mode

     One of the most common uses for the tail command is it's "follow" mode that you get when using the '-f' flag.  If we run a command like this:

tail -f /var/log/kern.log

     you'll see the last few lines from the file '/var/log/kern.log' printed to the screen, and the cursor will appear to 'hang' as if it was waiting for something:

     In this case, the tail command is waiting for more lines to be appended to the end of the file so it can print them to the screen.  The tail command will keep watching for new lines until you interrupt 'tail' by pressing and holding the CTRL key followed by the 'c' key.

     Using the tail command with the '-f' flag is a common way to monitor log files.  By default, the 'tail' command will output a few lines that were already at end of the file even when you use the '-f' flag.  This can be confusing if you're only interested in monitoring for log statements that happened after you issued the tail command.  Therefore, I like to include the '-n' flag with a value of 0 whenever I use '-f':

tail -f -n 0 /var/log/apache2/access.log

     This way, you won't get distracted by outdated information that was already in the file when you issued this tail command.  You can also follow multiple files at once with the 'tail' command like this:

tail -f -n 0 /var/log/apache2/access.log /var/log/apache2/errors.log

     After issuing the above command, tail will output any of the lines that are written to any of the files.  When multiple files are followed at once, 'tail' will usually print out a 'header' that indicates which file the corresponding lines came from:

==> /var/log/apache2/access.log <==

==> /var/log/apache2/errors.log <==

==> /var/log/apache2/access.log <==
Hello from access log!

==> /var/log/apache2/errors.log <==
Hello from error log!

     Most shells also support wildcards that make it even more convenient to specify a large number of files to follow at once with 'tail':

tail -f -n 0 /var/log/apache2/*

Following Files That Require Elevated Privileges

     This is a very specific, but quite common scenario that might cause you some confusion with the 'tail' command:  What if you want to use a wildcard to follow multiple files that are in a directory that is only readable by another user?  You might think 'oh, I'll just use sudo' and do something like this:

sudo tail -f -n 0 /var/log/apache2/*

     But this doesn't work!  You'll still get the permission issue since your shell will attempt to expand the wildcard (the '*' part) before rest of the command is even passed to sudo.  The exact solution will depend on which shell you use, but for the bash shell the following will work:

sudo sh -c 'tail -f -n 0 /var/log/apache2/*'

Use 'tail' To Find The 4 Most Recently Copied Files

     Let's say you're in the process of backing up some files to an external hard drive, and you'd like to see the progress of the last few files that were copied.  You could use the following 'ls' command to list all files in the current directory, with the most recently modified file shown last:

ls -ltr

     which outputs something like this:

...many many lines...
-rw-r--r-- 1 robert robert    8839640 Feb 27 00:30 my-cool-file-30356.dat
-rw-r--r-- 1 robert robert  194326506 Feb 27 00:30 my-cool-file-3093.dat
-rw-r--r-- 1 robert robert  665529770 Feb 27 00:30 my-cool-file-3396.dat
-rw-r--r-- 1 robert robert  629536792 Feb 27 00:31 my-cool-file-5005.dat
-rw-r--r-- 1 robert robert  987631920 Feb 27 00:31 my-cool-file-7551.dat
-rw-r--r-- 1 robert robert  654947606 Feb 27 00:31 my-cool-file-7606.dat
-rw-r--r-- 1 robert robert 1294157424 Feb 27 00:31 my-cool-file-8633.dat
-rw-r--r-- 1 robert robert   29884416 Feb 27 00:31 my-cool-file-8673.dat

     but this shows too much information.  It would look nicer if we could see only 4 of the most recently modified files.  We can do that using the tail command:

ls -ltr | tail -n 4

     and now we get the following output:

-rw-r--r-- 1 robert robert  987631920 Feb 27 00:31 my-cool-file-7551.dat
-rw-r--r-- 1 robert robert  654947606 Feb 27 00:31 my-cool-file-7606.dat
-rw-r--r-- 1 robert robert 1294157424 Feb 27 00:31 my-cool-file-8633.dat
-rw-r--r-- 1 robert robert   29884416 Feb 27 00:31 my-cool-file-8673.dat

     But it's inconvenient to manually run this over and over.  We can also leverage the 'watch' command to run this command over and over again so we get to see the file copy progress as it's happening:

watch -n 0.1 -d 'ls -latr | tail -n 4'

     The use of the '-d' flag above will cause 'watch' to highlight the differences, and the '-n' flag will cause the command to run again every 0.1 seconds instead of the default interval of 2 seconds.

Counting From Beginning Instead Of End

     You may encounter situations where you want to extract the 'end' of a file where you don't know in advance how long the 'end' will be.  The 'tail' command can still help you in these situations because you can specify an offset from the beginning where you'd like to start the output.  For example, let's consider the file 'some_lines.txt' with the following contents:

My Favourite Stonks:
GME
AMC
RKT
PLTR

     We can use the 'cat' command with the '-n' flag to add some line numbers for clarity:

cat -n some_lines.txt

     which produces this output:

     1	My Favourite Stonks:
     2	GME
     3	AMC
     4	RKT
     5	PLTR

     If we only wanted the last two lines, we would use the tail command like this:

cat -n some_lines.txt | tail -n 2

     which provides this output:

     4	RKT
     5	PLTR

     But, if we add a plus sign in front of the two, we get a different interpretation:

cat -n some_lines.txt | tail -n +2

     this prints out the 'end' of the file starting at line 2 continuing until the end (regardless of the total number of lines):

     2	GME
     3	AMC
     4	RKT
     5	PLTR

     If we then add some more data to this file:

echo "FOO1" >> some_lines.txt
echo "FOO2" >> some_lines.txt

     those lines will now be output too when we issue the same command:

cat -n some_lines.txt | tail -n +2

     will output:

     2	GME
     3	AMC
     4	RKT
     5	PLTR
     6	FOO1
     7	FOO2

Using 'tail' With Bytes Instead Of Lines

     You can also use the 'tail' command to extract the last few bytes instead of the last few lines with the '-c' flag.  Here is an example of using the tail command to extract the last byte of the file from the previous example:

cat -n some_lines.txt | tail -c 1

     This doesn't print any visible characters, so let's pipe the result into xxd to see what the byte is:

cat -n some_lines.txt | tail -c 1 | xxd

     and the result is:

00000000: 0a                                       .

     which shows that the last byte in the file is a newline, just as we'd expect.  Let's do this again and extract the last 4 characters:

cat -n some_lines.txt | tail -c 4

     which provides this output

LTR

     and if we pipe this into xxd:

cat -n some_lines.txt | tail -c 4 | xxd

     we'll see the following:

00000000: 4c54 520a                                LTR.

Finding The 3 Newest Books

     In the article on the sort command, we reviewed an example of how to find the 3 newest books from the following unsorted list:

Tropic of Cancer,Henry Miller,1934
Housekeeping,Marilynne Robinson,1981
Deliverance,James Dickey,1970
The Sun Also Rises,Ernest Hemingway,1926
The Great Gatsby,F. Scott Fitzgerald,1925
The Corrections,Jonathan Franzen,2001
The Berlin Stories,Christopher Isherwood,1946
Call It Sleep,Henry Roth,1935
Slaughterhouse-Five,Kurt Vonnegut,1969
Light in August,William Faulkner,1932

     We showed how you can use the sort command to sort each line in the file according according to the three different columns:

sort -t ',' -k 3,3n -k 2,2 -k 1,1 -s books.txt

     which produces this output:

The Great Gatsby,F. Scott Fitzgerald,1925
The Sun Also Rises,Ernest Hemingway,1926
Light in August,William Faulkner,1932
Tropic of Cancer,Henry Miller,1934
Call It Sleep,Henry Roth,1935
The Berlin Stories,Christopher Isherwood,1946
Slaughterhouse-Five,Kurt Vonnegut,1969
Deliverance,James Dickey,1970
Housekeeping,Marilynne Robinson,1981
The Corrections,Jonathan Franzen,2001

     We can this pipe this directly into the 'tail' command and tell it to only print out the last 3 lines:

sort -t ',' -k 3,3n -k 2,2 -k 1,1 -s books.txt | tail -n 3

     which produces this output:

Deliverance,James Dickey,1970
Housekeeping,Marilynne Robinson,1981
The Corrections,Jonathan Franzen,2001

     As you can see above, we've found the 3 newest books from an unsorted list using a combination of the 'sort' and 'tail' commands.

Finding The Most Popular Name

     In the article on the 'uniq' command, we reviewed an example of how to find the most popular name from a list of names:

Verity Jayda
Verity Jayda
Verity Jayda
Verity Jayda
Justy Kaiden
Christopher Rene
Christopher Rene
Christopher Rene
Branden McKenna
Branden McKenna

     the first step was to make sure the list of names was sorted (as a requirement of the 'uniq' command):

cat names.txt | sort

     then, the list of names is piped into the 'uniq' command with the '-c' flag which provides a count of the number of occurrences for each name:

cat names.txt | sort | uniq -c

     and the output of this is:

      2 Branden McKenna
      3 Christopher Rene
      1 Justy Kaiden
      4 Verity Jayda

     Now, we can use the 'sort' command again, but this time use numeric sorting to order the list according to which name has the most occurrences:

cat names.txt | sort | uniq -c | sort -n

     which provides this output:

      1 Justy Kaiden
      2 Branden McKenna
      3 Christopher Rene
      4 Verity Jayda

     Now, we just need to use the 'tail' command to pick out the last line in the output:

cat names.txt | sort | uniq -c | sort -n | tail -n 1

     and the output of this command is:

      4 Verity Jayda

     And the above name is the most popular name in the list.  This process lets go from an unsorted list of many names to a single record that tells what the most popular name is.

Potentially Unbounded Memory Use

     The tail command can potentially experience unbounded memory requirements in some cases.  This becomes obvious when you consider the fact that the tail command must be capable of working with streams where the length of the input it not known in advance.  If you want to output the last N lines from a stream, you need to be able to store at least N lines in memory so that when you finally detect the end of the stream you'll be able to output what came before it.

     In the interest of giving some numbers, here's a contrived example that reads a bunch of lines from /dev/urandom.  We then use 'head' to grab the first 10,000,000 lines and pipe them into 'tail' to get the last line.  This entire pipe is run as a sub-shell through /usr/bin/time so we can get some performance numbers:

/usr/bin/time -v sh -c 'xxd /dev/urandom | head -n 10000000 | tail -n -1 | wc -l'

     which gives some output that includes these lines:

	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.64
	Maximum resident set size (kbytes): 2104

     If we start increasing the number of lines that tail should output:

/usr/bin/time -v sh -c 'xxd /dev/urandom | head -n 10000000 | tail -n -100000 | wc -l'

     the max memory usage starts to increase:

	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.01
	Maximum resident set size (kbytes): 8768

     And increase:

/usr/bin/time -v sh -c 'xxd /dev/urandom | head -n 10000000 | tail -n -1000000 | wc -l'

	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.06
	Maximum resident set size (kbytes): 68656

     And increase...

/usr/bin/time -v sh -c 'xxd /dev/urandom | head -n 10000000 | tail -n -10000000 | wc -l'

	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.46
	Maximum resident set size (kbytes): 668992

     From the above output, we can see that a maximum of 668992KiB or 653.312MiB was required by at one time during the running of the above tail command.  Obviously, 10,000,000 lines is quite a lot, but you should keep this in mind for cases where the argument to the 'tail' command might be a variable in a script that could be arbitrarily large.  A similar problem can also occur with the non-POSIX feature of the head command that lets you specify a negative value to '-n'.

     And that's why the 'tail' command is my favourite Linux command.

A Surprisingly Common Mistake Involving Wildcards & The Find Command Published 2020-01-21	$1.00 CAD Terminal Block Mining Simulation Game	A Guide to Recording 660FPS Video On A $6 Raspberry Pi Camera Published 2019-08-01	The Most Confusing Grep Mistakes I've Ever Made Published 2020-11-02
An Overview of How to Do Everything with Raspberry Pi Cameras Published 2019-05-28	An Introduction To Data Science On The Linux Command Line Published 2019-10-16	Using A Piece Of Paper As A Display Terminal - ed Vs. vim Published 2020-10-05	What is SSH? Linux Commands For Beginners Published 2017-04-30