Essential Habits For Being Highly Productive In Linux

2019-12-30 - By Robert Elder

     The purpose of this article is to give beginner users of Linux some insights into the habits that make expert users extremely productive.  A special emphasis will be placed on working within a command-line environment.

17)  Alt + .

     This first habit involves using a shortcut to quickly type the last argument of the last command.  For example, let's review a case where you need to type a very long file or path name multiple times in the same terminal session.  In this example, we need to make a copy of a file and then edit the copy:

cp src/main/something/somethingelse/biglongfilename.c src/main/something/somethingelse/biglongfilename2.c

     After running the copy command, most Linux noobs would probably carefully type out the entire file name again:

vi src/main/something/somethingelse/biglongfilename2.c

     The faster way to accomplish this would be to press and hold the alt key followed by the period key which automatically pastes that last argument of the last command for us:

     vi (then press Alt + dot)

     Depending on your keyboard configuration and version of Linux, you may have to substitute the alt key with a different key depending on what your system treats as the 'meta' key.  The 'meta' key can also be configurable.  You can check your current configuration with this command:

xmodmap -pke | grep Meta

     The component of Linux that implements this functionality is called 'readline'.  You can type 'man readline' and then browse the fairly confusing documentation to learn about the different shortcuts that readline offers you in a commandline environment.

16)  Ctrl + a And ctrl + e

     There is another functionality of readline that will allow you to jump to the start or end of the current command that you're typing.  Consider a case where you're trying to commit some of your code changes and you type the following commit message only to realize that you're forgotten to type the word 'git' at the start of your command:

commit -m "Randomly changed the code to hopefully fix all the bugs, didn't test; should be fine for prod."

     In this situation, many people would consider using the arrow keys to slowly and painfully move the cursor all the way back to the start of the command so they can type the word 'git'.  Instead, you can just hold the control key followed by the 'a' key to immediately move the cursor to the start of the line:

git commit -m "Randomly changed the code to hopefully fix all the bugs, didn't test but should be fine for prod."

15)  Using Less to Read the Contents of Files and Pipes

     One of the most common things you'll need to do as a Linux expert is be able to quickly read files and raw data.  This can be accomplished by editors, but occasionally, you'll encounter use cases where editors don't work.  For example, if you have a huge file of many gigabytes, editors may crash or work slowly when attempting to view these files.  That's why you'll want use the 'less' command for these applications.  Less is also useful whenever you want to briefly capture and view the output of a command or shell pipe.  By piping the output of a command into less, you can read through it and then discard the output, all without needing to create temporary files.

14)  Using Grep For Filtering Files & Pipes

     Grep is a general-purpose command that can be used to search for matching text in files, but it's true power comes from that fact that you it can be used in so many different contexts.  If you're using grep right, you'll probably find that you use it many times per day to find matching files in a repository, matching variable names in your code, or filtering out log statements from your web server:

tail -f -n 0 *.logs | grep abc

13)  Use Command-Line Editors

     Another thing that Linux experts do frequently is use command-line editors.  The two most popular command-line editors are 'Vim' and 'Emacs'.  'Nano' is another common command-line editor that is more friendly to beginners.  There are several reasons why experts prefer command-line editors:  First, they're very low on memory usage compared to graphical editors like Eclipse, sublime text or atom.  For example, my copy of vim is currently using 62 Megabytes, whereas graphical editors usually need to use hundreds of megabytes or even gigabytes, especially when large projects are loaded.

     Another important reason to use command-line editors is to allow for easier editing of files on remote computers.  It is possible to remotely use graphical editors, but this always requires a faster connection and more bandwidth.  It also requires more careful security habits when securing your remote system.   Many programmers only ever need to edit files locally, but for many Linux system administrators, they may spend their entire careers working remotely on other computers.

12)  Use Man Pages

     Most people know about man pages in Linux, but only experts frequently use them.  This is because man pages can be hard to navigate, item #8 discusses a trick to make searching faster.

     It's worth emphasizing that there are man pages on nearly everything.  In fact, you can probably just think of any Linux term and if you type man <something> you'll probably get a man page for it.  This includes more than just commands, but also many builtin C functions too:

     for example:

man printf
man socket
man memcpy
man ls
man man

     A great tip is to type 'man' followed by first few characters of something that you think might have a man page, then tap the tab key.

11)  Using G to Jump to End of File

     Whenever you need to view or edit a file, the most common place you'll need to navigate to is at the end of the file.  For files with many lines of text, it can be time consuming to jump down one line or screen at a time.  Fortunately, there's a keyboard shortcut to jump directly to the end using Shift + G.  This feature works in a variety of editors and contexts such as in vim, less, and man pages.

10)  cd -, pushd + popd

     Often, you'll find yourself needing to change between two different directories that both have very long path names.  An inexperienced Linux user would probably just deal with re-typing the path names over and over each time.  However, the pros know that whenever you need to go back to the last directory you were in, you just use 'cd -'.

     To achieve a similar result and go back even further, you can use the pushd and popd commands.  The pushd command can be used just as you would use the cd command to navigate to a different directory.  When you want to go back, instead of using cd, just type popd to go back to the most recent directory that you were in when you issued the pushd command.  This feature uses a stack to keep track of historical directories, so you can use it for deeply nested navigation.

9)  Master Regular Expressions

     One of the most important things you can do as a Linux expert, is master regular expressions.

     Regular expressions are pieces of text that describe rules for matching other pieces of text.  They can be used for performing searches or searches with replacements.  Regular expressions are usually written inside the context of another programming language or editor.  For this reason, the exact regular expression syntax and behaviour can be slightly different depending of the context in which they're written.  

     Although there are many different flavours of regular expressions, most of them have remarkably similar syntax and behaviour.  This list shows a brief overview of the most commonly used regular expression features:

(   )  ->  Grouping
( | )  ->  Grouping with or.
[]     ->  Character classes.
*      ->  Zero or more
+      ->  One or more
.      ->  Any character other than newline.
$      ->  End of line.
^      ->  Beginning of line.

     Regular expressions can be confusing at times, but they are a requirement for efficient text manipulation.  They can be especially useful when you need to make bulk changes to source code that happens to obey very specific patterns.

     For example, an ideal use case of using regex replacements in code would be help with perform code refactoring.  Let's assume we're in the process of refactoring the function do_stuff() which appears everywhere our code in the following form:

do_stuff(struct * s1, struct * s2, struct * s3);

     In our case, we want to refactor every call to this function to pass the first and last arguments into another function first, and then past the result of that function into the do_stuff function as the second parameter.  In other words, for our final result, every call to the function do_stuff that looks like this:

do_stuff(s1, s2, s3);

     Would turn into this:

do_stuff(s2, another_func(s1, s3));

     We can use the following regex based replacement to accomplish this in vim:

%s/do_stuff(\([^,]\+\),\([^,]\+\),\([^,]\+\))/do_stuff(\2, another_func(\1, \3))/g

     Depending on how many instances of this function are in your code, you might suggest that it would be faster and easier to make all the changes manually.  However, the biggest benefit of using a regex replacement for this situation is that your result is more likely to be completely perfect or completely broken.  A completely broken result is more likely to be noticed and corrected immediately, whereas a manually manipulated result with a few subtle errors can last undetected for a very long time.

     Let's assume that we really did make a mistake in this case, and that we actually wanted to use the first and second arguments instead of the first and third.  We can simply press the 'u' key in vim to undo all of the changes, then instead of re-typing the entire regex again, simply press the 'up' arrow key to bring back the most recent replacement.  You can then quickly adjust the argument numbers and immediately re-apply the corrected regex in only a few seconds.  Imagine having to re-do correction like this by hand with many tens of function calls.  Now imagine how much work it would be if you made a mistake a third time!

     In this code change example, we have assumed that the function do_stuff() is never called with arguments that are also function calls. If this were the case, the specific regex shown here wouldn't work and we'd need to use a different approach to matching the arguments.

     It's worth pointing out that most of the biggest issues that keeps people from using regular expressions is not related to the difficulty of regular expressions themselves. Instead, it's usually the difficulty of knowing all the escaping rules of the enclosing language.  Most of the special characters used in regular expressions are also special characters that require escaping even outside of the regular expression itself.

     For example, here is what our replacement rule above would look like if we didn't need to escape anything:

%s/do_stuff(([^,]+),([^,]+),([^,]+))/do_stuff(\2, another_func(\1, \3))/g

     If we break this up in to parts, it becomes a lot less intimidating:

%s/  ->  Indicates search on all lines of file

do_stuff(([^,]+),([^,]+),([^,]+))  ->  The search pattern.  The pattern ([^,]+) captures each argument.

/  ->  Separator

do_stuff(\2, another_func(\1, \3))  ->  The replacement rule

/g -> End of replacement, and apply globally.

8)   Use '/' to Search Man Pages, Vim, or Less.

     Experts don't have time to carefully read every word of man pages or log files.  That's why they use the slash character to enter search mode and find what they're looking for immediately.

     For example, it can be easy to forget which command-line flag for grep is used to indicate that file names should be printed before matches.  Instead of re-reading the entire man page, you can type 'man grep', and then press the forward slash character to enter search mode.  Then, type the word 'filename' and press enter.  This highlights the word 'filename' everywhere and automatically scrolls down to the fist occurrence of the word 'filename' in this man page.  You can see that the first match of this search has scrolled down to exactly the information we want to see.  Here, it is evident that the command-line flag we're looking for is '-H'.

     If we had typed word 'file' instead of 'filename', there would be many more irrelevant matches before the one we want.  In this case, you can press the 'n' key which stands for 'next' to immediately jump to the next match.  In order to move as fast as possible, consider spamming the 'n' key as fast as you can.  Don't worry if you accidentally jump past the text you want, because holding the shift key while pressing 'n' will cause you to move back up to the previous match.

     This way you can completely handle navigating up and down the man page just by searching for something that you know will be above or below your current position in the text.

     This feature can be used in many different contexts, including man pages, less, vim, git log.  This is because programs like man pages and git actually use less as their text pager.  When you type 'man something', and start scrolling down, you're actually interacting the 'less' program not 'man'.  If you don't believe me, open a man page or type git log, then press 'h' and you'll be greeted by the documentation for 'less' program.

7)  The Find Command.

     The find command is extremely useful for doing a quick assay of a particular part of a filesystem.  A common and simple use case for the find command occurs after cloning a git repository or unzipping an archive.

     In this situation, you'll wonder "How many files are here, and where is everything?"  To answer this question quickly you can just run 'find .':

find .

     If the output includes way too many files to be meaningful, you can pipe the output into 'less', and use the slash search functionality that was just discussed to search around for something you might be interested in.

     The find command allows you to specify a search pattern too:

find -name '*.jpg'

     Note that it's possible to get subtly incorrect results if you forget the quotes and allow the asterisk character to expand in the shell.  A better way is to filter out file names is by piping the result of find into grep:

find . | grep jpg

     Finally, you can use the find command to execute arbitrary shell commands against anything that matches your search pattern.  For example, here is a command that will find all files under the current directory that have the .jpg extension and print out the width and height of each image:

find . -name '*.jpg' -exec identify -format "%Wx%H\n" {} \;

6)  Write Custom Scripts

     One of the best reasons to learn how to use the command line is so you can automate all of your work.  The upfront work of figuring out how to do something with a command is often much greater than it would be to just use a graphical interface for the same function.  However, if you can figure out how to do something with a command once, you can then do it thousands of times automatically without supervision.  This is why Linux experts will always carefully save their commands when doing something for the first time.  They know that the next time this task comes up, the saved commands can be refactored into a script and does everything automatically.  For example, here's a quick demo of a situation where we want to name images with time and dimension information:

     After some research, we can come up with this command to obtain the image dimensions:

identify -format "%Wx%H\n" foo.jpg

     And the following command to show the current date and time:

date '+%Y-%m-%d_%H-%M-%S'

     Now, we can use this command to manually rename our file:

mv foo.jpg yyy-xxx-foo.jpg

     Since we have all the individual steps now, the process can be automated by placing these commands inside a script that takes the image name as an argument:

IMAGE_NAME=${1}
DIMENSIONS=`identify -format "%Wx%H\n" "${IMAGE_NAME}"`
THE_DATE=`date '+%Y-%m-%d_%H-%M-%S'`
mv "${IMAGE_NAME}" "${THE_DATE}-${DIMENSIONS}-${IMAGE_NAME}"

     To automate what we just did, You can simply copy the above commands into a file and then add the execute permission:

chmod u+x rename_image.sh

     The script can then be run like this:

./rename_image.sh foo.jpg

     And here is an example result:

2019-12-25_20-09-47-1059x919-foo.jpg

5)  Effectively Use Paths Like '..' &'.'

     In Linux, every directory has two special entries which appear as one and two periods.  When you first learn about them, these entrires can seem as though they're special and magic since they don't act like normal files or directories.  However, when specifying paths, you can just treat them as though they were any other regular sub-directory.  The only difference is, the '.' entry acts as though it were a hard-link to whatever the current directory is, and the '..' directory acts as though it were a hard-link to the parent directory.  Effectively, they act as shortcuts in the filesystem.

     You can use the 'ls' command to see that the iNode number of the '.' directory has the same iNode number as the current directory.  Also, the '..' directory has the same iNode number as its parent directory:

ls -lia
ls -lia ../

     Most people have seen these two special entries used in the following cases:

./a.out
cd ..

     But many people are unaware that all of the following examples have meaningful interpretations as well:

cd .
cd ./../././../..
cd ././././././
cd ../../../../

4)  Use Git

     A significant number of new programmers don't use any form of source control.  They prefer to share zipped files by email or using services like Dropbox, which is not ideal for source code.

     All expert Linux users use at least one form of source control to keep track of their code changes, or in some cases other types of files as well.  The most commonly used form of source control is called 'git'.   If you don't know what git is, you can read this article: What is Git and Why Is It Useful?.

3)  Use ctrl+R

     Often, you'll find that you need to type a very long and hard to remember command that you've run recently.  If you can only remember a small part of the command, you can automatically bring it up by holding the CTRL key, followed by the R key, then typing the substring that appears in your command.  Once the command you want appears in this search feature, pressing the enter key will immediately run the full command without you needing to type the rest of it.  If you want to change the command slightly before you run it, you can press one of the right or left arrow keys to get back to a normal command-line prompt, but with the full command pasted and still editable.

2)  Use Up Arrow Key.

     For even quicker repetition or correction of commands, experts frequently use the up arrow key to manually search through recent previous commands.  This is especially useful when you type a long command, but accidentally include a small mistake that caused it to not run.  If you miss the command you want, you can also press the down arrow key to cycle back in the other direction.

1)  Use Tab Completion.

     The most important feature that Linux experts use is called tab completion.  While you're typing commands, the shell detects when you press the tab key and attempts to automatically complete the rest of the command by checking which valid possibilities exist.  For example, when typing the following:

ls -1 ~/Do

     If you press the 'tab key', you'll see two options for Downloads/ and Documents/ since these are the only two valid options for completing this command.  If you type one additional character:

ls -1 ~/Dow

     and then press tab again, the command is no longer ambiguous, so the word 'Downloads' will be typed out since there is no other valid option.

     This feature is especially useful when navigating to a different directory with a long path name:

cd /tmp/somedir1/src/main/files/docs/stuff/etc/

     Tab auto-completion can be surprisingly sophisticated.  For example, it can even work when specifying arguments that involve files on other computers over an SSH connection, such as with scp copy commands.

     Tab completion will also work with many programs that have nothing to do with directories, such as the ip command or man pages:

ip link show
man lin

     If you see someone you love struggling to type long filenames by hand, please talk to them about tab completion.

A Surprisingly Common Mistake Involving Wildcards & The Find Command Published 2020-01-21	$1.00 CAD Terminal Block Mining Simulation Game	A Guide to Recording 660FPS Video On A $6 Raspberry Pi Camera Published 2019-08-01	The Most Confusing Grep Mistakes I've Ever Made Published 2020-11-02
Use The 'tail' Command To Monitor Everything Published 2021-04-08	An Overview of How to Do Everything with Raspberry Pi Cameras Published 2019-05-28	An Introduction To Data Science On The Linux Command Line Published 2019-10-16	Using A Piece Of Paper As A Display Terminal - ed Vs. vim Published 2020-10-05