Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

2016-12-08 - By Robert Elder

     In this article, I will attempt to convince you that Expect is an extremely underappreciated tool for automating terminal applications in Unix.

     Why do I feel so strongly about this?  Well, if you're like me you know that the best way to make a great impression at a party is to boast about your excellent understanding of the Unix command-line.  However, if you really want to be the life of the party, you not only need to show that you know the commands, you must also demonstrate that you can automate everything.

     Once you achieve full automation, your development career will consist mostly of staring off into the distance and posing like superman, while your nearby co-workers (who don't know about Expect) remain covered in sweat, frantically trying to type their commands fast enough.  As we'll see below, there are cases where complete automation can only be achieved with the use of Expect (or one of its less popular alternatives).

Examples

     Let's jump right to the examples.  For simplicity, most of these examples are written as copy-and-pasteable 'bash one-liners' where the Expect script is specified as a multi-line string literal from the shell invocation of Expect.  A much better form in production would be to put the expect script inside its own file with an appropriate shebang at the top.  Note that some of the tasks below can be done using a much better method, but the point here is to demonstrate that we can do them using the Expect tool.

Automatically Open Bash Session and Type Characters

     The following example will open a bash session, and type 'ls' without actually executing the command.  The user will then be allowed to take over and press enter to execute the command, or use backspace to remove it.

expect -c '
spawn bash
sleep 0.3
send "ls"
interact
exit
'

Automatically Open Top and Change Layout

     This example will open top, delay for 2 seconds, then press 'M' to sort by memory, then press 'c' to toggle the command presentation, then press 'q' to exit with 2 second delays between each change.  Results may vary depending on your version of top.

expect -c '
spawn bash
sleep 0.3
send "top\n"
interact timeout 2 return
send "M"
interact timeout 2 return
send "c"
interact timeout 2 return
send "q"
sleep 0.3
exit
'

Automatically Open Vim Splits, Modify, Save, Exit

     This example will launch vim, open a split, type "Hi, how are you?", then switch to the other split and type "Im good thanks.".  Finally, it saves and exits from both files.

expect -c '
spawn vim -n /tmp/foo1
interact timeout 1 return
send "\x1b:split /tmp/foo2"
interact timeout 1 return
send "\n"
interact timeout 1 return
send "iHi, how are you?"
interact timeout 1 return
send "\x1b"
send "\x17\x17"
interact timeout 1 return
send "iIm good thanks."
interact timeout 1 return
send "\x1b:wq\n"
interact timeout 1 return
send "\x1b:wq\n"
interact timeout 1 return
exit
'

Trap Ctrl+D In A Bash Session

     This example will print 'You pressed Ctrl+D' every time you press Ctrl+D.  Ctrl+D won't actually be sent to bash.

expect -c '
spawn bash
interact {
  "\x04" {send_user "You pressed Ctrl+D\n"}
}
'

Automatically Trigger Up Key And Other Shortcuts In Bash

     This example will open up a bash shell and press up 2 times, then use the Ctrl+A hotkey to move the cursor to the first character then let the user interact

expect -c '
spawn bash
sleep 0.1
send -- "\x1b\[A"
send -- "\x1b\[A"
send -- "\x01"
interact
exit
'

Automatically Enter Sudo Password

     This is an example of something that you should never do, but you can do it if you want to anyway.  The preferred method of doing thing involves adding an entry to the sudoers file.

expect -c '
spawn bash
sleep 0.3
send -- "sudo ls\n"
expect "password for"
send -- "<Your Password>\n"
interact
exit
'

Automatically Trigger Bash Tab Completion

     Automatically trigger the event that makes tab completion suggestions show up in bash.

expect -c '
spawn bash
sleep 0.3
send "ls\x09\x09\x09"
interact
exit
'

Automatically Open Man Page And Jump To Specific Point

     Open the man page for gcc and automatically jump to the entry on '-Wall', then allow the user to interact.

expect -c '
spawn man gcc
send "/^\\s+-Wall\n"
interact
exit
'

Stop Calculating md5sums Once A Specific Hash Is Found

     Let's say you want to md5sum thousands of huge files and it would be great if you could terminate the 'md5sum *' command as soon as you've found a file with a particular checksum. You can do this:

expect -c '
spawn bash -c "md5sum ./\*"
expect "bd91bd1e383bb460f73b3e0500fa8cd0"
expect "\n" close
'

Choosing The Right Automation Tool

     Depending on the effect you're trying to achieve, most of the things that you could use expect for, are usually done much better using another method.  There are, however, a few tricky cases where the desired effect can't be achieved through a shell script, or through any programmatic interface that the application exposes.  The benefit you get by learning how to work with Expect, is that you can use it to automate pretty much anything that the user could do manually!

     In the above diagram, the terms I'm using can be defined as follows:  Shell scripts are the standard bash, dash, etc. scripts.  What I means by 'application scripts' are the build-in scripting languages that come with many types of applications.  In vim, this would be the '-c' commands you can place on the command-line or in a separately sourced file.  Programs like gdb allow for their own scripting via the -x argument.

     My goal in using this diagram is to illustrate that you could use Expect to emulate the type of stuff you see in shell scripts (like piping and I/O redirection).  You can also automate interactive applications like gdb (but gdb accepts its own scripts via the -x argument).  With Expect, you could take automating something like gdb even further by allowing partial automation so the user can interact in sticky situations, or introduce something dynamic, where Expect can capture output from a gdb session and send it to another application which decides how the rest of the debugging session will take place.

     Here is a list of situations where you might want to use Expect over something else:

• Performing tasks in a text editor normally done by the user (sleep, cursor navigation, control characters etc.)
• Automatically triggering tab completion in a terminal, then handing over control to the user
• Invoking line-editing shortcuts in bash.
• Automatically interact with programs like top in the same way a user would.
• Any situation where an application developer forgot to add an argument based or script-based way of interacting with an application, but did allow the interaction for users.
• Partial automation: Do part of something, then hand interactive over control to the user.
• Much more:  This only scratches the surface.

Autoexpect

     There is also a program called 'autoexpect' that you can use to help generate expect scripts for you.  You simply run

autoexpect

and various input and output information about your actions will be recorded to an expect script with the intention of making it possible to re-play what you just did.  In practice, I have found these scripts to be very verbose and not exactly what I want for a final product, but being able to see the output is very invaluable.  When I created this article, I used autoexpect a few times to verify that the right keystrokes were being sent.

Why Expect Is Underappreciated

     Expect is a critical dependency of the GNU ecosystem.  I tend to prefer investing my time learning and developing against tools that I know will be around for as long as possible.  Expect is a testing dependency of many core components of a modern GNU/Linux operating system, including the gcc compiler and LLVM.  This dependency exists through the DejaGnu package (which is also used by hundreds of other projects).  This fact alone suggests to me that Expect will probably still be supported on systems 20+ years from now.  The fact that 2 major compiler test suites are married to the behaviour of Expect suggests to me that it is also likely to be very stable.

     Expect has terrible brand distinctiveness.  This article is about a piece of software called 'Expect'.  This software is an interpreter and one of its most commonly used commands is also called 'expect'.  Furthermore, the Expect interpreter (or at least it used to be an interpreter?) doesn't really have a close ownership over the language it interprets, it was originally written as an extension to the TCL language.  Therefore, it can be hard to know whether to look for answers using vanilla TCL, or the special 'Expect' flavoured extension.  Since it's difficult to isolate exactly what 'Expect' is, many people will just think about it as 'some confusing TCL thing' and move on.

     Expect got popular before Google existed so its SEO is Terrible!  It is extremely difficult to find information about Expect on Google because 'Expect' is an extremely common word!  For example, see this search for 'expect how to expect something'.  I actually think that Expect might be the worst SEO optimized piece of software I've ever heard of!

     Browsing the TCL Wiki feels like discovering an ancient Roman empire.  After browsing around http://wiki.tcl.tk/201 you get a sense that beneath the surface of what you can find on Google, that there is likely a vast archaeological wonderland of incredible things that Expect can do that the average developer between the ages of 25-34 has never heard of.

Alternatives To Expect

     xdotool is an alternative that would also allow you to do things like send keystrokes to a terminal, although this would require you to have an active X server.  Expect will work even in ancient terminal only environments.

     The Wikipedia article on Expect lists a few alternatives, although none of them seem to be used as much as Expect.  Many of them are actually dead links, or projects with less than 40 stars on GitHub.

Conclusion

     Expect is a great tool for automating terminal applications, and it is extremely well supported.  It is a dependency of 2 major compilers, so it is likely to work everywhere for a long time.  You've probably never heard of it because it is extremely hard to google for, and it has terrible SEO.

A Surprisingly Common Mistake Involving Wildcards & The Find Command Published 2020-01-21	$1.00 CAD Terminal Block Mining Simulation Game	A Guide to Recording 660FPS Video On A $6 Raspberry Pi Camera Published 2019-08-01	The Most Confusing Grep Mistakes I've Ever Made Published 2020-11-02
Use The 'tail' Command To Monitor Everything Published 2021-04-08	An Overview of How to Do Everything with Raspberry Pi Cameras Published 2019-05-28	An Introduction To Data Science On The Linux Command Line Published 2019-10-16	Using A Piece Of Paper As A Display Terminal - ed Vs. vim Published 2020-10-05