Bash Shell Scripting

Comp 141/400D

Related Shotts Chapters: 

Many of the bash scripts used in these notes are here.

Some things we'll cover:


Shell Scripting

Introduction to Shell Scripting

A shell script is a sequence of shell commands, encapsulated into a file.

We can make the file executable (chmod +x filename) and launch the script as ./filename; otherwise we need "bash filename".

The first line of an executable shell script should be

    #!/bin/bash

The '#!' is often read as "shebang" (as in "the whole shebang"). Technically it is a comment to bash, because it begins with #. It tells the system what program to use to interpret the script (and it ca be #!/usr/bin/python3, for example). But the default is usually /bin/sh (which is not the same as bash).

A first script:

    #!/bin/bash

    # Here is a comment

    echo "Hello world!"

Let's put this in a file hello.sh (we'll use the extension .sh for a while, but typically executables in Unix do not get extensions). Now we'd like to run it. At this point, typing

    ./hello.sh

fails, but we can type bash hello.sh. To get ./hello.sh to work. we must make the file executable:

    chmod +x hello.sh

But we still need that ./ to run it. Also, our current directory has to be the same as the directory the program is in. One common strategy is:

We do the latter with

    PATH=$PATH:~/bin

The righthand side is the existing PATH, namely $PATH, followed by ~/bin (and separated by a ':')

Putting ~/bin at the end of the path means that we can't override any existing commands. If that's what we want to do, though, then PATH=~/bin:$PATH is a better choice.

We probably also want to add

    export PATH

Finally, Ubuntu-type distributions often add ~/bin to PATH automatically. Your Loyola virtual machines do this. From ~/.profile:

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

Long options

In scripts, the long options to commands are usually a better choice, for future readability: instead of ls -ad, consider ls --all --directory. 

Continuation lines

You can terminate lines with \ (make sure there is no space following!), and they will be seen by the shell as continued on the following line. This allows breaking up very long lines for readability

bash for

This one is frequently useful directly from the keyboard:

for i in *.text
do
   echo $i
done

for i in A B C D; do        # alternative do location
    echo $i
done

for i in *; do echo $i; done    # one-line version; prone to problems with too many or too few semicolons

This gives us a way of taking a long list of filenames (or command arguments) and dealing with them one at a time.

We've used "i" as the index variable in both cases, but it can be anything; "file" is also popular (assuming the index variable is taking on filenames)

bash arguments

Shell scripts can be given arguments on the command line. The first ten commands are $0 through $9, with $0 being the command name itself. The list of all the arguments is $* (though all the arguments in a bash array is $@, and is usually a better choice). Below is echo3.sh, which echoes arguments 0 through 3.

echo $0
echo $1
echo $2
echo $3

If more arguments are provided, they are ignored. If only two are provided, then $3 is the empty string, and is echoed as such, creating a blank line. Note the appearance of $0. Also, note what happens to ./echo3.sh foo 'bar baz' quux

Next, consider this echoeach command:

#!/bin/bash

for i in $*
do
    echo $i
done

Try ./echoeach.sh foo bar baz.

A more interesting demo is ./echoeach foo 'bar baz' quux. What does it do that is wrong? It gets four arguments, which is not what we want. A fix is to use "$@", with the quotes. $@ is a bash array of all the arguments (never mind exactly what this is), and putting quotes on it quotes each element in turn. This technique of putting quotes on does not work with "$*".

You are strongly encouraged to use "$@" for the list/array of command-line arguments when the arguments might have embedded spaces.

There are a couple other shell argument variables. One is $#, which is the number of arguments in all, not including $0. It is very useful for determining if the right number of arguments have been supplied. See the "mycal" example in the case section.

You can use the command shift to pop off the first argument $1, and move $2 to $1, $3 to $2 etc. $# is updated. See the while section below.

bash if and boolean expressions

Suppose we want our script to check for some condition, and have the result of that check affect its further action. The bash if command helps here. We also need bash conditions. Any command can be used as a condition. Bash looks at the command's exit code, with true represented by an exit code of 0 (the normal case) and false represented by anything else. (Note that this is the reverse of the C convention.) We will start with the test command, and the equivalent [ command:

    test expression

    [  expression  ]

(These are not exactly equivalent, in that [ checks if there is a matching ] at the end, while test does not.)

There is a newer version of test, [[   ]], that also includes support for regular-expression matching.

Here are some file tests:

    test -f file    # file is a regular file

    test -d file    # file is a directory

    test -e file    # file exists; useful for wildcard matching

    test -L file    # file is a symlink

    test -r file    # file is readable (also -w for writable)

    test file1 -nt file2    # file1 is newer than file2, in terms of the file-modification date

Here is nodirs.sh, which checks if there are no subdirectories of the current directory. It also uses the return code to signal this.

for i in *
do
    # echo $i
    if test -d $i
    then
        echo "directories found"
        exit 1
    fi
done

echo "no directories"
exit 0

Here are some string tests.

-n string        The length of string is greater than zero.
-z string        The length of string is zero.
string1 = string2
string1 == string2        string1 and string2 are equal. Single or double equal signs may be used. Most people use ==, but it is not Posix.
string1 != string2        string1 and string2 are not equal.
string1 > string2        string1 sorts after string2.  Warning: > must be escaped from the shell
string1 < string2        string1 sorts before string2.   Warning: < must be escaped from the shell


Shotts answer.sh, modified to take a command-line argument


Here are some numeric tests. These only work for integers

    integer1 -eq integer2            integer1 is equal to integer2.
    integer1 -ne integer2            integer1 is not equal to integer2.
    integer1 -le integer2             integer1 is less than or equal to integer2.
    integer1 -lt integer2              integer1 is less than integer2.
    integer1 -ge integer2            integer1 is greater than or equal to integer2.
    integer1 -gt integer2             integer1 is greater than integer2.

((  )) tests for numeric zero

You can make Boolean combinations with -a for and, -o for or, and ! for not.

Command Substitution

Sometimes we want to make use of the output of a subcommand. We us a special syntax that converts the output of the command to a string, which can be tested in the parent script. Just enclose the subcommand in $(    ). This is traditionally called "command substitution".

Example:

    ls -l $(which dash)

This gets "ls -l" information about the command

The expr command evaluates an arithmetic expression (where operands have to be separated from numbers by spaces, and '*' must be escaped from the shell to avoid globbing). If we create a shell "loop", here is how we can increment a variable:

    i=0

    i=$(expr $i + 1)

We can also use the double-parentheses trick, i=$(($i + 1)). Note the inner $ is needed to get the value of i, and the outer one is part of the $((  )) construct.

The cut command is useful for extracting particular fields from the output of another command. It is possible to use it to extract a set of columns by column numbers, thus allowing the parsing of the output of ls, but the use of cut is easier for output that is essentially in "csv" (comma-separated values") format (the separator does not have to be a comma).

Linux has a builtin command basename that takes a filename like /usr/bin/dircmp and returns the "base" part, "dircmp". What if we want to remove the "extension" from a file; that is, convert foo.text or foo.pdf to just foo? We'll treat the string as two fields with separator '.', and use cut to get the first field. Note that fields in cut are numbered starting with 1, not 0.

    echo foo.pdf | cut -d. -f1

Or, if the filename is in $file and we want to set the "root" filename to the variable rootname,

    rootname=$(echo $file | cut -d. -f1)

What does this do to foo.bar.baz?

There's a fancier, if somewhat more incomprehensible, way:

    echo "${file%.*}"

From the manual

${parameter%%word}

The word is expanded to produce a pattern [here .*] and matched according to the rules described below. If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the value of parameter with the shortest matching pattern deleted.

On my laptop, I often prefer to disable the touchpad quickly from the keyboard. I have a command that works like the list below. I'm trying to extract the number in the string "id=12", so I first use cut with the separator =, and then with the default TAB separator.

DEVNAME='Synaptics TM3625-010'

TOUCHPADNUM=$(xinput list | grep "$DEVNAME" | cut -d= -f2 | cut -f1)

echo "TOUCHPADNUM = $TOUCHPADNUM   ARG=$ARG"

xinput --disable $TOUCHPADNUM

ifaddrs

#!/bin/bash
# lists address and length for given interface

set -eu

ip addr show  | while read -r line
do
   if [[ "$line" =~ 'inet ' ]]        # a match
   then
       ippair=$(echo $line | cut -d' ' -f2)
       addr=$(echo $ippair|cut -d/ -f1)
       len=$(echo $ippair|cut -d/ -f2)
       echo $addr $len
   fi  
done
exit 0


Bash Control Structures

Here is a summary of the four main control structures; there is more detail further on.

    if

if test ! -e $filename                    # note where the ! goes, for not
then
    echo "file $filename not found"
fi

    case

case $MONTH in
    january)                    # no quotation marks
        MONTH=01;;              # you can also put the ;; on a line by itself
    february)
        MONTH=02;;
    *)                          # these are actually patterns; * matches anything
        MONTH=13;;
esac

    for

for i in *.text
do
    if grep --silent hello $i
    then
        echo "$i says hello"
    fi
done

    while

i=0
sum=0
while test $sum -lt 1000000
do
    i=$(( $i + 1))
    sum=$(($sum+$i))
done
echo $i

This adds consecutive numbers until the sum is greater than a million, and prints how many were needed.


Comments

A comment line begins with #. Such lines are "invisible" to bash, and so can not be the entire body of an if statement. Use the empty statement (a colon) for that:

    if [ -e $file ]
    then
        :
    else
        echo "file not found"
    fi

This is another way to write

    if [ ! -e $file ]
    then
        echo "file not found"
    fi

It is handy for those of us who can't get the ! syntax correct.

Basic settings

I recommend always including these at the start.

set -eu
set -o pipefail

The set built-in command lets you control the bash environment. set -e makes your script exit immediately if one stage of a pipeline fails. This is usually what you want. Similarly, the option setting set -o pipefail makes sure that pipeline error codes get returned properly.

set -u causes an error if you try to use an unset (ie "undeclared") variable. You should always use this.


Useful Helpers

awk: this is the universal parser, but it's a programming language in its own right.

cut: cuts out part of a line. Lines can be split into fields by any delimiter

tr: translates one set of characters to another.

bash case

My example will be mycal, below, which converts written-out months to numeric form for the cal command (actually, cal nowadays does take written-out months)

#!/bin/bash
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
# mydate: convert 'oct 24' to '10 2024' for the cal command
# Actually, modern versions of cal do accept the month as "jan" or "january", etc.
# Try the month 9 1752

DEBUG=1

set -eu
set -o pipefail

if [ $# -ne 2 ]
then
    echo "usage: mycal month year"
    exit 1
fi

MONTH=$1
YEAR=$2

if test $YEAR -lt 100        # two-digit year
then
    YEAR=$(($YEAR + 2000))
fi

MONTH=$(echo $MONTH | tr '[A-Z]' '[a-z]')    # convert to lowercase; escape char classes from shell
if test $DEBUG -eq 1;then echo "MONTH is $MONTH"; fi

if [[ $MONTH =~ ^[a-z] ]]    # if MONTH starts with a letter; this is a regular expression
then
case $MONTH in
    january|jan)
        MONTH=01;;       
    february|feb)
        MONTH=02;;
    march|mar)
        MONTH=03;;
    april|apr)
        MONTH=04;;
    may)
        MONTH=05;;
    june|jun)
        MONTH=06;;
    july|jul)
        MONTH=07;;
    august|aug)
        MONTH=08;;
    september|sep|sept)
        MONTH=09;;
    october|oct)
        MONTH=10;;
    november|nov)
        MONTH=11;;
    december|dec)
        MONTH=12;;       
    *)            # this is a pattern match; "*" matches everything
        echo "$MONTH is not a month"
        exit 1;;
esac
fi

if test $DEBUG -eq 1;then echo "MONTH is $MONTH, YEAR is $YEAR"; fi
 
cal $MONTH $YEAR

Some things to note here:

bash for

This is the workhorse of bash loops. Typically we are interested in looping over a set of globbed filenames:

for i in *.text
do
    if grep --silent hello $i
    then
        echo "$i says hello"
    fi
done

If a .text file has the string "hello" in it, it is included in the output.

Note the use of grep as a test condition for if; there is no test command here! Also note the --silent.

bash while

The opening example was to find the first N such that 1+2+...+N >= 1,000,000. This can't in general be done with for loops. One good use of while is when reading from a file. Here is linecounter:

if [ $# -ne 1 ]
then
        echo "usage: linecounter filename"
        exit 1
else
        FILENAME=$1
fi

echo "FILENAME is $FILENAME"

NUMLINES=0

while read x
do
    NUMLINES=$(($NUMLINES + 1))
    echo "$NUMLINES: $x"
done < $FILENAME

echo $NUMLINES

Note the argument-count check ($#). Also note that we're reading each line into a shell variable x, but not using x at all.

Here's another use of while to print all the arguments, like echoeach.sh above:

#!/bin/bash
while test $# -gt 0
do
    echo $1
    shift
done

After each shift, the first command-line argument gets popped, and all the others move up a notch (so $2 becomes $1, etc). $# is also updated.

Bash Variable Scope

In Java, a declared local variable "shadows" a global variable of the same name. In Python, you can use the value of a global variable in a function, without declaration, but if you assign to a global variable, you have to have already declared that variable to be "global" for it to work.

What about bash?

There aren't really local variables, or functions for that matter. There are no variable declarations at all. But the following attempt to count lines fails very badly. Why?

NUMLINES=0

cat $FILENAME | while read x
do
    NUMLINES=$(($NUMLINES + 1))
done
echo NUMLINES is $NUMLINES

It returns zero! The problem is that we are reading the file and piping it into the while loop, so the while loop can increment for each line. But the pipe means that the while loop will be executing in a separate process. When NUMLINES is updated in the while loop, it has no effect on the NUMLINES echoed at the end. This kind of thing can be very frustrating to deal with.

One fix is to use input redirection, instead of a pipe.

A handy rule is to beware of pipes in shell scripts, particularly if variable updates are used.

Options Processing for bash scripts

Let's start with this script that reads a file (the filename is provided as the single argument), converts the entire file to lowercase, and writes to stdout. It has no options itself, but we'll use it in something else that does. The script works by reading in the input file one line at a time, and using the tr command to convert that line to lowercase, and then writing to stdout.

#!/bin/bash
set -eu
set -o pipefail

if [ $# -ne 2 ]
then
    FILENAME=$1
fi

while read THELINE
do
    echo $THELINE | tr A-Z a-z        # convert THELINE to lowercase, and write it to stdout
done    < $FILENAME

Now we want to embed this in another script, lcbunch, that converts all the regular files listed on the command line as file arguments. lcbunch also has the following options, supplied as command-line "option arguments":

    -v                    print the name of each file converted on stdout
    -e extension    add the supplied extension to each new file created
    -d dirname     put the converted files in the given directory (which must already exist)

The user must supply either -e or -d. Both can be supplied. All options must come before any files. As an example,

    lcbunch -v -d mydir *.text

converts all the .text files and puts the converted files into mydir.

We will do this two ways, first using bare-bones bash, and the shift command, and then with getopts, which automates this kind of option processing.

Here is the bare-bash version:

#!/bin/bash
#    -v                    print the name of each file converted on stdout
#    -e extension    add the supplied extension to each new file created
#    -d dirname     put the converted files in the given directory (which must already exist)

set -eu
set -o pipefail

OPTDONE=0
VERBOSE=0
OUTDIR="."
EXTENSION=""
USAGE="usage: lcbunch [-v] [-e extension] [-d directory] files"

while test $OPTDONE -eq 0 -a $# -ge 1            # options processing is done in this loop
do
    case $1 in
       -v)
           # echo "setting VERBOSE"
           VERBOSE=1
           shift    # for the -v
           ;;
       -e)
           EXTENSION=$2
           # echo "setting EXTENSION to $EXTENSION"
           shift    # for the -e
           shift    # for the option value following -e
           ;;
       -d)
           OUTDIR=$2
           shift    # for the -d
           shift    # for the option value following -d
           ;;
       *)
           OPTDONE=1
           ;;
    esac
done

if test -n "$EXTENSION"
then
    EXTENSION='.'$EXTENSION        # add the dot
fi

# check if EXTENSION or OUTDIR was supplied
if test -z "$EXTENSION" -a "$OUTDIR" == "."    # neither changed
then
    echo "Must supply -e or -d"
    echo $USAGE
    exit 2
fi

# At this point, $* just consists of files to be converted
while test $# -gt 0
do
    INFILE=$1
    if test -f $INFILE                # make sure it's a regular file
    then
        OUTFILE=$OUTDIR/${INFILE}${EXTENSION}    # curly braces to avoid running into one another
        if test $VERBOSE -eq 1; then echo "converting $INFILE" to $OUTFILE; fi
        lcasecopy $INFILE > $OUTFILE        # copy this file
    fi
    shift                    # next file
done

The first while loop does the options processing. We work through the $* string of all arguments (with shift), after setting shell variables for all the option defalts (VERBOSE, EXTENSION and OUTDIR). If the first argument is -v, we set VERBOSE, and shift. Now $1 is the next argument, and we go around the while loop again.

Likewise, if the first argument is -e, we set EXTENSION to $2. We then shift twice, to pop both those values off of $*. Similarly for -d.

As long as we have options, either -v alone or an -e or a -d with something following, we keep going. Each time we start the while loop, if there are any more options then $1 must be -v, -e or -d. When we are done with the options, and get to the files, the case *) matches, and we set OPTDONE=1. This leads to our exiting from the loop, and moving on to the file-processing section.

After the while loop, we add a "." to EXTENSION, if necessary, and check that either EXTENSION or OUTDIR is different (otherwise we'll be writing files in place, which is bad).

At this point we've popped all the options from $*, and all that's left is the files to convert. We handle them one at a time, shifting after each one. We're done when the argument count, $#, reaches zero.

Note that we redirect the stdout of lcasecopy into the filename $OUTFILE

The getopts version

The only thing that changes is the first loop. It is now

while getopts "ve:d:" opt
do
    case $opt in
       v)
           echo "setting VERBOSE"
           VERBOSE=1
           ;;
       e)
           EXTENSION=$OPTARG            # OPTARG is set by getopts
           echo "setting EXTENSION to $EXTENSION"
           ;;
       d)
           OUTDIR=$OPTARG
           ;;
       *)
           echo $USAGE
           exit 1
           ;;
    esac
done

shift $(($OPTIND-1))

getopts is built into bash. The first string is the list of option letters, and whether something must follow them. "ve:d:" means that the options are -v, -e and -d, and the latter two have an argument following them (as indicated by the colon). The opt is a shell variable name of our choosing.

getopts also sets the two special variables OPTARG and OPTIND. OPTARG is the option argument, if one is expected; it plays the role of $2 in the first version. OPTIND is how many places in $* we have advanced.

We do not do a shift after each argument. Instead we do it all at the end, with shift $(($OPTIND-1)).

We also omit the hyphen before the option letters: they are v, e and d, not -v, -e and -d.


bash debugging

The first and most basic technique is to print out key variables at selected points, with

    echo $myvar

or, better yet,

    echo "filename is $myvar"

or something else that will help you understand what you're looking at.

It is helpful to have these print out only if some variable, eg DEBUG, is set to 1:

DEBUG=1

if test $DEBUG -eq 1; then echo "filename is $myvar"; fi

debug mode

There is also debug mode, enabled by running your script as

    bash -x myscript.sh

Debug mode prints out

As an example let's look at numsum:

#!/bin/bash
# adds up the integers from 1 to $1

NUM=$1
SUM=0
while test $NUM -ne 0
do
    SUM=$(expr $SUM + $NUM)
    NUM=$(( $NUM - 1))        #alternative way to do arithmetic
done
echo $SUM

The output of bash -x numsum 4 looks like this (some comments added).

+ NUM=4
+ SUM=0
+ test 4 -ne 0            # first time through the loop
++ expr 0 + 4
+ SUM=4
+ NUM=3
+ test 3 -ne 0            # second time through the loop
++ expr 4 + 3
+ SUM=7
+ NUM=2
+ test 2 -ne 0            # third time through the loop
++ expr 7 + 2
+ SUM=9
+ NUM=1
+ test 1 -ne 0            # fourth time through the loop
++ expr 9 + 1
+ SUM=10
+ NUM=0
+ test 0 -ne 0
+ echo 10
10

The lines beginning with ++ show external commands (not built-ins) that are invoked. In this case, that's expr.

Recall the example from earlier of linecounter where I piped the file into the while loop, instead of redirected the input. Use of debug mode let me see that the NUMLINES variable was being regularly incremented, only to revert to 0 at the end. From that, and with a little help from StackExchange, I was able to figure out that pipes live in subshells, and subshell variables have no connection to parent-shell variables.

You can turn debug mode on and off from within your script dynamically with

    set  -x    # enable debugging

    set +x    # disable debugging

Note that minus, "-", is used to add debugging, and plus, "+", is used to take away debugging.

enabling syntax highlighting in your text editor can also help here. In vi/vim, you do that with :syntax on, in command mode.

Syntax errors

Here are a few common ones:

Note that leaving off a do or then can be confusing, since bash does allow multiple commands as part of if/while tests, and the do/then marks the end of this list.

Shotts has a good section on this in Chapter 30.

Expansion problems

Recall that bash does line expansion whenever a command (even a built-in) is executed. This exposes you to issues with file-name globbing and variable expansion. The Shotts example is this:

number=            # empty string!
if test $number = 1
then
    echo something
fi

We should have written if test "$number" = 1, with quotes, but we didn't. So the second line expands to if test = 1, which is an illegal test expression.

Expansion problems in bash are legion. Watch out!

If we create a file with the above (so that the if test $number = 1 is line 7) and run it, we get

    line 7: test: =: unary operator expected

That's mysterious, since looking at the code we're clearly using = as a binary operator. But here it is with bash -x:

+ number=
+ test = 1
trouble: line 7: test: =: unary operator expected

Now it is clear that the arguments to test are = 1, and indeed something is missing.

If we put "" around $number (as we should), the original script runs fine (though as written it produces no output, because "" is not equal to 1, and there is no else clause).

A couple more bash-specific issues are: