July 3
Shotts:
Many of the bash scripts used in this lecture are here.
Some things we'll cover:
This is an overview of the basics of shell scripting
Control structures: expressions and test
strings
files
numbers
if test
! -e $filename
# note where the ! goes, for not
then
echo "file $filename not found"
fi
case
$MONTH in
january)
# no quotation
marks
MONTH=01;; # you can also put the ;;
on a line by itself
february)
MONTH=02;;
*)
# these are actually
patterns; * matches anything
MONTH=13;;
esac
for i in
*.text
do
if grep --silent hello $i
then
echo "$i says hello"
fi
done
Shell scripts can be given arguments on the command line. The first ten arguments are $0 through $9, with $0 being the command name itself. (But there is no $10, or anything higher!) The list of all the arguments is $* (though all the arguments in a bash array is $@, and is usually a better choice). The number of arguments is $#. Below is echo3.sh, which echoes arguments 0 through 3.
#!/bin/bash
# echo3.sh
echo $0
echo $1
echo $2
echo $3
Notice that in this script we are referencing arguments even if they are not supplied! (This will break with the set -u statement, which prohibits the use of undefined shell variables, even, say, $3.)
If more than three arguments are provided to echo3.sh, they are ignored. If only two are provided, then $3 is undefined, and is (without the set -u option) equivalent to the empty string, and is echoed as such, creating a blank line. Note the appearance of $0. Also, note what happens to ./echo3.sh foo 'bar baz' quux: argument two is the string 'bar baz'.
Next, consider this echoeach command:
#!/bin/bash
for i in $*
do
echo $i
done
Try ./echoeach.sh foo bar baz. Try it also with fewer arguments, or more. Note that we are not accessing any undefined arguments.
A more interesting demo is ./echoeach foo 'bar baz' quux. What does it do that is wrong? It gets four arguments, which is not what we want. A fix is to use "$@", with the quotes. $@ is a bash array of all the arguments (never mind exactly what this is), and putting quotes on it quotes each element in turn. This technique of putting quotes on does not work with "$*".
You are strongly encouraged to use "$@" for the list/array of command-line arguments when the arguments might have embedded spaces.
The number of arguments is $#, not including $0. It is very useful for determining if the right number of arguments have been supplied. See the "mycal" example in the case section.
You can use the command shift to pop off the first argument $1, and move $2 to $1, $3 to $2 etc. $# is updated. See the while section below.
Suppose we want our script to check for some condition, and have the result of that check affect its further action. The bash if command helps here. We also need bash conditions. Any command can be used as a condition. Bash looks at the command's exit code, with true represented by an exit code of 0 (the normal case) and false represented by anything else. (Note that this is the reverse of the C convention.) We will start with the test command, and the equivalent [ command:
test expression
[ expression ]
There is a newer version of test, [[ ]], that also includes support for regular-expression matching.
Here are some file tests. They mostly use one-letter dash options. Most of them are unary tests.
test -f file # file is a regular file
test -d file # file is a directory
test -e file # file exists; useful for wildcard matching
test -L file # file is a symlink
test -r file # file is readable (also -w for writable)
test file1 -nt file2 # file1 is newer than file2, in terms of the file-modification date
Here is nodirs.sh, which checks if there are no subdirectories of the current directory. It also uses the return code to signal this.
for i in
*
do
# echo $i
if test -d $i
then
echo "directories found"
exit 1
fi
done
echo "no directories"
exit 0
Here are some string tests.They use one-letter dash options (-n and -z) and operator symbols.
-n string
The length of string is greater than zero.
-z string The length of string is
zero.
string1 = string2
string1 == string2 string1 and
string2 are equal. Single or double equal signs may be used. Most people
use ==, but the official Posix standard is to use =.
string1 != string2 string1 and
string2 are not equal.
string1 > string2 string1 sorts
after string2. Warning: > must be escaped from the shell
string1 < string2 string1 sorts
before string2. Warning: < must be escaped from
the shell
Example: Shotts answer.sh, modified to take a command-line argument
#!/bin/bash
# test-string: evaluate the value of a string
ANSWER=$1
if [ -z "$ANSWER" ]; then
echo "There is no answer." >&2
exit 1
fi
if [ "$ANSWER" = "yes" ]; then
echo "The answer is YES."
elif [ "$ANSWER" = "no" ]; then
echo "The answer is NO."
elif [ "$ANSWER" = "maybe" ]; then
echo "The answer is MAYBE."
else
echo "The answer is UNKNOWN."
fi
Here are some numeric tests. These only work for integers. They use two-letter dash options.
integer1 -eq integer2
integer1 is equal to integer2.
integer1 -ne integer2
integer1 is not equal to integer2.
integer1 -le integer2
integer1 is less than or equal to
integer2.
integer1 -lt integer2
integer1 is less than integer2.
integer1 -ge integer2
integer1 is greater than or equal to
integer2.
integer1 -gt integer2
integer1 is greater than integer2.
(( )) tests for numeric zero
You can make Boolean combinations with -a for and, -o for or, and ! for not. The ! for not goes after the test command.
Any command can be an if condition; bash goes by the return code of the command. See the while section for an example
A comment line begins with #. Such lines are "invisible" to bash, and so can not be the entire body of an if statement. Use the empty statement (a colon) for that:
if [ -e $file ]
then
:
else
echo "file not found"
fi
This is another way to write
if [ ! -e $file ]
then
echo "file not found"
fi
It is handy for those of us who can't get the ! syntax correct.
I recommend always including these at the start.
set -eu
set -o pipefail
The set built-in command lets you control the bash environment. set -e makes your script exit immediately if one stage of a pipeline fails. This is usually what you want. Similarly, the option setting set -o pipefail makes sure that pipeline error codes get returned properly.
set -u causes an error if you try to use an unset (ie "undeclared") variable. You should always use this.
Sometimes we want to test the output of a subcommand. We us a special syntax that converts the output of the command to a string, which can be tested in the parent script. Just enclose the subcommand in $( ). This is traditionally called "command substitution".
Example:
ls -l $(which dash)
This gets "ls -l" information about the command
The expr command evaluates an arithmetic expression (where operands have to be separated from numbers by spaces, and '*' must be escaped from the shell to avoid globbing). If we create a shell "loop", here is how we can increment a variable:
i=0
i=$(expr $i + 1)
We can also use the double-parentheses trick, i=$(($i + 1)). Then the '*' does not need to be escaped, and you don't need spaces around each token, so this is better overall. Note the inner $ is needed to get the value of i, and the outer one is part of the $(( )) construct.
The cut command is useful for extracting particular fields from the output of another command. It is possible to use it to extract a set of columns by column numbers, thus allowing the parsing of the output of ls, but the use of cut is easier for output that is essentially in "csv" (comma-separated values") format (the separator does not have to be a comma).
Linux has a builtin command basename that takes a filename like /usr/bin/dircmp and returns the "base" part, "dircmp". What if we want to remove the "extension" from a file; that is, convert foo.text or foo.pdf to just foo? We'll treat the string as two fields with separator '.', and use cut to get the first field. Note that fields in cut are numbered starting with 1, not 0.
echo foo.pdf | cut -d. -f1
Or, if the filename is in $file and we want to set the "root" filename to the variable rootname,
rootname=$(echo $file | cut -d. -f1)
What does this do to foo.bar.baz?
There's a fancier way:
echo "${file%.*}"
From the manual
${parameter%%word}The word is expanded to produce a pattern [here .*] and matched according to the rules described below. If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the value of parameter with the shortest matching pattern deleted.
awk: this is the universal parser, but it's a programming language in its own right.
cut: cuts out part of a line. Lines can be split into fields by any delimiter
tr: translates one set of characters to another.
My example will be mycal, below, which converts written-out months to numeric form for the cal command (actually, cal nowadays does take written-out months)
#!/bin/bash
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
# mydate: convert 'oct 24' to '10 2024' for the cal command
# Actually, modern versions of cal do accept the month as "jan" or
"january", etc.
# Try the month 9 1752
DEBUG=1
set -eu
set -o pipefail
if [ $# -ne 2 ]
then
echo "usage: mycal month year"
exit 1
fi
MONTH=$1
YEAR=$2
if test $YEAR -lt 100 # two-digit
year
then
YEAR=$(($YEAR + 2000))
fi
MONTH=$(echo $MONTH | tr '[A-Z]' '[a-z]') # convert to
lowercase; escape char classes from shell
if test $DEBUG -eq 1;then echo "MONTH is $MONTH"; fi
if [[ $MONTH =~ ^[a-z] ]] # if MONTH starts with a
letter; this is a regular expression
then
case $MONTH in
january|jan)
MONTH=01;;
february|feb)
MONTH=02;;
march|mar)
MONTH=03;;
april|apr)
MONTH=04;;
may)
MONTH=05;;
june|jun)
MONTH=06;;
july|jul)
MONTH=07;;
august|aug)
MONTH=08;;
september|sep|sept)
MONTH=09;;
october|oct)
MONTH=10;;
november|nov)
MONTH=11;;
december|dec)
MONTH=12;;
*)
# this is a pattern match; "*" matches everything
echo "$MONTH is not a month"
exit 1;;
esac
fi
if test $DEBUG -eq 1;then echo "MONTH is $MONTH, YEAR is $YEAR"; fi
cal $MONTH $YEAR
Some things to note here:
This is the workhorse of bash loops. Typically we are interested in looping over a set of globbed filenames:
for i in
*.text
do
if grep --silent hello $i
then
echo "$i says hello"
fi
done
If a .text file has the string "hello" in it, it is included in the output.
Note the use of grep as a test condition for if; there is no test command here! Also note the --silent.
The opening example was to find the first N such that 1+2+...+N >= 1,000,000. This can't in general be done with for loops. One good use of while is when reading from a file. Here is linecounter:
if [ $#
-ne 1 ]
then
echo "usage: linecounter
filename"
exit 1
else
FILENAME=$1
fi
echo "FILENAME is $FILENAME"
NUMLINES=0
while read x
do
NUMLINES=$(($NUMLINES + 1))
echo "$NUMLINES: $x"
done < $FILENAME
echo $NUMLINES
Note the argument-count check ($#). Also note that we're reading each line into a shell variable x, but not using x at all.
Here's another use of while to print all the arguments, like echoeach.sh above:
#!/bin/bash
while test $# -gt 0
do
echo $1
shift
done
After each shift, the first command-line argument gets popped, and all the others move up a notch (so $2 becomes $1, etc). $# is also updated.
In Java, a declared local variable "shadows" a global variable of the same name. In Python, you can use the value of a global variable in a function, without declaration, but if you assign to a global variable, you have to have already declared that variable to be "global" for it to work.
What about bash?
There aren't really local variables, or functions for that matter. There are no variable declarations at all. But the following attempt to count lines fails very badly. Why?
NUMLINES=0
cat $FILENAME | while read x
do
NUMLINES=$(($NUMLINES + 1))
done
echo NUMLINES is $NUMLINES
It returns zero! The problem is that we are reading the file and piping it into the while loop, so the while loop can increment for each line. But the pipe means that the while loop will be executing in a separate process. When NUMLINES is updated in the while loop, it has no effect on the NUMLINES echoed at the end. This kind of thing can be very frustrating to deal with.
One fix is to use input redirection, instead of a pipe.
A handy rule is to beware of pipes in shell scripts, particularly if variable updates are used.
Let's start with this script that reads a file (the filename is provided as the single argument), converts the entire file to lowercase, and writes to stdout. It has no options itself, but we'll use it in something else that does. The script works by reading in the input file one line at a time, and using the tr command to convert that line to lowercase, and then writing to stdout.
#!/bin/bash
set -eu
set -o pipefail
if [ $# -ne 2 ]
then
FILENAME=$1
fi
while read THELINE
do
echo $THELINE | tr A-Z a-z
# convert THELINE to lowercase, and write it to
stdout
done < $FILENAME
Now we want to embed this in another script, lcbunch, that converts all the regular files listed on the command line as file arguments. lcbunch also has the following options, supplied as command-line "option arguments":
-v
print the
name of each file converted on stdout
-e extension add the
supplied extension to each new file created
-d dirname put the
converted files in the given directory (which must already exist)
The user must supply either -e or -d. Both can be supplied. All options must come before any files. As an example,
lcbunch -v -d mydir *.text
converts all the .text files and puts the converted files into mydir.
We will do this two ways, first using bare-bones bash, and the shift command, and then with getopts, which automates this kind of option processing.
Here is the bare-bash version:
#!/bin/bash
#
-v
print the name of each file converted on stdout
# -e extension add the supplied
extension to each new file created
# -d dirname put the converted
files in the given directory (which must already exist)
set -eu
set -o pipefail
OPTDONE=0
VERBOSE=0
OUTDIR="."
EXTENSION=""
USAGE="usage: lcbunch [-v] [-e extension] [-d directory] files"
while test $OPTDONE -eq 0 -a $# -ge 1
# options processing is done in
this loop
do
case $1 in
-v)
# echo
"setting VERBOSE"
VERBOSE=1
shift # for the -v
;;
-e)
EXTENSION=$2
# echo
"setting EXTENSION to $EXTENSION"
shift # for the -e
shift # for the option value following -e
;;
-d)
OUTDIR=$2
shift # for the -d
shift # for the option value following -d
;;
*)
OPTDONE=1
;;
esac
done
if test -n "$EXTENSION"
then
EXTENSION='.'$EXTENSION
# add the dot
fi
# check if EXTENSION or OUTDIR was supplied
if test -z "$EXTENSION" -a "$OUTDIR" == "." # neither
changed
then
echo "Must supply -e or -d"
echo $USAGE
exit 2
fi
# At this point, $* just consists of files to be converted
while test $# -gt 0
do
INFILE=$1
if test -f $INFILE
# make sure
it's a regular file
then
OUTFILE=$OUTDIR/${INFILE}${EXTENSION} # curly braces
to avoid running into one another
if test $VERBOSE -eq 1; then
echo "converting $INFILE" to $OUTFILE; fi
lcasecopy $INFILE >
$OUTFILE # copy this file
fi
shift
# next file
done
The first while loop does the options processing. We work through the $* string of all arguments (with shift), after setting shell variables for all the option defalts (VERBOSE, EXTENSION and OUTDIR). If the first argument is -v, we set VERBOSE, and shift. Now $1 is the next argument, and we go around the while loop again.
Likewise, if the first argument is -e, we set EXTENSION to $2. We then shift twice, to pop both those values off of $*. Similarly for -d.
As long as we have options, either -v alone or an -e or a -d with something following, we keep going. Each time we start the while loop, if there are any more options then $1 must be -v, -e or -d. When we are done with the options, and get to the files, the case *) matches, and we set OPTDONE=1. This leads to our exiting from the loop, and moving on to the file-processing section.
After the while loop, we add a "." to EXTENSION, if necessary, and check that either EXTENSION or OUTDIR is different (otherwise we'll be writing files in place, which is bad).
At this point we've popped all the options from $*, and all that's left is the files to convert. We handle them one at a time, shifting after each one. We're done when the argument count, $#, reaches zero.
Note that we redirect the stdout of lcasecopy into the filename $OUTFILE
The only thing that changes is the first loop. It is now
while
getopts "ve:d:" opt
do
case $opt in
v)
echo
"setting VERBOSE"
VERBOSE=1
;;
e)
EXTENSION=$OPTARG
# OPTARG is set by getopts
echo
"setting EXTENSION to $EXTENSION"
;;
d)
OUTDIR=$OPTARG
;;
*)
echo $USAGE
exit 1
;;
esac
done
shift $(($OPTIND-1))
getopts is built into bash. The first string is the list of option letters, and whether something must follow them. "ve:d:" means that the options are -v, -e and -d, and the latter two have an argument following them (as indicated by the colon). The opt is a shell variable name of our choosing.
getopts also sets the two special variables OPTARG and OPTIND. OPTARG is the option argument, if one is expected; it plays the role of $2 in the first version. OPTIND is how many places in $* we have advanced.
We do not do a shift after each argument. Instead we do it all at the end, with shift $(($OPTIND-1)).
We also omit the hyphen before the option letters: they are v, e and d, not -v, -e and -d.
The first and most basic technique is to print out key variables at selected points, with
echo $myvar
or, better yet,
echo "filename is $myvar"
or something else that will help you understand what you're looking at.
It is helpful to have these print out only if some variable, eg DEBUG, is set to 1:
DEBUG=1
if test $DEBUG -eq 1; then echo "filename is $myvar"; fi
There is also debug mode, enabled by running your script as
bash -x myscript.sh
Debug mode prints out
As an example let's look at numsum:
#!/bin/bash
# adds up the integers from 1 to $1
NUM=$1
SUM=0
while test $NUM -ne 0
do
SUM=$(expr $SUM + $NUM)
NUM=$(( $NUM - 1))
#alternative way to do arithmetic
done
echo $SUM
The output of bash -x numsum 4 looks like this (some comments added).
+ NUM=4
+ SUM=0
+ test 4 -ne 0 #
first time through the loop
++ expr 0 + 4
+ SUM=4
+ NUM=3
+ test 3 -ne 0 #
second time through the loop
++ expr 4 + 3
+ SUM=7
+ NUM=2
+ test 2 -ne 0 #
third time through the loop
++ expr 7 + 2
+ SUM=9
+ NUM=1
+ test 1 -ne 0 #
fourth time through the loop
++ expr 9 + 1
+ SUM=10
+ NUM=0
+ test 0 -ne 0
+ echo 10
10
The lines beginning with ++ show external commands (not built-ins) that are invoked. In this case, that's expr.
Recall the example from earlier of linecounter where I piped the file into the while loop, instead of redirected the input. Use of debug mode let me see that the NUMLINES variable was being regularly incremented, only to revert to 0 at the end. From that, and with a little help from StackExchange, I was able to figure out that pipes live in subshells, and subshell variables have no connection to parent-shell variables.
You can turn debug mode on and off from within your script dynamically with
set -x # enable debugging
set +x # disable debugging
Note that minus, "-", is used to add debugging, and plus, "+", is used to take away debugging.
enabling syntax highlighting in your text editor can also help here. In vi/vim, you do that with :syntax on, in command mode.
Here are a few common ones:
Note that leaving off a do or then can be confusing, since bash does allow multiple commands as part of if/while tests, and the do/then marks the end of this list.
Shotts has a good section on this in Chapter 30.
Recall that bash does line expansion whenever a command (even a built-in) is executed. This exposes you to issues with file-name globbing and variable expansion. The Shotts example is this:
number=
# empty string!
if test $number = 1
then
echo something
fi
We should have written if test "$number" = 1, with quotes, but we didn't. So the second line expands to if test = 1, which is an illegal test expression.
Expansion problems in bash are legion. Watch out!
If we create a file with the above (so that the if test $number = 1 is line 7) and run it, we get
line 7: test: =: unary operator expected
That's mysterious, since looking at the code we're clearly using = as a binary operator. But here it is with bash -x:
+ number=
+ test = 1
trouble: line 7: test: =: unary operator expected
Now it is clear that the arguments to test are = 1, and indeed something is missing.
If we put "" around $number (as we should), the original script runs fine (though as written it produces no output, because "" is not equal to 1, and there is no else clause).
A couple more bash-specific issues are: