July 18
Shotts:
Coming up, probably during the week of July 21, there will be a series of quizzes on Sakai, each involving the writing of one bash script. You can do them all together, or one at a time. These will count like a midterm exam.
hello.c:
#include <stdio.h>
#include "hello.h"
void main() {
printf("hello, world!\n");
}
gcc hello.c
gcc -o hello hello.c
make and Makefile
hello: hello.c hello.h
gcc -o hello hello.c
What make actually does: touch hello.h
The existence of .h files make dependencies complicated in C. Make solves this problem.
The IDE vs make debate
pretty much all platforms
support make
it's straightforward to add complicated
testing or other alternative builds to Makefiles
The command line gives greater execution flexibility
than GUIs
On the other hand, Makefiles can be confusing
Makefile and leading tab
diction
download diction-1.11.tar.gz from
ftp.gnu.org/gnu/diction/
tar xzf diction-1.11.tar.gz
cd
./configure
make
now touch one of the source files and make
again
make install
signatures
gpg --import gnu-keyring.gpg
;; from ftp.gnu.org/gnu
gpg --verify diction-1.11.tar.gz.sig
The trust rabbit hole: WARNING: This key is not certified with a trusted signature!
configure options: Search for "Installation directory options"
ImageMagick
git clone
https://github.com/ImageMagick/ImageMagick.git
cd ImageMagick
./configure
make
Make and java: yes you can do it. but java files pretty much all compile independently. Still, make is helpful in that you can use it to recompile only the files you changed.
So far, everything you've done on your VM has been done as user "comp141". We can become the superuser, or root, if we know the root password. The command
su -
prompts for that password, and then we have a root shell (the hyphen argument above makes sure that we run all the login customization as root).
But if we don't have the root password, we can also use the sudo
command:
sudo bash
The password to type is your regular user password.
Warning: the usual advice these days is to use sudo to run individual commands, not to create a root shell. With the latter, one mistake and it's all over.
Being root lets you look at the log files in /var/log, for example. It also lets you add new users, reconfigure the network and install packages.
If you have a root password, the su command might be a better bet.
It is common to need to be root to install packages
[Shotts chapter 14] Yes, you can compile packages. But it's a chore, and when there are compilation problems it's a real chore.
There are many different distributions of Linux: Ubuntu, Debian, Red Hat, Mint, CentOS, Fedora, Gentoo .... One of the biggest differences between them is the style of package management, and to some extent the back-end maintenance of packages.
The high-level package tool on Debian-like distributions, which includes Ubuntu, is apt (or apt-get, an older version).
It's always a good idea to start with apt-get update, which updates the known list of package repositories. Some installations:
dpkg -l lists all installed packages. Note, however, that some packages will likely have been auto-installed in the process of installing some other package.
[Shotts Chapter 17] The unix find command is pretty handy, though it doesn't let you search "inside" complicated filetypes like .docx or .pdf. The syntax is:
find directory search-options
There may be no search options. For example, find ~ lists every file within your home directory. find ~ |wc -l counts them. find ~ -type d |wc -l counts your directories and subdirectories.
What if you want to find a file in or below the current directory by name, say "foo.text"? find . -iname '*foo*' works, where -iname is case-insensitive search (there is also -name for case-sensitive search). Note that you need the quote marks, if there is a match for *foo* in the current directory. We don't want shell filename expansion to get in the way here; we want the asterisks to be "expanded" by find.
Other useful options are -empty, -mtime, -newer, -perm mode, -type [d|f|...]. -maxdepth levels,
On page 229, Shotts describes an interesting technique for dealing with files with spaces in them. You use the -print0 option to find, which prints a null-separated list. You can then pipe this list into xargs, which repeats a given command on each element of the list of strings piped into it,. The example is
find ~ -iname '*.jpg' -print0 | xargs --null ls -l
This will run ls -l $file on each $file that is in the output of find. With the -print0 and --null (yes, one has a single hyphen and the other a double), the individual files are null-separated, and the filenames can have spaces in them without ambiguity.
(Note that, in the Unix world, the extension .jpeg, with the e, is more common.)
Then there is the -exec option. This runs a command on every file found. For example,
find . -type f -exec file {} ';'
The {} represents the name of the file in question, and the ';' marks the end of the file command and the return to find. it almost always has to be quoted so bash doesn't interpret it as an end-of-command indication. Next, let's search for the string 'key' in any files in files10/project
find . -exec grep -i key {} \;
We can't tell where the files are, and we have a lot of messages about directories there. How about this:
find . -type f -and -exec grep -i -H key {} \;
The -H makes grep always print the file name, and the -type f check prevents checking directories.
There are optimizations to do the inner command just once on the whole lot of files found.
[Shotts chapter 15] If I plug in my usb drive, where does it appear in the filesystem? On Windows, it would get a new drive letter, like G:. On Ubuntu, it's usually in /media/username.
This is achieved through the mount command (automatically in this case). A disk device has a filesystem on it, which is a tree of directories, and files. There is one "root" directoriy. The mount command attaches such a device to a specific directory on the existing filesystem.
The file /etc/fstab lists what gets mounted where, by default.
A typical physical disk usually has multiple partitions. Each partition has a filesystem. You can view these with fdisk, but you can also destroy your disk so be careful. Fdisk takes a parameter representing the device for the entire disk, eg /dev/nvme0n1.
Mount shows a lot of "virtual" mounts. But we can focus on the disk mounts with mount | grep nvme0n1.
As for filesystems, they can be a variety of types. NTFS is the most common windows filesystem. Linux filesystems include ext2, ext3, ext4, btrfs, xfs, and more. Part of the disk is set aside as an array of inodes, which contain the file's permissions. Directories are tables of pairs <filename, inode>. Linux shows you the inode numbers with ls -i.
[Shotts chapter 16] Systems have network devices. Most acquire their IP address on startup through the Dynamic Host Configuration Protocol, or dhcp.
Once we're connected, we have these:
Shotts p 210. This is the secure shell, a remote-login (and remote-command-execution) utility. Per-user files (like keys) are kept in ~/.ssh.
This is based on public-key authentication. If you, on host A, want to log in to host B, you connect and ask for some authentication data from B. Unless this is your first connection, you already have B's public key. B can send a message signed with B's private key, and A can validate it.
Initial public/private key pairs are created by ssh-keygen.
Previously received public keys are kept in the file known_hosts.
Typically, new hosts are assumed to be trusted. This is called Trust On First Use, or TOFU.
Once B has validated its identity to A (to prevent A from falling prey to sending its passwords to the wrong host), the user on A might log in the usual way, by providing a username and password. The problem here is that random password guessing is still possible. (This is why the Loyola CS dept no longer allows password-based ssh logins, except through the VPN.)
A better approach is to authenticate by pre-arrangement. A will place A's public key in B's file authorized_keys. This means that A is allowed to log in to B without providing a password. A must prove it has the matching private key by encrypting some message selected by B using its private key. A sends it back to B; if B can decrypt the message with A's public key, all is good.
Getting ones public key (typically in
RSA was the original algorithm. Elliptic-curve algorithms are much more popular, and shorter.
If any permissions on the .ssh directory or any of its files are not what they should be (for example, if the private key is readable by anyone other than the ownin user), then the connection fails.
[Shotts chapter 35] Arrays in bash are arrays of strings. This is quite practical. In a bash script, $* is the unquoted string of all arguments, and "$*" is the quoted string of all arguments. Neither is quite right. Here is an updated version of echoeach.sh (in files7):
#!/bin/bash
echo '$*'
for i in $*
do
echo $i
done
echo '"$*"'
for i in "$*"
do
echo $i
done
echo '"$@"'
for i in "$@"
do
echo $i
done
If we invoke ./echoeach.sh foo 'bar baz' quux,
I used to use $* all the time, but it fails miserably for arguments with spaces. Here is my current script word, which starts OpenOffice on a file:
PROG=/usr/bin/soffice
$PROG "$@" 2>/dev/null &
(the 2>/dev/null just makes the stderr messages go away. They are usually useless.)
[Shotts chapter 25] The apache webserver can be installed with "apt install apache2". We can then open a browser on the same machine, and go to "http://localhost". We get the default page.
How about adding another html page (eg foo.html)? It goes in /var/www/html.
How about adding a shell script? Like this:
#!/bin/bash
echo "<html>
<head><title>System Information
Page</title></head>
<body>
<h1>System Information Page</h1>
<p>
hostname: $HOSTNAME
</body>
</html>"
What we want is for the script to run, and have the output show up as the web page. Note that we've got a multiline quote here, and we've got a system bash variable (HOSTNAME) in the script.
We have to enable this in /etc/apache2/sites-enabled/default.conf, with some weird magic. We also have to put the script itself in /usr/lib/cgi-bin, not /var/www/html. Having websites run scripts, with possibly untrusted input, can be risky; we don't want anything unexpected to be runnable.
It still doesn't work. But there's a fix, that turns out to have to do with HTTP itself, not apache2.
Docker creates "isolated namespaces" within your operating system, usually called containers. Depending on configuration, a process running in a container may not be able to see processes outside the container, or files (you can specify what files are visible within the container), or network devices, etc. Containers can provide a great deal of isolation.
You can read more at docs.docker.com/get-started.
We can run the "welcome to Docker" container with this:
docker run -d -p 8080:80 docker/welcome-to-docker
Now use any browser to go to localhost:8080.
The following command runs the firefox container created by jlesage, with access via port 5800 from the host system. That is, if you run
docker run -d -name=firefox -p 5800:5800 -v /docker/appdata/firefox:/config:rw jlesage/firefox
and then go with your non-containerized browser to localhost:5800, you will see the containerized firefox. You can have the non-containerized browser be, for example, Chrome.
Note that if I try to access the url file:///home/pld/141/now/index.html, it works in a non-containerized browser but fails in the container browser, which has no access to /home/pld.
Also, this accessing via a specific port is a bit of a hack. But it sure beats creating a Docker container with direct access to the framebuffer, /dev/fb0, or even the Xwindows DISPLAY
You can see your running docker containers with docker ps.
You can see your downloaded docker images with docker images, though this only applies to images downloaded the normal way.
You can (usually, again depending on configuration) run a shell in a docker container with docker exec. For the jlesage container above, right now the container id is 8c18a24aaaa3, so we can run a shell in it with
docker exec -it 8c18a24aaaa3 sh
But look around: this is a very stripped-down environment. Check the number of files in /usr/bin! Or /usr/lib! Or /home! And check out /etc/passwd. In fact, bash is missing from the docker container; you must use sh here.
You stop a docker container with
docker stop docker_id