July 25
ImageMagick
git clone
https://github.com/ImageMagick/ImageMagick.git (from within
~/141/now/imagemagick)
cd ImageMagick
./configure
make
The usual interface is via the magick command; see imagemagick.org/script/convert.php.
utilities/magick mountains.jpg mountains.png # simple example of image conversion
[Shotts chapter 14] Yes, you can compile packages. But it's a chore, and when there are compilation problems it's a real chore. (And the last time I compiled LibreOffice, in April 2025, it took 10 hours.)
There are many different distributions of Linux: Ubuntu, Debian, Red Hat, Mint, CentOS, Fedora, Gentoo .... One of the biggest differences between them is the style of package management, and to some extent the back-end maintenance of packages.
The high-level package tool on Debian-like distributions, which includes Ubuntu, is apt (or apt-get, an older version).
It's always a good idea to start with apt-get update, which updates the known list of package repositories. Some installations:
You control your own list of trusted repositories by editing /etc/apt/sources.list.
Most people find package names by googling, say, "how do I install the c compiler on ubuntu"
dpkg -l lists all installed packages. Note, however, that some packages will likely have been auto-installed in the process of installing some other package.
[Shotts Chapter 17] The unix find command is pretty handy, though it doesn't let you search "inside" complicated filetypes like .docx or .pdf. The syntax is:
find directory search-options
There may be no search options. For example, find ~ lists every file within your home directory. find ~ |wc -l counts them. find ~ -type d |wc -l counts your directories and subdirectories.
What if you want to find a file in or below the current directory by name, say "foo.text"? The most common usage pattern for find is find . -name 'foo.*' This will find any file below the current directory with filename of the form foo.*. Note that the quotes around foo.* are pretty important; otherwise, if there is a foo.c in the current directory then it will match, and you will only be looking for foo.c. We don't want shell filename expansion to get in the way here; we want the asterisks to be "expanded" by find.
For case-insensitive search, find . -iname '*foo*' works, where -iname is case-insensitive search (there is also -name for case-sensitive search).
Other useful options are -empty, -mtime, -newer, -perm mode, -type [d|f|...]. -maxdepth levels,
On page 229, Shotts describes an interesting technique for dealing with files with spaces in them. You use the -print0 option to find, which prints a null-separated list. You can then pipe this list into xargs, which repeats a given command on each element of the list of strings piped into it,. The example is
find ~ -iname '*.jpg' -print0 | xargs --null ls -l
This will run ls -l $file on each $file that is in the output of find. With the -print0 and --null (yes, one has a single hyphen and the other a double), the individual files are null-separated, and the filenames can have spaces in them without ambiguity.
(Note that, in the Unix world, the extension .jpeg, with the e, is more common.)
Then there is the -exec option. This runs a command on every file found. For example,
find . -type f -exec file {} ';'
The {} represents the name of the file in question, and the ';' marks the end of the file command and the return to find. it almost always has to be quoted so bash doesn't interpret it as an end-of-command indication. Next, let's search for the string 'key' in any files in files10/project
find . -exec grep -i key {} \;
We can't tell where the files are, and we have a lot of messages about directories there. How about this:
find . -type f -and -exec grep -i -H key {} \;
The -H makes grep always print the file name, and the -type f check prevents checking directories.
There are optimizations to do the inner command just once on the whole lot of files found.
[Shotts chapter 15] If I plug in my usb drive, where does it appear in the filesystem? On Windows, it would get a new drive letter, like G:. On Ubuntu, it's usually under /media/username.
This is achieved through the mount command (automatically in this case). A disk device has a filesystem on it, which is a tree of directories, and files. There is one "root" directoriy. The mount command attaches such a device to a specific directory on the existing filesystem.
The file /etc/fstab lists what gets mounted where, by default.
A typical physical disk usually has multiple partitions. Each partition has a filesystem. You can view these with fdisk, but you can also destroy your disk so be careful. Fdisk takes a parameter representing the device for the entire disk, eg /dev/nvme0n1.
Mount shows a lot of "virtual" mounts. But we can focus on the disk mounts with mount | grep nvme0n1.
As for filesystems, they can be a variety of types. NTFS is the most common windows filesystem. Linux filesystems include ext2, ext3, ext4, btrfs, xfs, and more. Part of the disk is set aside as an array of inodes, which contain the file's permissions. Directories are tables of pairs <filename, inode>. Linux shows you the inode numbers with ls -i.
Traditionally, an inode contains, in addition to the file's permission bits, some information about the location of the actual disk blocks:
The last two are the "indirect" block and the "double indirect" block. Sometimes there is a "triple indirect" block too. A double-indirect block allows 2000 * 2000 blocks, which is a 32 GB file.
The ext4 filesystem stores extents rather than each individual block number. An extent is a row of contiguous blocks, represented by the address of the first block and the number of blocks in the extent.
Newer filesystems (like ext4) also have some sort of "journal" file to allow disk recovery if the system crashes in mid-write.
Disk geometry: disk access is faster if the blocks of a file are on the same disk "track", and faster still if they are contiguous on that track.
[Shotts chapter 16] Systems have network devices. Most acquire their IP address on startup through the Dynamic Host Configuration Protocol, or dhcp.
Once we're connected, we have these:
Shotts p 210. This is the secure shell, a remote-login (and remote-command-execution) utility. Per-user files (like keys) are kept in ~/.ssh.
This is based on public-key authentication. If you, on host A, want to log in to host B, you connect and ask for some authentication data from B. Unless this is your first connection, you already have B's public key. B can send a message signed with B's private key, and A can validate it.
Initial public/private key pairs are created by ssh-keygen.
Previously received public keys are kept in the file known_hosts.
Typically, new hosts are assumed to be trusted. This is called Trust On First Use, or TOFU.
Once B has validated its identity to A (to prevent A from falling prey to sending its passwords to the wrong host), the user on A might log in the usual way, by providing a username and password. The problem here is that random password guessing is still possible. (This is why the Loyola CS dept no longer allows password-based ssh logins, except through the VPN.)
A better approach is to authenticate by pre-arrangement. A will place A's public key in B's file authorized_keys. This means that A is allowed to log in to B without providing a password. A must prove it has the matching private key by encrypting some message selected by B using its private key. A sends it back to B; if B can decrypt the message with A's public key, all is good.
Getting ones public key (typically in
RSA was the original algorithm. Elliptic-curve algorithms are much more popular, and shorter.
If any permissions on the .ssh directory or any of its files are not what they should be (for example, if the private key is readable by anyone other than the ownin user), then the connection fails.
[Shotts chapter 35] Arrays in bash are arrays of strings. This is quite practical. In a bash script, $* is the unquoted string of all arguments, and "$*" is the quoted string of all arguments. Neither is quite right. Here is an updated version of echoeach.sh (in files7):
#!/bin/bash
echo '$*'
for i in $*
do
echo $i
done
echo '"$*"'
for i in "$*"
do
echo $i
done
echo '"$@"'
for i in "$@"
do
echo $i
done
If we invoke ./echoeach.sh foo 'bar baz' quux,
I used to use $* all the time, but it fails miserably for arguments with spaces. Here is my current script word, which starts OpenOffice on a file:
PROG=/usr/bin/soffice
$PROG "$@" 2>/dev/null &
(the 2>/dev/null just makes the stderr messages go away. They are usually useless.)
[Shotts chapter 25] The apache webserver can be installed with "apt install apache2". We can then open a browser on the same machine, and go to "http://localhost". We get the default page.
How about adding another html page (eg foo.html)? It goes in /var/www/html.
How about adding a shell script? Like this:
#!/bin/bash
echo "<html>
<head><title>System Information
Page</title></head>
<body>
<h1>System Information Page</h1>
<p>
hostname: $HOSTNAME
</body>
</html>"
What we want is for the script to run, and have the output show up as the web page. Note that we've got a multiline quote here, and we've got a system bash variable (HOSTNAME) in the script.
We have to enable this in /etc/apache2/sites-enabled/default.conf, with some weird magic. We also have to put the script itself in /usr/lib/cgi-bin, not /var/www/html. Having websites run scripts, with possibly untrusted input, can be risky; we don't want anything unexpected to be runnable.
It still doesn't work. But there's a fix, that turns out to have to do with HTTP itself, not apache2.
Docker creates "isolated namespaces" within your operating system, usually called containers. Depending on configuration, a process running in a container may not be able to see processes outside the container, or files (you can specify what files are visible within the container), or network devices, etc. Containers can provide a great deal of isolation.
There are two primary reasons for this:
As for security, the application running inside the container cannot alter or read any files outside the container.
For installation convenience, application dependencies can be a nightmare. What if App A requires version 1.0 of a system-wide library, but App B requires version 2.0? This happens all the time, despite the general idea that later versions of a library are supposed to be backwards compatible with earlier versions. Docker allows bundling an application executable and a more-or-less minimal set of dependencies into a container. That container can then be encapsulated as a Docker "image", and installed at will.
Security is complicated, and verifying security is tedious and expensive. Much of the marketing of Docker is about the installation issue.
The ancestor of Docker is the early chroot() system call. Called within a directory, only files within that directory would be visible. Typically, if the parent directory were /usr/foo, then there would be a directory /usr/foo/bin, with links to executables; within the chroot environment this would appear to be /bin. Once chroot() was called, there was no way for that process or for any of its children to get back to a full view of the filesystem.
Chroot() suffered from at least two general classes of vulnerabilities. One was that someone might be able to create /usr/foo/etc/passwd, appearing within the container as /etc/passwd, and use a special entry in that file to elevate privileges from within the container. The second was that someone could run mknod from within the container to create a device file for the entire original hard disk. For example, my hard disk corresponds to the special-device files /dev/sda and /dev/sda1. If these files could be created using mknod from within the chroot environment, then the device could be "mounted" from within the chroot environment, and it's game over.
You can read more about running Docker at docs.docker.com/get-started.
We can run the "welcome to Docker" container with this:
docker run -d -p 8080:80 docker/welcome-to-docker
Now use any browser to go to localhost:8080.
The following command runs the firefox container created by jlesage, with access via port 5800 from the host system. That is, if you run
docker run -d --name=firefox -p 5800:5800 -v /docker/appdata/firefox:/config:rw jlesage/firefox
and then go with your non-containerized browser to localhost:5800, you will see the containerized firefox. You can have the non-containerized browser be, for example, Chrome.
Note that if I try to access the url file:///home/pld/141/now/index.html, it works in a non-containerized browser but fails in the container browser, which has no access to /home/pld.
Also, this accessing via a specific port is a bit of a hack. But it sure beats creating a Docker container with direct access to the framebuffer, /dev/fb0, or even the Xwindows DISPLAY
You can see your running docker containers with docker ps.
You can see your downloaded docker images with docker images, though this only applies to images downloaded the normal way.
You can (usually, again depending on configuration) run a shell in a docker container with docker exec. For the jlesage container above, right now the container id is 8c18a24aaaa3, so we can run a shell in it with
docker exec -it 8c18a24aaaa3 sh
(-i: interactive; keep STDIN open; -t: allocate a virtual "tty" serial line)
But look around: this is a very stripped-down environment. Check the number of files in /usr/bin! Or /usr/lib! Or /home! And check out /etc/passwd. In fact, bash is missing from the docker container; you must use sh here.
You stop a docker container with
docker stop docker_id
But this only "suspends" it. To remove it completely, you use
docker rm docker_id
(One container = multiple namespaces)
Ultimately, namespaces (and thus containers) are created through the clone() system call, which is a bit like fork() except that it has a flags parameter which is the OR of separate fixed flag values for each namespace type.
You can list all Linux namespaces on a system, of any type (not just the ones Docker uses) with lsns. Actually, individual namespaces all do have a type: one of net, pid, mnt, ipc, user, cgroup. Using the -t option, you can view the namespaces of that type (a Docker container will often consist of several namespaces, each providing a different form of isolation). On my laptop, I have these:
net 61
pid 37
mnt 22
ipc 55
user 61
cgroup 2
This accounts for all 238.
The first column of the lsns output (the NS column) is an inode number. Each namespace is associated with one. Files are what were originally pointed to by inodes, but namespace inodes have nothing to do with files. They were just an existing "thing" that could be pressed into service by kernel developers to identify these new namespace thingies.
Namespaces are also often identified by the PID of the process that started them (as seen in the outside process table; within the container this PID is almost always 1). lsns shows this pid in the PID column. And Docker containers have a DockerID value too, though lsns does not show it. But note that PIDs can change, and the starting process can create children and then exit. (It can also just exit, meaning we have a container with no processes, sort of a ghost, which is weird.)
For a given Docker container, let's find the namespaces it uses. Start with docker ps:
The DockerID here is a19b18935bc8. The next step is to find the Linux process for this container. We can run docker inspect -f '{{.State.Pid}}' a19b18935bc8 (note the "." before State). The output of docker inspect dockerID is a large JSON structure; with -f {{.State.Pid}} we are asking for the Pid attribute of the State attribute of this JSON structure. On my machine, I got 3553053.
Searching the main process table (with ps, outside the container), I get
root 3553053 0.0 0.0 8904 4992 ? Ss Jul25 0:00 nginx: master process nginx -g daemon off;
This is an nginx process, so so far all is good (nginx is a web server, and welcome-to-docker runs exactly that).
From within the container, with docker exec -it a19b18935bc8 sh, we find that this same nginx process has PID 1. (We can also get into this container with nsenter -t 3553053 -p -r sh, which is a sort of lower-level access mechanism.)
If we run "sleep 100 &" within the container (or "busybox sleep 100 &"; lots of processes sleep but likely there is only one busybox), we see one PID for this process within the container and another one in the main process table. That is, the namespace/container processes appear both within the namespace and also (with a different PID) outside. You can't hide a cryptominer in a container, in other words.
Now let's try to find the namespaces. For this we use the /proc filesystem, outside the container, which has all sorts of useful information about processes.
One thing we can do is look in /proc/3553053. The ns subdirectory contains links to inodes associated with the namespace. These are a bit tricky to do much with, however. For one thing, every process has a namespace of every type, but most of these will be the default, or global, namespace. Two nonglobal namespaces can have the same inode, if they were created together, or different inodes, if they were created sequentially. But two processes (in the main process table) with the same mnt inode, say, are definitely in the same namespace.
Let's start with
ls -l /proc/[1-9]*/ns
I have the output in a file inodes.text.
Step 1: let's find the default cgroup inode. Find process 1; the vi search pattern is /^1\/ns. The inode number is 4026531835.
Step 2: let's find all cgroup entries with a different inode number: cat inodes.text | grep cgroup | grep -v 4026531835.
Or there's this command, which extends the pipeline above to extract the inode numbers with cut and sort/uniq them:
grep cgroup inodes* | grep -v 4026531835 | cut -d[ -f2 | cut -d] -f1| sort | uniq
It turns out there are only two:
4026536528
4026536730
The ps command will take you from a PID to an inode number: ps -h -o pidns -p 3553053 would return the inode number for the pid namespace.
We can also, for each namespace type, say cgroup, run:
lsns -t cgroup
We get the above two inodes plus the default one.
If that process/container created a namespace of this type, this command should show a namespace instance tied to process 3553053.
We can also run
for nst in net pid mnt ipc user cgroup; do lsns -t $nst; done | grep 3553053
to get them all at once.