Dan Darg's Blog

Category: Uncategorized

  • Launching Subprocesses in Python3

    Motivation

    It’s important in the world of IT to know a scripting language for data processing, task automation, etc. For simple tasks (e.g. moving files, starting programs), my goto scripting language is Bash. But when I need something with more tools and precision (e.g. parsing html), I use python3.

    I decided in recent times to move towards coding in a more cross-platform manner, which basically means less bash, and more python. That meant that I needed to get more comfortable with python’s system of launching subprocesses. For years I’d been in the habit of copy/pasting code like this (which I probably grabbed from Stack Overflow originally), without really thinking through what was happening:

    import subprocess
# Launch some shell script CMD
p = subprocess.Popen(
    CMD,
    shell=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT
)
# Wait for it to finish
p.wait()
# Save the result to a str variable
result: str = p.stdout.read().decode('utf8')

    Not only is this a lot of code compared to the Bash equivalent (i.e. just $CMD), I also wasn’t very clear on whether all of these verbose arguments and methods were really needed (Spoiler: they’re not!), and this had been a mental block to using python3 more liberally in the past. So having actually now read the subprocess documentation, I figured I’d consolidate what I’d learnt there by trying to describe it all to a younger self.

    Standard Streams Recap

    Skip this section if you’re already well-versed with “Standard Streams” (stdin, stdout, and stderr) — you’ll need this to understand the above code block. If you are not well-versed with them then here is a super swift crash course (with focus on the two output streams: stdout, and stderr).

    What streams “are” is a bit abstract (think inner OS/kernel magic), and it’s easier to learn to think in terms of how you work with them. Let’s start by thinking in general terms about the sort of things one wants to be able to do when writing a program:

    • receive input information
    • send signals/data to other running processes
    • interact with devices (files in storage, graphics cards, network cards, etc.)
    • signal to developers/administrators the state and/or output of the program
    • start child processes (that can also do all of the above)

    The item in red is what output streams are all about, which will be our focus. When you write a program, you want to be able to get information about the state of the program to the user who will run the program but you want that end user to get to decide how that information is to be viewed/processed.​*​ ​†​

    Since you do not know in advance what the user will want to do with the information that you will want the program to broadcast, the operating system (OS) provides you with the ability to send messages out of your program in a sort of “un-opinionated” manner. More specifically, the OS lets you place a label on the data you want to emit (viz. “standard” or “error”), but where such messages will go, and how they will be used, will not be decided at the moment you write the program. Rather, the user of the program will be in control as to where data output with the “standard” label will be sent, and where data output with the “error” label will be sent.​‡​

    The purpose of providing the programmer with two spaces to publish information is that it will allow the end user to separate these messages by, for example, viewing the messages sent to stdout on the terminal and saving messages sent to stderr to a file.

    To see this in action, we’ll consider the simple command “ping www.google.com” run on a linux terminal. (I choose ping because it runs indefinitely, allowing us to examine the properties of this process before it ends.)

    If your network connection is ok, this program will print to stdout every second. Now, the terminal is itself a program — a special program — that is designed to receive input, run programs and, being a program, it can (and does) send messages to stdout and stderr.

    Where do those messages “end up”? We can find the PID of the ping process (ps -ef | grep -Ei "PID|ping"), which is 5381 in this case, and then examine the use that PID in the following command on linux:

    sudo ls -l /proc/5381/fd
lrwx------ 1 root root 64 Sep 28 00:10 0 -> /dev/tty1
lrwx------ 1 root root 64 Sep 28 00:10 1 -> /dev/tty1
lrwx------ 1 root root 64 Sep 28 00:10 2 -> /dev/tty1

    The file descriptors shown in this print out (/proc/5381/fd/0, /proc/5381/fd/1, and /proc/5381/fd/2) tell you that the stdin, stdout and stderr respectively for this process all “point to” /dev/tty1. This is a linux virtual device file that you can think of as a handle or interface to a driver for the (emulated) terminal. (This is the same terminal that ping was started in, which can be confirmed by running the command tty.) Since ping prints to stdout, and since stdout points to the terminal emulator, the data is sent there and displayed on the screen accordingly.

    As stated earlier, the destination of messages sent to stdout and stderr is only determined at the moment that the byte code for the program is converted into a new process by the OS. In the case of a linux terminal, the processes that are started therein, such as ping, inherit by default the same standard-stream destinations as those of the terminal. This is why the file descriptors above all point to the terminal by default /dev/tty1. But we can change what the file descriptors will point to when we start the process by using redirects.

    For example, if we now begin a new process in the terminal with ping www.google.com 1> /dev/null, then we get a new PID (5816), and override the default value of the file descriptor (which woould have been /proc/5816/fd/1 -> /dev/tty1), so that we won’t see any regular printout to terminal. Examining the file descriptors again for the new ping process:

    sudo ls -l /proc/5816/fd
lrwx------ 1 root root 64 Sep 28 00:10 0 -> /dev/tty1
lrwx------ 1 root root 64 Sep 28 00:10 1 -> /dev/null
lrwx------ 1 root root 64 Sep 28 00:10 2 -> /dev/tty1
...

    … confirms that stdout is pointing to /dev/null — the linux “black hole” — so the messages now just get thrown away. Likewise, if we now redirect stderr to a file, and stdout to stderr when starting ping:

    ping www.google.com 2> /tmp/temp.txt 1>&2 
sudo ls -l /proc/5816/fd
lrwx------ 1 root root 64 Sep 28 01:25 0 -> /dev/pts/0
l-wx------ 1 root root 64 Sep 28 01:25 1 -> /tmp/temp.txt
l-wx------ 1 root root 64 Sep 28 01:25 2 -> /tmp/temp.txt

    … then we get nothing printed to the terminal, and the output of ping ends up in /tmp/temp.txt, as expected.

    A few notes are useful here if this is new-ish to you:

    • The numbers around the redirect symbol > represent the standard streams as follows: stdin: 0, stdout: 1, stderr: 2. So 1>&2 means redirect stdout to stderr, etc.
    • A redirect symbol > without a number in front is short for 1> (redirect stdout to something)
    • You need an ampersand & after the > symbol whenever you redirect to a number representing a standard stream, otherwise the terminal will read e.g. 2>1 as “redirect stderr to a file named 1“. Don’t use an ampersand though if redirecting to a file path.
    • The order of the redirects in the earlier example might seem counterintuitive at first. You might expect it to look like ping www.google.com 2>&1 1>&2 /tmp/temp.txt, which looks as though it reads “redirect stderr to stdout, and stdout to a file”. But if you think of these redirects in terms of setting what the file descriptors point to, and reading these commands from left to right, then you see that at the moment that the terminal reads 2>&1 it will set the file descriptor /proc/5816/fd/2 to point to the same destination held by /proc/5816/fd/1, which has not been changed yet from its default value; so this redirect will not have any effect, and stderr will still print to screen. That is why you need to first set one of the streams to point to the file (e.g. /proc/5816/fd/1 -> /tmp/temp.txt), and then set the other stream to point to the same thing as the previous stream (e.g. /proc/5816/fd/2 -> /proc/5816/fd/1 -> /tmp/temp.txt).

    You can also print messages to another terminal window by identifying its device file (tty), and then redirecting stdout to that device (e.g. echo hello > /dev/tty2).

    In summary, since most of us learn about programming and system admin in a terminal, it’s easy to come to think of the programs that we’re used to launching therein as being in some sense bound up with, or unusable without, the terminal. But once you realize that all the programs you are used to launching from the terminal have no intrinsic tie to the terminal, and that the terminal has been conveniently determining the default destinations for the standard streams of the programs you’ve been running in it, you can begin to appreciate the need to be able to explicitly set the streams of programs that are not launched by the terminal.

    Python3 Subprocess.run and .Popen

    Now let’s go back to the python code I’d been pasting/copying for several years and see if we can understand and simplify what’s happening with the subprocess module. 

    import subprocess
# Launch CMD
p = subprocess.Popen(
    CMD,
    shell=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT
)
# Wait for it to finish
p.wait()
# Save the result to result str
result: str = p.stdout.read().decode('utf8').strip()

    The first thing I learned by reading the subprocess documentation is that (blush), I wasn’t even using the recommended method:

    The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle

    The run function is a simplified wrapper around the more complete Popen function I had been using, and you basically want to use it whenever you want to execute a synchronous​§​ process of finite duration. For example, you can view your PWD contents with the following in the python3 REPL: 

    >>> import subprocess
>>> subprocess.run(['/bin/ls'])
temp1    temp2

    (Note: if you launch python3 in a terminal then it will inherit the shell’s environment variables, including $PATH, which means that you often don’t need to spell out the full path to the executable as I’ve done here.)

    Notice that the executed program, though launched as a separate process, still prints to the same terminal as the python3 REPL. We can discern why this happens by going through the same procedure we went through earlier, i.e. by launching an ongoing process like ping:

    >>> import subprocess
>>> subprocess.run(['/bin/ping','www.google.com'])
64 bytes from 172.253.63.104: icmp_seq=1 ttl=101 time=1.31 ms
64 bytes from 172.253.63.104: icmp_seq=2 ttl=101 time=1.42 ms
...

    … and, in a separate window, finding the PID and examining the file descriptors of that process:

    ❯ sudo ls -l /proc/18175/fd
lrwx------ 1 root root 64 Sep 28 21:29 0 -> /dev/tty1
lrwx------ 1 root root 64 Sep 28 21:29 1 -> /dev/tty1
lrwx------ 1 root root 64 Sep 28 21:29 2 -> /dev/tty1

    The ping process evidently inherited the same file descriptors as its parent process (the python3 REPL), which itself inherited those descriptors from its parent, the terminal. So both python3 and ping will print to the same terminal.

    Now, we want to be able to launch processes in python3 with something akin to redirection in the terminal. In particular, we want to be able to pipe the standard output streams of the process we launch with the subprocess module to the parent python3 process and to be able to capture that data as a python3 variable. We do that by providing stdout and stderr arguments, as shown in the following example:

    >>> from subprocess import run, PIPE
>>> url = 'www.google.com'
>>> p = run(['/bin/ping','-c','2',url], stdout=PIPE, stderr=PIPE)
>>> p.stdout.decode('utf8')
'PING www.google.com (172.217.2.100) 56(84) bytes of data. ...'

    Notice this time that we made ping run for only a finite duration by supplying the --count=2 arguments, and that the process did not print to terminal while running. This is because the stdout=PIPE argument has an effect similar to the terminal redirection command (1>).

    Where/how was ping‘s stdout redirected? We can investigate as before by rerunning the above code (but without ‘-c’, ‘2’ to make the process run indefinitely), finding the PID of the new ping process in another window, and examining that process’ file descriptors:

    ❯ sudo ls -l /proc/33463/fd/
lrwx------ 1 root root 64 Sep 28 20:58 0 -> /dev/tty1
l-wx------ 1 root root 64 Sep 28 20:58 1 -> 'pipe:[110288]'
l-wx------ 1 root root 64 Sep 28 20:58 2 -> 'pipe:[110289]'
...

    As we can see, ping‘s stdout is now being directed to a device labelled 'pipe:[110288]' (and stderr to a device labelled 'pipe:[110289]'). This “pipe” is an OS in-memory “unnamed” device​¶​ whose purpose is to connect a write-able file descriptor of one process to a read-able file descriptor of another process. (Pipes connect a process to a process, redirects connect a process to a file.) The number 110288 is the ID for the inode of the pipe device file in the filesystem. You can get more information on the pipe device file with the lsof (“list open files”) utility:

    ❯ lsof | grep -E "PID|110288"
COMMAND   PID   ... FD     TYPE    DEVICE ...    NODE    NAME
python3   33462 ... 3r     FIFO    0,13   ...    110288  pipe
ping      33463 ... 1w     FIFO    0,13   ...    110288  pipe

    Here we can see that the pipe shows up in relation to the python3 and ping processes, with PID 33462 and 33463 respectively. The FD column gives the file descriptor number for the pipe file, and the letters r and w refer to read and write permissions. Referring to the previous ls -l command, we can confirm here that /proc/33463/fd/1 does indeed point to this pipe device file, and it does have write-only permissions.

    Lets now look at the corresponding python3 file descriptors:

    > ls -l /proc/33462/fd/
lrwx------ 1 dwd dwd 64 Sep 28 20:59 0 -> /dev/tty1
lrwx------ 1 dwd dwd 64 Sep 28 20:59 1 -> /dev/tty1
lrwx------ 1 dwd dwd 64 Sep 28 20:59 2 -> /dev/tty1
lr-x------ 1 dwd dwd 64 Sep 28 20:59 3 -> 'pipe:[110288]'
lr-x------ 1 dwd dwd 64 Sep 28 20:59 5 -> 'pipe:[110289]'

    Here we can see that the python3 parent process has kept its standard streams pointing to /dev/tty1 (so you can still interact with it through the terminal). In addition, it has created two new file descriptors (3 and 5) pointing to the two pipes we created in our subprocess.run command (one for stdout, one for stderr). The file descriptor /proc/33462/fd/3, as we have seen, is the read-only end of the pipe emanating from the stdout file descriptor of the ping process. This “non-standard stream” file descriptor is created by the python3 process according to its underlying C code. That code is responsible for marshaling the data emitted from the pipe into a python runtime variable and, hence, why we are able to see the result of the ping process printed out in the python3 REPL as a string.

    For reference here is some relatively simple C code demonstrating inter-process communication through pipes, the sort of thing you’d find in python3‘s source code.

    Let’s return to the subprocess module. The other argument worth taking special note of is shell=False. When set to True, subprocess.run passes the first argument (that the documentation recommends be a string now instead of an array of strings) to the /bin/sh program for execution as a script. So now python3 launches a single child process — the sh shell — that can launch arbitrarily many further child processes. That obviously has the advantage of letting you do more complex sequences of commands in a script-like written format, and lets you take advantage of shell features like setting/expanding env variables. Piping to stdout and stderr works the same: any process you invoke in your shell script that writes to either stream will contribute to the strings that become accessible in run().stdout.decode('utf8') and run().stderr.decode('utf8').

    There are only two disadvantages that I can discern of using the shell=True argument:

    • Overhead: it takes resources to start a new shell
    • Complexity: there’s something to be said about keeping your calls to the outside world short, simple and infrequent

    Finally, let’s review the subprocess.Popen that subprocess.run wraps around. The main difference is that run is blocking, while Popen is not. That is, if you start a process with run, it will wait to finish before proceeding to the next line of python code. Again, this is great when you just want to get e.g. the stdout of a command-line tool dumped straight into a string variable:

    >>> from subprocess import run
>>> date = run('date', capture_output=True).stdout.decode('utf8')
>>> print(date)
'Mon Sep 28 23:44:49 EDT 2020\n'

    Note: the argument (..., capture_output=True) is provided in python3.6+ as a short cut for (..., stderr=PIPE, stderr=PIPE).

    Popen, by contrast, is a class constructor that will start the subprocess, return an object that lets you communicate with that process, and then you’ll immediately move onto the next line of code. This is useful if you want to launch a lot of processes, like network requests, in parallel. You then control the timing of the processes with the Popen.wait() method. The Popen object also has more complex data structures owing. toits asynchronous nature, meaning that, for example, you have to convert a buffer to a string using an intermediary .read() method. The equivalent code with Popen to the above run code is thus the more verbose pattern I had been using for so long:

    >>> from subprocess import Popen
>>> p = Popen('date', capture_output=True)
>>> p.wait()
>>> date = p.stdout.read().decode('utf8')
>>> print(date)
'Mon Sep 28 23:44:49 EDT 2020\n'

    Summary

    I expect I’ll be using the following patterns a lot going forward.

    from subprocess import Popen, run, PIPE
### Simple stuff
run(['mv', 'temp', 'temp2'])
### Simple I/O stuff
date = run(['date'], capture_output=True).stdout.decode('utf8')
### Parallel stuff
p1 = Popen('curl -o foo.html https://www.foo.com')
p2 = Popen('curl -o bar.html https://www.bar.com')
p1.wait()
p2.wait()

    (And, yes, I know there are native python3 equivalents to all these commands.)

    Further reading

    https://www.linusakesson.net/programming/tty/

    https://www.informit.com/articles/article.aspx?p=2854374&seqNum=5

    https://lucasfcosta.com/2019/04/07/streams-introduction.html

    1. ​*​
      Is the message to be literally “viewed” (i.e. on the screen), is it to be “stored” (i.e. saved to a file on disk), is it to be “piped” (i.e. imbibed as input information by another program), or ignored (i.e. discarded)?
    2. ​†​
      The “user” of your program could of course be someone who writes a program that calls your program
    3. ​‡​
      The key here is that you don’t need to know what this channel “is”, only that the user will be provided with a systematic means to determine where messages designated for that channel will end up.
    4. ​§​
      I.e. your code will wait for it to finish
    5. ​¶​
      An unnamed device is one that does not show up in the /dev/

  • Raspberry Pi Cluster Part I: Goals, Hardware, Choosing OS

    A while back I built a raspberry cluster with 1 x RPi4 and 2 x RPi3b devices. This was shortly after the release of the RPi4, and, due to the many fixes that it required, I didn’t get far beyond hooking them up through a network switch.

    Now that RPi4 has had some time to mature, I decided to start again from scratch and to document my journey in some detail.

    Goals

    Being able to get computers to coordinate together over a network to achieve various tasks is a valuable skill set that has been made affordable to acquire thanks to the RPi Foundation.

    My goals are to build a cluster in order to figure out and/or practice the following technical competencies:

    • Hardware Skills: acquiring, organizing, and monitoring the cluster hardware
    • Networking Skills: setting up a network switch, DHCP server, DNS server, network-mounting drives, etc.
    • Dev Ops: installing, updating, managing the software, and backing everything up in a scalable manner
    • Web Server Skills: installing apache and/or nginx, with load balancing across the cluster nodes; also, being able to distribute python and node processes over the cluster nodes
    • Distributed-Computing Skills: e.g. being able to distribute CPU-intensive tasks across the nodes in the cluster
    • Database Skills: being able to create shards and/or replica nodes for Mysql, Postgres, and Mongo
    • Kubernetes Skills: implement a kluster across my nodes

    Those the are the goals; I hope to make this a multi-part series with a lot of documentation that will help others learn from my research.

    Hardware

    RPi Devices

    This is a 4-node cluster with the following nodes:

    • 1 x RPi4b (8GB RAM)
    • 3 x RPi3b

    I’ll drop the ‘b’s from hereon.

    The RPi4 will serve as the master/entry node. If you’re building from scratch then you may well want to go with 4xRpi4. I chose to use 3xRPi3 because I already had three from previous projects, and I liked the thought of having less-power hungry devices running 24×7. (Since their role is entirely one of cluster pedagogy/experimentation, it doesn’t bother me that their IO speed is less than that of the RPi4. Also, the RPi4 really needs some sort of active cooling solution, while the RPi3b arguably does not, so my cluster will only have one fan running 24/7 instead of 4.)

    Equipment Organization

    I know from my previous attempt that it’s really hard keeping your hardware neat, tidy and portable. It is important to me to be able to transport the cluster with minimal disassembly, and I therefore sought to house everything on a single tray and with a single power cable to operate it. That means that my cluster’s primary connection to the Internet would be by wifi but, importantly, I’ve insisted that the nodes communicate to each other over ethernet through a switch. The network schematic therefore looks something like this:

    RPi Cluster Network Schematic

    The RPi4 will thus need to act as a router so that the other nodes can access the internet. Since each node has built-in wifi, I’m also going to establish direct links between each node and my home wifi router, but these shall only be used for initial setup and debugging purposes (if/when the network switch fails).

    To keep the RPi nodes arranged neatly, I got a cluster case for $20-$30. Unfortunately, the RPi4 has a different physical layout which spoils the symmetry of the build, but it also makes it easy to identify it. I also invested in a strip plug with USB-power connectors, so that I would only need a single plug to connect the cluster to the outside world. I was keen to power the RPi3s through the USB connectors on the strip plug in order to avoid having 5 power supplies,​*​ which gets bulky and ugly IMO.

    Finally, I had to decide about what sort of storage drives I would use on my RPi3s. For the RPi4s, there was no question that I would need an external SSD drive to make the most of its performance.

    BEWARE about purchasing an SSD for your RPi4! Not all drives work on the RPi4 and I lost a ton of time/money with Sabrent. This time round I went with this 1TB drive made by Netac. So far, so good. If $130 is too pricey then just get a 120/240GB version in the $20-40 range. (I only got 1TB because I have plans to use my cluster to do some serious picture-file back ups and serving).

    For the RPi3s, which I expected to use a lot less in general, there is not nearly as much to be gained from an external SSD. Also, I wanted to limit the cost of the set up as well as the number of cables floating around the cluster and so I decided to start off with SD Cards for the RPi3bs, though I am wary of this decision (and deem it likely that I will regret this decision as soon as one of them fails). I’m using 3x64GB Samsung Evo Plus (U3 speed). I’ll be sure to benchmark their performance once set up.

    I also got a 2TB HDD drive to provide the RPi3s with some more durable read-write space, and on which I’ll be able to backup everything on the cluster.

    I got a simple network switch, some short micro-USB cables , and some short flexible ethernet cables. Be careful with your ethernet cables; you want them short to keep your cluster tidy, but make sure they are not too rigid as a result; in my previous attempt I got these cables that were short but so rigid that they created a lot of torque between the switch and node connectors, and made the whole cluster look/feel contorted.

    I also got a high quality power supply for my RPi4 since it, being the primary node that will undergo the most work, and having two external storage drives to power, needs a reliable voltage.

    Finally, I also got a bunch of USB-A and USB-C Volt/Amp-Meters for a few bucks from China, because I like to know the state of the power going through the nodes.

    So, in total, I calculate that the equipment will have cost ~$500. It’s added up, but that’s not bad a for computing cluster.

    4-Node RPi Cluster Hardware

    And, yes, I need a tray upgrade.

    Choosing an OS

    When it came to choosing an OS, the only two I considered viable candidates were Raspbian OS (64 bit beta), or Ubuntu 20.04 LTS server (64 bit).

    I went with Ubuntu in the end because my project is primarily pedagogic in nature and so, by choosing Ubuntu, I figured I’d be deepening my knowledge of a “real world” OS. I also just generally like Ubuntu, and it has long been my OS of choice on cloud servers.

    For the RPi3s, I used the Raspberry Pi Imager application to select the Ubuntu server 20.04 and burned that image onto each SD card.

    Raspberry Pi Imager

    For the RPi4 though I wanted to boot from an external SSD drive, and this isn’t trivial yet with the official Ubuntu image. I therefore opted to use an image posted here that someone had built using the official image but with a few tweaks to enable booting from an external USB device. (It required you to first update the RPi4’s EEPROM, but I had already done that. It’s easily googled.)

    Initial Setup

    Once the cluster hardware had been assembled and wired up, I powered everything on and then had to go through each fresh install of ubuntu and perform the following:

    • Login with ‘ubuntu’, ‘ubuntu’ credentials
    • Connect to wifi following this advice; note: you need to reboot after calling sudo netplan apply before it will work! (My netplan conf is included in the next part of this series.)
    • Update with sudo apt update; sudo apt upgrade -y
    • Set the timezone with sudo timedatectl set-timezone America/New_York; if you want to use a different timezone then list he ones available with timedatectl list-timezones.
    • Add a new user ‘dwd’ (sudo adduser dwd) and assign him the groups belonging to the original ubuntu user (sudo usermod -aG $(groups | sed "s/ /,/g") dwd)
    • Switch to dwd and disable ubuntu (sudo passwd -l ubuntu)
    • Install myconfig and use it to further install super versions of vim, tmux, etc. See this post for more details.
    • Install oh-my-zsh, powerlevel10k, and zsh-autosuggestions.
    • Install iTerm2 shell integration
    • Install nvm
    • Create a ~/.ssh/authorized_keys file enabling public-key ssh-ing
    • Change the value of /etc/hostname in order to call our nodes rpi0, rpi1, rpi2, rpi3.

    This workflow allowed me to get my four nodes into a productive state in a reasonably short amount of time. I also set up an iTerm2 profile so that my cluster nodes have a groovy Raspberry Pi background, making it quick and easy to distinguish where I am.

    RPi4 node at the ready with tmux, vim, oh-my-zsh, powerlevel10k

    Finally, we also want to allocate memory “swap space” on any device not using. an SD card. (Swap space is the space you allocate on your storage disk that will get used if you use up your RAM. Most linux distros nowadays will not allocate swap space to your boot drive by default, so it has to be done manually.)

    Since only the RPi4 has an external drive, that’s all we’ll lset up for now. (Later, once we have a single network mounted HDD drive available to each node, we’ll allocate swap space there.) Use the following to add 8GB of swap:​†​ 

    sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

    Finally, add the following line to /etc/fstab to make this change permanent: /swapfile swap swap defaults 0 0

    Summary

    That’s it for part I. In the next part, we’re going to set up our ethernet connections between the RPi nodes using our network switch.

    1. ​*​
      4 x RPi + 1 x Network Switch
    2. ​†​
      According to lore it’s best practice to only add ~1/2 your RAM size as swap. However, I’ve never encountered issues by going up to x2.

  • Installing Windows 10 when all you have is a Mac (UEFI Version)

    In a previous article I described how to install Windows 10 on a machine when all you have otherwise is a Mac. The problem is that you need Windows 10 to install Windows 10, so I had to use Virtualbox on the Mac to turn my USB device into a Windows-10 installer.

    The Problem

    I installed Windows 10 onto my refurbished Dell T5810 workstation using the process described in the previous article. This however only worked if you booted your destination drive using “Legacy” mode. That was OK for a while, but I wanted to boot using UEFI because, so I’m told, that’s the more modern/sophisticated way to do things and it will make your machine boot faster.

    Now, I originally assumed that my installation of Windows 10, carried out as it was with the official Microsoft Installer software within Virtualbox, would have made my installation of Windows 10 UEFI compatible by default. So I booted my workstation into System Setup (by pressing F2 on restart), and started playing around with settings. I very naively assumed that Dell would not allow me to set things in such a way as to make it impossible for me to get back into System Setup. Nope. I tried some settings following the advice of some random crapy article I googled, and my workstation stopped working altogether. Powering it on left me with a totally blank screen.

    I followed some advice to get the CMOS back to factory settings by TWICE removing the watch battery from the motherboard, holding the power button for 30 seconds (to drain any residual charge from the capacitors, etc.), but this did not solve the problem. However, the third time I tried this I decided to leave the battery out overnight just in case it needed several hours to completely reset and, thankfully, that worked!

    The Solution

    Upon further research, I came to realize that my installation of Windows 10 was fundamentally incapable of UEFI booting because that feature is simply not integrated into my original USB installer; an option that, as I said, the official Microsoft Software does not offer. Again, how naive of me to assume that a product straight from Microsoft to support its historic flagship product would not offer, let alone provide by default, the most modern booting option.

    Anyhow, long story short, you need to create the USB device installer using a different software product called rufus. So follow all the steps in the previous article up to the point where you have a Windows 10 instance running on Virtualbox, you’ve downloaded the Microsoft MediaCreationTool, and you’ve arrived at this window:

    MediaCreationTool: Choose ISO file this time!

    Now, instead of creating our Windows 10 installer directly onto the USB device, you need to instead create an ISO file and save it to your simulated Desktop.

    Next, go to https://rufus.ie/downloads and save the latest .exe file to your Virtualbox instance’s Downloads folder.

    Download the rufus tool from https://rufus.ie/downloads

    Once downloaded, run rufus as administrator to get this interface:

    Under the “Device” field you need to select your USB device (which, here, is confusingly labelled by its partitions). Next, in the “Boot selection” field you need to select “UEFI:NTFS”, and make sure the Partition Scheme is GPT. In the field next to the “Boot selection” field, labelled “Select”, you need to locate the ISO file you downloaded using the MediaCreationTool. Make sure the rest of your settings are the same as in the image above (or are otherwise sensible), and hit START to format your USB device.

    Once that process is finished you can quit Virtualbox and, boot up your target machine with this USB device, and install Windows 10 from scratch, and then you can expect it to boot with UEFI.

  • Installing Windows 10 when all you have is a Mac (Legacy/Non-UEFI Version)

    Important! This advice has been deprecated and is only applicable if you’re trying to install Windows 10 on a machine without UEFI support. If your machine does have UEFI support, then start here.

    Background

    Skip this section if you just want to jump to the “how I did it” content.

    I haven’t used Windows in a serious way for about 15 years now. I enthusiastically converted to Apple in ~2005 and never expected to want to own a PC again. From about 2005 to 2015, I held Microsoft with contempt and couldn’t imagine why anyone would use their products. IMO, Steve Jobs raised the bar incredibly high as to what a modern personal-computer company can be.

    It turns out though that Microsoft eventually took notice (circa Balmer’s exit?) and began raising its own bar. It began to get my notice about ~5 years ago when it started developing some fantastic open source products (typescript and Visual Studio Code in particular).

    I’d also started hearing rumors about linux being integrated into Windows 10. Curiosity to check it out, and to be able to better interact with students and interns who had Windows machines, lead me to finally commit to getting a machine on which I could give Window’s a fresh shot.

    I’d also been wanting to try out some photogrammetry software for a while that requires an Nvidia GPU. That, and other motivations, lead me to seek out a server/workstation on to which I could install Windows 10. I found a great deal on Newegg.com for a Dell Precision T5810. For $290 (+ $15 tax) I got 64 GB DDR4 RAM, a E5-2620 v3 2.4GHz 6-Core (x2 hyper-thread) Intel Xeon CPU, and a simple GPU (Quadro NVS 295). The only catch was it had no drives and, therefore, no starting OS.

    I had some drives sitting around, and figured it would be easy to just install Windows 10 myself. However, I was surprised to learn upon googling “Install Windows from USB” that almost all solutions assumed you already had access to a Windows 10 machine. Getting round this chicken-and-egg problem with just a Mac took some research that I hope others can benefit from.

    Installing Windows 10 without Windows 10

    Requirements

    • Mac with recent OS and Homebrew installed
    • At least ~25GB free hard drive space
    • An external SSD drive (see below; a recent USB 3.0 flash drive is supposed to work but didn’t seem to in my case)

    Basic Approach

    After some digging, I concluded that there are two basic ways to get a Windows 10 installer onto a USB drive starting with just a Mac:

    1. Run a modified “Boot Camp Assistant” from my Mac’s /Applications/Utilities/Boot Camp Assistant.
    2. Emulate Windows 10 on your Mac.

    If you want to go with the first approach then you can consult this gist that seems to be thoroughly researched. However, I it had an aura of hackiness about it, so I opted to go with an emulation solution.

    To be clear, I really dislike emulating whole operating systems, so the goal here is to create-use-delete a VM Windows 10 instance as quickly and simply as possible so that we can get Windows 10 properly installed on a separate dedicated machine.

    Install Windows 10 via Virtualbox

    First, you’ll need to obtain a copy of Windows 10 as an ISO file. You can get that direct from Microsoft. I went for Windows 10, 64-bit, with English language. The downloaded ISO file is ~5GB. Save this to your Downloads folder.

    While that’s downloading, install virtualbox and its extension pack on your Mac with:

    brew cask install virtualbox
    brew cask install virtualbox-extension-pack

    Open virtualbox from /Applications and click on the blue-spiky-ball “New” icon to create a new virtual machine.

    Virtualbox Main UI

    You’ll be guided through a few steps. Give the VM a name, choose Windows (64-bit) as your OS, and decide how much RAM you want the VM to be able to use. (I wasn’t sure what to pick here since I didn’t know how RAM-hungry Windows 10 is or the mechanics of virtualbox in these regards. My machine has 16GB, so I figured I’d allot ~6GB. I monitored virtualbox during the heavy parts and it pretty much maxed out all my Mac’s available memory at times, but I’m still not sure how this setting is handled in that regard.)

    When you’ve chosen your memory size, select “Create a virtual hard disk” and then click “Create”.

    Virtualbox wizard to allot RAM.

    You’ll then be asked about how/where you want the hard disk for the VM to be set up. The first time I did this, I gave a ball park guess of ~20GB, but later found that this wasn’t enough. Windows 10 actually requires a minimum of 22,525MB, hence why I recommended earlier that you allot at least 25GB. However, if you can spare it, then go for even more like 40GB — this is the amount I chose (second time trying!), and it seemed to work fine for me.

    Leave the other default settings and click “Create”.

    Virtualbox wizard to allot hard-disk space and file format.

    Once the VM is created, we need to enable the VM to access the USB device.

    Now, as I mentioned earlier, a USB flashdrive is supposed to work, but I tried 3 different flash drives and found that they would all fail late in the process (and get extremely hot). Luckily for me, I had an external SSD drive lying around so I tried that and it worked no problem. I can only speculate that the work involved in writing to a USB drive passing through the Mac OS to/from the Windows 10 instance was too intense for relatively slow flash drives, and thus requires an SSD. For reference, the SSD drive I used can be found here on Amazon. (If you want to give as fast USB thumdrive a go then good luck to you, but I’ll assume from hereon that you’re using an SSD.)

    Anyhow, once you have your SSD interted into your Mac, select the Windows 10 item in the left column of main virtualbox view so that its various settings and properties and can be viewed on the right of that view. Go down to the USB section and click on the word “USB”.

    The main view of the virtualbox interface.

    This will open a menu enabling you to add the SSD drive that we will want to make available to our virtual Windows 10 instance.

    Virtualbox menu to pass USB device through to virtual Windows 10 instance.

    You also must also make sure to select “USB 3.0 (xHCI) Controller”, or your Windows 10 instance won’t detect the USB. (Note: these USB 2.0 and 3.0 options are only available because you installed the virtualbox-extension-pack earlier.)

    One last thing before we try to start our Windows 10 instance: go to virtualbox preferences, and select the “Display” tab. Change the “Maximum Guest Screen Size” to “none” and then click “OK”. This will prevent the simulated screen from showing up in a smallish box that will make it rather tedious to interact with the Windows 10 simulation.

    Virtualbox preferences; enable the view of instance to scale.

    Now we’re ready to start our Windows 10 instance, so click “Start” at the main interface. The first time you run the Windows 10 instance, virtualbox will prompt you to select a “disk” with which to boot the new virtual machine. You need to go through the drop-down selector to add the ~5GB Windows 10 ISO file that you saved to your Downloads folder earlier.


    Virtualbox wizard to select Windows 10 ISO file.

    Once the disk has been selected from your Downloads folder, click “Start” to launch the VM simulating a first installation of Windows 10. Along the way, you’ll have to answer standard setup questions, accept terms, etc. Keep things simple: don’t sign into a Microsoft account (go with “Offline Account” and “Limited Experience”). When prompted, do not try to use an activation key (just press “I don’t have a key”). When you come to select an OS, I went with Windows 10 Professional (though things might have been more streamlined if I’d gone with the simpler Home Edition). On the screen “Choose privacy settings for your device”, I switched basically every service off. Decline all of the personalization, Cortana-spy-on-you functionality, etc.

    Switch off all of the invasive-data options in the Windows 10 setup.

    This process took me about 10 minutes to get through.

    An important step is to choose “Custom: Install Windows Only” since we are not upgrading a system from a previous incantation of Windows.

    Eventually, you’ll end up with a working Windows 10 interface.

    Installing a Windows 10 installer within an virtual Windows 10 machine onto a USB device

    Once in a working Windows 10 instance, we need to install the program that will turn our USB device into a portable Windows 10 installer. Open the Edge browser (the icon on the Desktop is easiest), go to:

    www.microsoft.com/en-us/software-download/windows10

    … and click on “Download tool now” under “Create Windows 10 installation media”.

    Opt to save this download, and then double click on that exe in the Downloads folder. This will launch the “Windows 10 Setup” wizard. Select “Create installation media (USB flash drive, DVD, ISO file) for another PC” when prompted.

    You’ll then need to select/confirm your Windows version, language and target architecture.

    Finally, you’ll be asked to choose your USB device. If you set up the pass-through options correctly earlier, then it will show up as the sole option.

    Click next to start the installation onto the USB disk. This will take an external SSD drive about 10-15 minutes to complete. As I mentioned, I also tried three USB flash drives, and they each started getting sluggish after about 10 minutes (slowing to a halt at ~50%), and then reported an obscure error after about 20 minutes (having got very hot!). If you don’t have an external SSD handy, then I’ve heard good things about the Sandisk Extreme Pro.

    Once the process completes, you can power down the VM instance, right click on the item in the main virtualbox interface, and remove it. It will then give you an option to remove all related files. This will free up your disk space.

    Using the SSD as a boot drive on a Dell T5810

    It’s beyond the scope of this article to go over the general details of installing an OS from such a USB device, but I’ll quickly mention the smooth ride I had from thereon with my refurbished Dell T5810.

    In my case, I had to add a main drive to the T5810 (a 1TB SSD Samsung), and then I powered on the machine with the external SSD plugged into one of the USB 3.0 slots. The first boot took a while (~1 min as I recall) to show anything, but then the Dell logo showed up and I pressed F2 in order to enter System Setup. There I was able to select Legacy Boot and ensure that the external SSD would be used early on in a legacy-BIOS boot. Exiting that menu caused the system to reboot from the external SSD and entered me into a Windows 10 installation wizard as expected.

    The only hiccup I encountered concerned an error when trying to select the internal SSD drive (I was told Windows 10 could not be installed there), but that problem was quickly solved by this absolutely fantastic ~1 min video on Youtube.

    I now have Windows 10 working great on the T5810, and I’m so impressed with it that I’ll have to write another article soon on that subject!

  • iTerm, bash/zsh, tmux & vim

    Intro

    Having realized that I’d been spending more and more time in an ssh shell in the last year or two, I decided that I needed to up my terminal game. I’d been using bash and vim for a while now, but wanted to fine-tune my developer experience by:

    • giving zsh a try,
    • improving my vim setup,
    • being able to split up my terminal window, and
    • enabling myself to get productive quickly on new machines.

    This last point is key. I often find myself on new systems and hate not having all of my configurations available. So I decided that I would set up a github repo that I could quickly call from anywhere to provision all of my favorite tools and settings (env vars, paths, aliases, functions, config files, executables, etc.).

    This turned out to be a heftier project than I expected, so I won’t take you through step-by-step everything I did, but I will describe the overall approach, share my code, and encourage you to embark on your own workflow-revamping project.

    Choosing Technologies

    iTerm

    I’ve been using a MacBookPro for many years now with iTerm2 as my terminal of choice. It’s overall a more complete/customizable application than the native Terminal App, and has some great additional features such as “profile switching”​*​ and the ability to print images directly in the terminal, even if you’re on a remote machine (without X-forwarding, etc.). If you’re on a Mac, then take the time to set up iTerm2.

    Bash & Zsh

    I’d been a bash-only guys for many years, but heard good things about zsh, especially powerlevel10k and ohmyzsh, so figured I’d make the switch for my everyday shelling. My scripts will remain in bash, of course, but I want to be able to have a decent bash experience as soon as possible on any machine, as well as the ability to install zsh with minimal additional effort if I expect to use that machine repeatedely.

    Vim

    Every IT person, it seems, has to choose between vim and emacs. I like keeping my fingers on the home keys, and so I concluded long ago that emacs wasn’t for me (due its dependence on the ctrl key). If I had to recommend vim or emacs to a beginner, I would say vim mainly because I like that it makes you think about editing in a whole new way (viz. modally).

    In any case, Vim has been my go-to terminal editor for some years, but I didn’t use it for any serious editing, and that left me frustrated at times when I was on a remote server and wanted to dig through some code for a quick fix. My goal here was therefore to improve my speed, learn more tricks, and set things up so that I could do proper IDE-like editing in the terminal on any machine.

    One thing I’ve come to accept in using vim is that it just sucks without a ton of plugins and settings in your .vimrc file. (E.g. I consider the Esc key unusable on my Mac keyboard, especially since Apple made it part of the prone-to-crash power bar). I also can’t stand vim’s default copy/paste system, so I have to employ a lot register logic to make items available to the system clipboard. In short, I gave up long ago of trying to use vim’s ‘out-of-the-box’ settings, which made my need to easily ship my .vimrc around the internet all the more pressing.

    Tmux

    Before I set myself this project I had never bothered using a mutliplexer. That meant that if I wanted to have multiple ssh shells available at one time, I’d have to create separate tabs in iTerm2 and initiate multiple ssh tunnels. Not only was this slow and inefficient, but it meant that I simply couldn’t have multiple shells running if, for example, I was working on a machine’s boot terminal. So I decided that I would learn a multiplexer.

    The toughest call I had was choosing tmux or screen for my terminal multiplexing. My main concern was choosing the one that was more commonly pre-installed on linux distros (in case I find myself wanting to be quickly productive on a system without root privileges). I could not get a definitive answer to this question, but the impression I got was that although screen is historically the more commonly pre-installed multiplexer, tmux is on the rise (the most notable case being RHEL/Centos 8 dropping screen in favor of tmux). So, for better or for worse, I went with tmux.

    Note: iTerm2 has tmux sort of built into it, but I did not want to use that since I wanted to get used to standard tmux for use on any system.

    Result: myconfig

    The result of my workflow revamping was a repo called myconfig. Now, all I have to do is copy and paste the command in this repo’s Quick Start section into the shell of a Mac, RHEL/Centos or Debian machine that already has git. This command will clone/update the myconfig repo to /tmp/myconfig, and then source /tmp/myconfig/tmp_install.sh, which leaves you in a bash shell on the new machine that (i) detects whether I’m on a Mac, Debian or RHEL/Centos machine, and (ii) accordingly sets paths, functions, etc. to leave me within a productive state within seconds! (It even initiates a background installation of vim plugins if vim is detected on the system!)

    This “temporary” installation of myconfig and its contents is kept entirely within /tmp so that I don’t muddy whatever person’s account I might be using. However, the shell is endowed with a function myconfig_full_installation that I can run if I want to install myconfig to the account’s home directory, as well as key config files such as .tmux.conf, .vimrc, .bashrc, and .zshrc.

    Myconfig also provides me with scripts to build the latest, feature-complete vim, tmux, zsh and git executables (in case I find myself on a machine with outdated software).

    In summary, I can now have most of my settings/config established on any of my target machines with just a single copy/paste command.

    Building vim and tmux from source

    One of the trickier parts of creating myconfig was enabling me to access a high-feature build of vim and tmux on my target machines without sudo permissions.

    Not all installations of vim are born equal. To examine the features available on your installation, you can type vim --version and examine the items prefixed with ‘+’ or ‘-‘. The two features I need for my vim usage is +clipboard and +python3. These features are not available on the default vim installation on Centos 7 or even Ubuntu 18. So I had to create scripts to install vim from source code with these features on my two flavors of linux, which you can see here and here.

    In the case of Centos 7, I needed to be able to build it from source without sudo permissions. To develop that script, I found it very helpful to simulate my target machine using the docker image from yamamon/centos7-minimal, and installing only those packages that had been made available to me by that machine’s admin (viz. using `yum list installed | grep ABC` ).

    Summary

    Ever since investing in my workflow I not only feel like my efficiency has increased, and I learned a surprising amount of devops/admin stuff by having to research these various technologies. I also just enjoy my ssh-ing a lot more now. The combination of tmux, iTerm2, zsh (along with oh-my-zsh and powerlevel10k), has turned my boring old shell into power-zone work of art.

    So if you haven’t got your own system sorted out for quickly transferring your workflow to new machines, then I recommend it and hope that myconfig can give you some helpful tips along the way.

    1. ​*​
      This feature allows you to configure a profile (i.e. color, background images, etc.) that automatically gets selected when you move to a specified directory, host, user, etc. It allows me to instantly discern where that shell “is”.