Network Booting an Embedded Linux Target

Embedded devices, like the BeagleBone Black (BBB) featured in this article, have limited resources. Their processors are optimized for cost and reduced power consumption — making them slower and less capable than workstations and server processors. RAM follows suit, where quantity and speed typically lag behind the development station. Similarly, nonvolatile storage space is also limited. For these reasons, it make little sense to use an embedded device for development operations, such as compiling source code. However, embedded devices, especially reference boards like the BBB, should be utilized for testing interim software builds continuously throughout the development process.

Keeping an embedded target in the test loop will have obvious benefits. Primarily, the hardware interfaces are available for use. This allows for the test bench to keep hardware in-the-loop. Additionally, the processor architecture and endianness match the deployment case. Confidence in test builds is then higher, as the compiled results run under the production binary execution environment.

There are also drawbacks to using the embedded target. Scarcity can become an issue. Usually low-cost reference boards, like the BBB, are the solution to providing ample test platforms to a team. Unfortunately, lately with the pandemic, supply issues and long lead-times have bested even the Bone’s attempt to reduce the scarcity bottleneck. Another drawback can often be the available space on the embedded filesystem. Typically, development builds are larger than release builds. Worse yet, there are usually a number of experimental builds deployed at any given time. More disk space is needed than embedded targets can provide! Also consider the wear on the device. Embedded storage has a shelf life — especially for write operations. Hammering a device with builds on a substantial project can easily cause premature failure. A final drawback to examine is time. Change is rapid during development projects, and reconfiguration and reimaging of embedded devices can consume valuable time when responding to change.

I can’t offer a solution to scarcity, but with a network filesystem mounted at root, I can offer remediation for the aforementioned problems when keeping an embedded target in the test loop.

Prerequisites

This guide should be adoptable for many embedded Linux devices, but can be followed specifically for the BBB with the equipment below.

If you are new to the BBB, check out my introductory article, Waking Up the Old Bone.

WARNING: Backup all data from the BBB. To correct mistakes in this process, the device may need to be reflashed.

Definitions

These network addresses can be changed to meet your needs, of course.

Desired Boot Sequencing

An embedded target can locate the Linux OS on a remote location and download it over TFTP. Enabling this functionality will provision a development environment where the kernel image can be readily updated and applied to the target with a simple reboot. Additionally, the Linux kernel mounts a NFS share, from the specified Host, at the root of the file system. This opens up a myriad of benefits. The NFS share will have ample capacity. The underlying storage can have enterprise-grade write endurance. The NFS server can backup the share for disaster recovery. And, development workstations can post build results to the share and see them instantly available on the target.

Preparing the Target File System

Some preparation work will be required to adjust the standard image for network booting.

Boot the Target to the Downloaded Image

It’s time to head over to the Latest Images for the BeagleBoard. The recommended version of Debian for the SD card will suffice. Don’t pick a flasher image!

On the Host system, use the following commands to download the image, verify the integrity by matching the posted checksum, and unzip the image to the local directory:

you@host:~$ wget https://debian.beagleboard.org/images/bone-debian-10.3-iot-armhf-2020-04-06-4gb.img.xz
you@host:~$ sha256sum bone-debian-10.3-iot-armhf-2020-04-06-4gb.img.xz >> debian_checksum.txt
you@host:~$ xz -d bone-debian-10.3-iot-armhf-2020-04-06-4gb.img.xz

Insert the microSD card system to the Host. You’ll need to identify the disk. I did this as follows:

you@host:~$ sudo fdisk -l | grep /dev/sd

From the output, I used the expected size of the microSDHC card to identify the disk:

Disk /dev/sda: 447.13 GiB, 480103981056 bytes, 937703088 sectors
/dev/sda1     2048   1050623   1048576   512M EFI System
/dev/sda2  1050624 937701375 936650752 446.6G Linux filesystem
Disk /dev/sdb: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors
/dev/sdb1   2048 1953525134 1953523087 931.5G Linux filesystem
Disk /dev/sde: 29.74 GiB, 31914983424 bytes, 62333952 sectors
/dev/sde1  *     8192 7372799 7364608  3.5G 83 Linux

Use the dd program to write the image to the microSDHC card:

you@host:~$ sudo dd bs=64K if=bone-debian-10.3-iot-armhf-2020-04-06-4gb.img of=/dev/sde

Return to the Target, and put a marker file on the file system to identify which root has mounted:

you@target:~$ sudo touch /is-local-memory.txt

Shutdown the BBB, and insert the microSDHC card. Hold down the S2 button as you restore power in order to boot the bone off the SD card. (The S2 button is near port 45 on the P8 header.) The S2 button can be released after a few seconds.

Once the board boots, run the following command and confirm there are no results:

you@target:~$ ls -l /is-local-memory.txt

Configure Static Networking

At the time of this writing, the Debian Linux image uses the connman program to manage the network interfaces. (As much as I love Linux, I wonder how many times the wheel must be reinvented on how to configure a network adapter.) Unfortunately, my testing has revealed that connman does not function correctly on the BBB after a NFS boot is performed. No worries, though, it’s pretty simple to walk back to the traditional /etc/network/interfaces file!

First, uninstall connman:

you@target:~$ sudo apt --auto-remove purge connman
you@target:~$ sudo rm -rf /run/connman

Now, configure eth0 in /etc/network/interfaces:

# The primary network interface
auto eth0
iface eth0 inet static
    address 192.168.10.20/24
    gateway 192.168.10.1

Next, add a nameserver to /etc/resolv.conf. (The connman service used to manage this file, so you probably have to create it.) Obviously, use whatever nameserver is appropriate for your environment, or 8.8.8.8 if you feel extra lazy.

nameserver 192.168.10.1

Optionally, disable IPv6 by adding the following to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Reboot the Target and confirm the IP address is statically assigned. Ping an external host by name, like google.com, to confirm the DNS resolution is operational.

Stage the Target File System

Shutdown the Target and transfer the microSDHC card to the Host. Mount the partition, and copy the files to the location from which a future NFS share will be configured:

you@host:~$ mkdir sd-card-p1
you@host:~$ sudo mount -o ro /dev/sde1 sd-card-p1
you@host:~$ sudo mkdir -p /srv/nfs/bbb-rootfs
you@host:~$ sudo cp -a sd-card-p1/* /srv/nfs/bbb-rootfs/
you@host:~$ sudo umount sd-card-p1
you@host:~$ rm -rf sd-card-p1

Just a couple of edits remain before the file system is configured properly. The first modification necessary is to disable the autoconfiguration of the eth0 adapter. The Linux kernel on the target will configure eth0 as part of the boot process, since the kernel command line has arguments for mounting the root file system via NFS.

Remove or comment out the “auto eth0” stanza in /srv/nfs/bbb-rootfs/etc/network/interfaces:

# Due to NFS boot, the eth0 adapter should not be autoconfigured!
# auto eth0
iface eth0 inet static
    address 172.28.4.20/24
    gateway 172.28.4.1

Next, we need adjust the file system table. The Linux kernel will mount the root file system as-directed by the command line arguments. As such, we need to remove the old mount entry that was necessary when booting from the microSDHC card.

Remove or comment out the root mount declaration in /srv/nfs/bbb-rootfs/etc/fstab:

# / will be mounted by the NFS kernel boot!
# /dev/mmcblk0p1  /  ext4  noatime,errors=remount-ro  0  1

Configuring the Host System

Herein I will describe basic installation recipes for the two necessary services the Host must provide — TFTP and NFS. Each of these protocols is known to have security vulnerabilities, so take the necessary precautions.

All Host configuration commands and file edits within this section have been tested using Ubuntu 20.04 LTS.

Setting up the NFS Share

Ubuntu provides the following package for installation of a NFS server:

you@host:~$ sudo apt install nfs-kernel-server

Once installed, tailor the /etc/exports file:

# Takeoff Technical
#  *) Set BeagleBone Black root file system to the local network
#  *) rw = read/write
#  *) secure = only allow client requests that use port numbers below 1024
#  *) no_subtree_check = optimization for reliability - reduces server workload
#  *) async = a performance improvement with a small risk when unclean server restart
#  *) no_root_squash = used for diskless clients - does not map root to anonymous
/srv/nfs/bbb-rootfs 192.168.10.20(rw,secure,no_subtree_check,async,no_root_squash)

Start the server and confirm it is operational before continuing.

you@host:~$ sudo systemctl start nfs-kernel-server.service
you@host:~$ systemctl list-units --all --type=service --no-pager | grep nfs-server

Setting up the TFTP Server

Ubuntu provides the following package for installation of the TFTP service:

you@host:~$ sudo apt-get install tftpd-hpa

After installation, tailor the /etc/default/tftpd-hpa file:

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="192.168.10.10:69"
TFTP_OPTIONS="--secure --ipv4"

Restart the TFTP service:

you@host:~$ sudo systemctl restart tftpd-hpa.service

Create a directory for the boot data:

you@host:~$ mkdir /srv/tftp/bbb-boot

Adjust the Host system’s /etc/fstab to add a binding mount for the TFTP data by appending the following lines:

# TFTP mounts
/srv/nfs/bbb-rootfs/boot /srv/tftp/bbb-boot none bind,ro 0 1

Complete the mount:

you@host:~$ sudo mount -a

Configure the Firewall

Refer to https://wiki.debian.org/SecuringNFS for additional information.

We need to pick some ports to bind the NFS services to. Use the following command to list the ephemeral port range:

you@host:~$ sysctl net.ipv4.ip_local_port_range

The result shows the configuration of net.ipv4.ip_local_port_range, which is probably set to the range 32768-60999. These are the ports the system will give out to sockets when no explicit port is requested in a bind call. When configuring services, I like to choose ports outside of this range. I know ports under 1024 are for system use, so I pick from the range 1025-32767. The port numbers for STATD and MOUNTD below have no rhyme or reason beyond that!

Edit /etc/default/nfs-common:

STATDOPTS="--port 16765 --outgoing-port 16766"

Edit /etc/default/nfs-kernel-server:

RPCMOUNTDOPTS="--manage-gids --port 16767"

Add the rules to the firewall:

# Allow  TFTP from the Target to the Host
you@host:~$ sudo ufw allow from 192.168.10.20 to 192.168.10.10 port 69 proto udp

# Allow NFS from the Target to the Host
you@host:~$ sudo ufw allow from 192.168.10.20 to 192.168.10.10 port 111
you@host:~$ sudo ufw allow from 192.168.10.20 to 192.168.10.10 port 2049

# Allow STATD from the Target to the Host
you@host:~$ sudo ufw allow from 192.168.10.20 to 192.168.10.10 port 16765
you@host:~$ sudo ufw allow from 192.168.10.20 to 192.168.10.10 port 16766

# Allow MOUNTD from the Target to the Host
you@host:~$ sudo ufw allow from 192.168.10.20 to 192.168.10.10 port 16767

Configuring the Target System

An embedded target runs the Linux OS over the network when the bootloader is configured to command it so. The BBB has a clever bootloader, and it is relatively easy to provide it instructions for NFS booting the kernel. This is achieved by placing a text file on the local filesystem. Before getting to that, it’s worth exploring the implementation first, because understanding the boot environment will empower you to troubleshoot and make further customizations.

The Universal Bootloader: Das U-Boot

The BBB, like many (most?) other embedded Linux devices, uses Das U-Boot. This bootloader performs the expected operations, as best-described by Derek Molloy:

The U-Boot Environment

U-Boot is typically configured with a series of environment variables that dictate the boot sequence. Most contain settings relevant to optional components of a boot. Some contain logic that will be executed in the U-Boot shell. Unfortunately, by nature, the U-Boot environment variables that contain logical expressions are displayed all on one line — regardless of the complexity. For example, have a look at the variable ‘boot’, which is important to understand when studying how U-Boot decides to boot the BBB:

boot=${devtype} dev ${mmcdev}; if ${devtype} rescan; then gpio set 54;setenv bootpart ${mmcdev}:1; if test -e ${devtype} ${bootpart} /etc/fstab; then setenv mmcpart 1;fi; echo Checking for: /uEnv.txt ...;if test -e ${devtype} ${bootpart} /uEnv.txt; then if run loadbootenv; then gpio set 55;echo Loaded environment from /uEnv.txt;run importbootenv;fi;echo Checking if uenvcmd is set ...;if test -n ${uenvcmd}; then gpio set 56; echo Running uenvcmd ...;run uenvcmd;fi;echo Checking if client_ip is set ...;if test -n ${client_ip}; then if test -n ${dtb}; then setenv fdtfile ${dtb};echo using ${fdtfile} ...;fi;gpio set 56; if test -n ${uname_r}; then echo Running nfsboot_uname_r ...;run nfsboot_uname_r;fi;echo Running nfsboot ...;run nfsboot;fi;fi; echo Checking for: /${script} ...;if test -e ${devtype} ${bootpart} /${script}; then gpio set 55;setenv scriptfile ${script};run loadbootscript;echo Loaded script from ${scriptfile};gpio set 56; run bootscript;fi; echo Checking for: /boot/${script} ...;if test -e ${devtype} ${bootpart} /boot/${script}; then gpio set 55;setenv scriptfile /boot/${script};run loadbootscript;echo Loaded script from ${scriptfile};gpio set 56; run bootscript;fi; echo Checking for: /boot/uEnv.txt ...;for i in 1 2 3 4 5 6 7 ; do setenv mmcpart ${i};setenv bootpart ${mmcdev}:${mmcpart};if test -e ${devtype} ${bootpart} /boot/uEnv.txt; then gpio set 55;load ${devtype} ${bootpart} ${loadaddr} /boot/uEnv.txt;env import -t ${loadaddr} ${filesize};echo Loaded environment from /boot/uEnv.txt;if test -n ${dtb}; then echo debug: [dtb=${dtb}] ... ;setenv fdtfile ${dtb};echo Using: dtb=${fdtfile} ...;fi;echo Checking if uname_r is set in /boot/uEnv.txt...;if test -n ${uname_r}; then gpio set 56; setenv oldroot /dev/mmcblk${mmcdev}p${mmcpart};echo Running uname_boot ...;run uname_boot;fi;fi;done;fi;

Obviously, this makes for a tough read. To remedy, I’ve created a script that can output complex one-liners, like the declaration of ‘boot’, into code with whitespace in the right locations. See u-boot/parse-multicmd-vars.py in my Helpful Scripts Repository. This allows you to quickly reformat the environment to view in a text or code editor for study. To analyze the boot-time environment, press the spacebar to interrupt the boot sequence, as-prompted. (You’ll need to use the TTL serial cable.) Then, run the printenv command. Copy all of the resulting text from the serial console and paste into a file. Supply this file as an argument to the script, and the code will output a version of the file with keyword separation and whitespace indentation added for readability.

After running the script, let’s now reexamine the ‘boot’ variable:

boot=
    ${devtype} dev ${mmcdev};
    if ${devtype} rescan;
    then
        gpio set 54;
        setenv bootpart ${mmcdev}:1;
        if test -e ${devtype} ${bootpart} /etc/fstab;
        then
            setenv mmcpart 1;
        fi;
        echo Checking for: /uEnv.txt ...;
        if test -e ${devtype} ${bootpart} /uEnv.txt;
        then
            if run loadbootenv;
            then
                gpio set 55;
                echo Loaded environment from /uEnv.txt;
                run importbootenv;
            fi;
            echo Checking if uenvcmd is set ...;
            if test -n ${uenvcmd};
            then
                gpio set 56;
                echo Running uenvcmd ...;
                run uenvcmd;
            fi;
            echo Checking if client_ip is set ...;
            if test -n ${client_ip};
            then
                if test -n ${dtb};
                then
                    setenv fdtfile ${dtb};
                    echo using ${fdtfile} ...;
                fi;
                gpio set 56;
                if test -n ${uname_r};
                then
                    echo Running nfsboot_uname_r ...;
                    run nfsboot_uname_r;
                fi;
                echo Running nfsboot ...;
                run nfsboot;
            fi;
        fi;
        echo Checking for: /${script} ...;
        if test -e ${devtype} ${bootpart} /${script};
        then
            gpio set 55;
            setenv scriptfile ${script};
            run loadbootscript;
            echo Loaded script from ${scriptfile};
            gpio set 56;
            run bootscript;
        fi;
        echo Checking for: /boot/${script} ...;
        if test -e ${devtype} ${bootpart} /boot/${script};
        then
            gpio set 55;
            setenv scriptfile /boot/${script};
            run loadbootscript;
            echo Loaded script from ${scriptfile};
            gpio set 56;
            run bootscript;
        fi;
        echo Checking for: /boot/uEnv.txt ...;
        for i in 1 2 3 4 5 6 7;
        do
            setenv mmcpart ${i};
            setenv bootpart ${mmcdev}:${mmcpart};
            if test -e ${devtype} ${bootpart} /boot/uEnv.txt;
            then
                gpio set 55;
                load ${devtype} ${bootpart} ${loadaddr} /boot/uEnv.txt;
                env import -t ${loadaddr} ${filesize};
                echo Loaded environment from /boot/uEnv.txt;
                if test -n ${dtb};
                then
                    echo debug: [dtb=${dtb}] ...;
                    setenv fdtfile ${dtb};
                    echo Using: dtb=${fdtfile} ...;
                fi;
                echo Checking if uname_r is set in /boot/uEnv.txt...;
                if test -n ${uname_r};
                then
                    gpio set 56;
                    setenv oldroot /dev/mmcblk${mmcdev}p${mmcpart};
                    echo Running uname_boot ...;
                    run uname_boot;
                fi;
            fi;
        done;
    fi;

We now see the import control constructs provided by the U-Boot environment variables. The U-Boot shell will run this logic and behave differently based on the results of several key checks along the way. Of particular interest, when configuring for network boot, are the lines highlighted above. As you can see, if we place a file called uEnv.txt at the root of the Linux filesystem, and if that file contains definitions of client_ip and uname_r, we will execute the command nfsboot_uname_r.

nfsboot_uname_r=
    echo Booting from ${server_ip} ...;
    setenv nfsroot ${server_ip}:${root_dir}${nfs_options};
    setenv ip ${client_ip}:${server_ip}:${gw_ip}:${netmask}:${hostname}:${device}:${autoconf};
    setenv autoload no;
    setenv serverip ${server_ip};
    setenv ipaddr ${client_ip};
    tftp ${loadaddr} ${tftp_dir}vmlinuz-${uname_r};
    tftp ${fdtaddr} ${tftp_dir}dtbs/${uname_r}/${fdtfile};
    run nfsargs;
    bootz ${loadaddr} - ${fdtaddr};

Now we’re getting somewhere! This command will both configure for the NFS root filesystem and execute TFTP calls to load the kernel image from the server — exactly what we’re looking for!

Customizing U-Boot via uEnv.txt

Now, with some more crafty analysis, it would be possible to construct the file uEnv.txt from scratch. However, it’s not necessary. In fact, the newer Debian releases for the BBB include a sample of the file, nfs-uEnv.txt, that can be quickly adopted for the specific development environment.

Boot the Target. Check for the marker file we created earlier to confirm we are back on the local filesystem. You should get one result!

you@target:~$ ls -l /is-local-memory.txt

Now rename the reference file for NFS boot at the filesystem root:

you@target:~$ sudo mv /nfs-uEnv.txt /uEnv.txt

Make the following edits:

##client_ip needs to be set for u-boot to try booting via nfs

client_ip=192.168.10.20

#u-boot defaults: uncomment and override where needed

server_ip=192.168.10.10
gw_ip=192.168.10.1
#netmask=255.255.255.0
#hostname=
#device=eth0
#autoconf=off
root_dir=/srv/nfs/bbb-rootfs
#nfs_options=,vers=3
#nfsrootfstype=ext4 rootwait fixrtc
tftp_dir=bbb-boot/

#Use this to troubleshoot over serial
#optargs=nfsrootdebug

##use uname_r= only if TFTP SERVER is setup for uname_r boot:
uname_r=4.19.94-ti-r42

U-Boot will recognize the file on each boot, load the environment variables, and run the command to boot off the network! Reboot the Target and check for the marker file for confirmation. You should also see the NFS mount in the file system table:

you@target:~$ mount | grep nfs

Setting a Mount for uEnv.txt Maintenance

To disable or modify the settings of the NFS boot, you need access to the uEnv.txt file on the local memory that is no longer the active root. Search for the appropriate unmounted device. Use commands similar to these:

you@target:~$ ls -l /mmc*
you@target:~$ mount | grep mmc

The unmounted device on the BBB is likely mmcblk1p1 or mmcblk0p1, but your specific Target may have something slightly different.

I like to create a script in the home folder to save time on future maintenance.

you@target:~$ echo "mkdir ~/local-memory-p1" > maint-mount.sh
you@target:~$ echo "sudo mount /dev/mmcblk0p1 ~/local-memory-p1" >> maint-mount.sh
you@target:~$ chmod 700 maint-mount.sh

To change the NFS boot settings, I modify ~/local-memory-p1/uEnv.txt and reboot. To disable NFS boot, I move ~/local-memory-p1/uEnv.txt to ~/local-memory-p1/nfs-uEnv.txt and reboot.

Summary

Using a development board, like the BBB, as a Target system is a great way to develop integrations between hardware and software. Your efficiency will improve, significantly so on larger projects, by leveraging the Target’s network booting feature. This blends the computational and disk access power of a server with the hardware reference platform of the Target.