• FooCabFirmwareHowTo

WORK-IN-PROGRESS, FOLLOW WITH CAUTION!

FooCabFirmwareHowTo describes how we create firmware images for the PC Engines APU (and APU2), Ubiquiti ERX, D-Link DIR860L-B1, Meraki MR24, Netgear WNDR3800, Buffalo WZR600DHP, TP-Link WDR3600, and similar devices we have been working on. This document is primarily for describing what we do. This is the document to read if you want to try building the software yourself!

Historical Context

• The FooCab is named after the CloneArmyBox and subsequent NewCloneArmyBox (or NuCab) they are meant to replace. These predecessors were based on recycled PCs that we obtained for free and, running Debian GNU/Linux, repurposed as routers/gateway devices for our network installations. More modern FooCab's are based on OpenWrt-supported hardware and run a particular set of packages and custom configuration on top of OpenWrt.

OpenWrt Project

• In order to take advantage of the newest and shiniest features and bugfixes, as well as to add our own packages to images, we build the firmware ourselves rather than relying on specific binary image releases of OpenWrt. If you have (or can get easily) a Linux box with a few development tools, you can do this yourself without much difficulty. You will also need an internet connection.

  OpenWrt is built using a Makefile system called a buildroot. The Makefile infrastructure handles all the details of downloading source code, building the cross-compiling toolchain, compiling the packages and composing image files suitable for flashing (or otherwise writing) onto the device. The buildroot system you obtain from OpenWrt is itself not very large. Instead, it has pointers to the source packages for everything it is going to build, and construction implies retrieving them to a local archive if you haven't already.

  We are typically building the most recent (or very nearly) version from OpenWrt's git tree.

• OpenWrt Buildroot – About

• OpenWrt Buildroot – Installation

• OpenWrt Feeds

• OpenWrt Buildroot – Usage

Bleeding Edge

• Building the current version of a developing code base can sometimes not work. Don't be afraid to report non-workingness. Over time, bit by bit, you learn about how the buildroot works and can begin to repair problems yourself and submit patches. Don't be afraid. Try, learn, develop. The advantage of building from the main development trunk over one of the releases is that we can help ensure that we find bugs in the main development branch so that our firmware continues to work in the future. If we hug too closely to "safety" we risk a longer delay between breakage and detection of that breakage, meaning much more work to correct the breakage. So far, this has worked out well. We do sometimes encounter breakage, but if we notice and report it, and work to help fix it, it gets fixed faster and everyone is happy.

Checkout

• There are often many ways of doing things and this is no exception. We'll show how we do it. If you like another way better, and it works for you, feel free to deviate. First thing to do is to check out the OpenWrt master branch. In a directory where you ordinarily build software (we'll use ~/src in the examples here) and where you have at least several gigabytes of free disk space:

  • cd ~/src
    git clone https://git.openwrt.org/openwrt/openwrt.git

  Then, change directories into your shiny new buildroot, what is known in OpenWrt as $TOPDIR, or in our case, ~/src/openwrt:

  • cd openwrt

Configuration

• The first thing to do after checking out the buildroot is to set up any feeds you want to use. A feed is basically a URL to a tree of packages that can be built using the buildroot. The primary feed is one called "packages". You almost certainly want to enable that one. Feeds are configured in a file called $TOPDIR/feeds.conf. We are using significantly different feeds from the defaults, so create a file feeds.conf. A good starting point is to copy feeds.conf.default to feeds.conf and add the line:

  • src-git ptpcab https://github.com/RussellSenior/ptpcab.git

  This is what we use:

  • src-git packages https://git.openwrt.org/feed/packages.git
    src-git luci https://git.openwrt.org/project/luci.git
    src-git routing https://git.openwrt.org/feed/routing.git
    src-git telephony https://git.openwrt.org/feed/telephony.git
    src-git ptpcab https://github.com/RussellSenior/ptpcab.git

  Update and install the feeds, thusly:

  • scripts/feeds update -a
    scripts/feeds install -a

  The OpenWrt buildroot uses a linux kconfig-style configuration interface and stores the package configuration in a file called .config. Minimized versions of these files can be found in ptpcab feed, which is checked out into $TOPDIR/feeds/ptpcab. See, for example: $TOPDIR/feeds/ptpcab/ptp-node/files/.config-*. There is a version for each router. Currently the following routers are supported:

  • alix2
  • apu
  • dir860l-b1
  • erx
  • espressobin
  • mr24
  • net48xx
  • rb493g
  • rocket
  • wdr3600
  • wndr3800
  • wzr600dhp

The following devices are resource constrained these days, but can be built by disabling nocatauth and perl and removing the then-unneeded /www files:

• airrouter
• ath25
• wgt634u

• To apply a hardware profile, run a command like the following, replacing ${machinetype} with one of the identifiers from the list above.

  • cp feeds/ptpcab/ptp-node/files/.config-${machinetype} .config
    make defconfig

  If you're using a different hardware platform that isn't on the list above, contact RussellSenior for porting information. If you want to look around or add/remove packages, do:

  • make menuconfig

  Note that selecting <M> for a package will build the package but not include the package in the image. Selecting <*> for a package builds the package and includes it in the image. Non-included packages can be added later using the opkg utility from the running image.

Node-specifics and Customization

• OpenWrt provides a mechanism for overriding the vanilla builds by overlaying custom files in the image by placing them in $TOPDIR/files/. These files are copied in to the filesystem tree before the image is constructed, but after any package files are copied. This allows you to override package default configurations. There is not a mechanism at the present time for *removing* files.

• To generate a suitable tree of files, check out the configurator:

  [Note: before the clone, it is recommended that you check your umask. The recommended value is 0022, which will produce user writable files but not group writable files. Git apparently only tracks the execute bit, not the group bit.]

  • cd /src
    git clone git://github.com/personaltelco/ptp-openwrt-files.git
    cd ptp-openwrt-files

  The splash page files are constructed using node.js software. However, unless you modify the source code of these, you needn't build that, a static versions of those files is included in the ptp-openwrt-files git repository.
• If necessary, add (or arrange to have added) your node's information to the PTP API manager. The script you just checked out pulls data from the API to populate a templating system to create the proper (or very nearly proper) $TOPDIR/files overlay.

  NOTE: the configurator now (as of July 2017) assumes some "key" infrastructure that third parties probably won't have. (A TODO item is probably to have it do something sensible in the absence of the key infrastructure):

  • generating a root password hash for the device, pulls the value from a local file which the builder must populate
  • obtaining a dropbear hostkey from a local repository, which the builder must populate, pointed at with the HOSTKEYDIR shell environment variable (we are building the hostkeys on the build host, because of the better entropy on a non-embedded system);

  • obtaining a key/crt pair and a ca.crt for use with the Personal Telco OpenVPN infrastructure, which the builder must populate, pointed at with the PKI shell environment variable.

  Then run the configurator:

  • perl FOOCAB.pl --node <mynodename>

  or

  • perl FOOCAB.pl --host <myhostname>

  Then move the generated tree into the OpenWrt buildroot tree:

  • mv output $TOPDIR/files/

• If you want to be part of the PersonalTelcoVPN (and why wouldn't you?), your key needs to be signed by a PTP certification authority. Generate a csr and send it to Russell or someone else with a sub-ca signing key, and they will give you back a ${hostname}.crt and a ca.crt. We are using the easyrsa3 toolkit these days.
• As generated, the $TOPDIR/files tree contains some things that aren't quite production ready. You may want to edit them by hand before building. If so, it's advisable to keep a version as-generated so that it's easy to compare with a recursive diff.

  • cp -a $TOPDIR/files $TOPDIR/files-orig

• One thing you will almost certainly want to change, when you are ready, is the SSID set on the wifi. It will be generated as "test.personaltelco.net/$DEVICE" where $DEVICE is something like ALIX or WDR3600. It is generated this way because in Russell's build environment he doesn't want his testing infrastructure to be confused with his active node. The file to change is $TOPDIR/files/etc/uci-defaults/ptp.defaults.
• If you have a private network configured on the network, you can look in $TOPDIR/files/etc/uci-defaults/ptp.private.defaults.

• There is a system for reporting IP addresses back to the mothership for the purpose of updating dynamic DNS. This system uses a cryptographic hash which relies on a shared secret for validation purposes (i.e. to prevent hijacking dynamic DNS). Please contact RussellSenior to coordinate on that shared secret.

• Lastly, be sure that a valid ssh public key for you has been added to $TOPDIR/files/etc/dropbear/authorized_keys, and that the permissions on that file and the $TOPDIR/files/etc/dropbear directory are not group writable. If you don't know the root password and dropbear refuses your public key, you will need help or a serial console to get back into the device.

Building

• As part of the initial build process, the make system will download the necessary source tarballs, git checkouts or what-have-you, for the software you are building, and it will place them in a directory $(TOPDIR)/dl. If you have a persistent location for these, you may want to create a symlink in $TOPDIR. That way, if you mess up the git checkout tree, you can blow it away and not waste all that downloading effort. E.g. something like this (assuming that directory exists):

  • ln -s ../package-source dl

  To build the cross-compiling toolchain, packages and image, execute the following command (adjust the -j value as appropriate, I've had best results matching the number of cores, ignoring hyperthreading):

  • make -j16 BUILD_LOG=1 IGNORE_ERRORS=m V=99

  If things go reasonably well, the last thing you'll see is a list vaguely similar to this (filenames will vary by target device):

  • [...]
    Generating index for package ./wireless-tools_29-4_mipsel.ipk
    Generating index for package ./zlib_1.2.3-5_mipsel.ipk
    make[2]: Leaving directory `~/src/lede'
    make[1]: Leaving directory `~/src/lede'

  If things don't go well, you can use my one-liner to try to identify the package that failed to build:

  • for i in $(find logs -name compile.txt) ; do tail $i | grep -H --label=$i 'Error ' ; done | grep -v ignored | awk -F: '{ print $1 }' | sort -u

  then look at the relevant build log file and try to deduce what went wrong. If a package build fails consistently, you can try a clean/compile sequence, like this:

  • make -j16 BUILD_LOG=1 IGNORE_ERRORS=m V=99 package/<pkgname>/{clean,compile}

  if you are desperate, you can rebuild everything relevant to the target device like so:

  • make -j16 BUILD_LOG=1 IGNORE_ERRORS=m V=99 dirclean world

  On a quad-core Intel Core i7, the WgtCab build finishes in about 17 minutes, your mileage may vary. You can look in $TOPDIR/bin/ and find the images and packages that were built.

Flashing

For updating an exiting OpenWrt installation, you should scp your image in $TOPDIR/bin/targets/$arch/$profile/openwrt-*-squashfs-sysupgrade.bin to the device's /tmp/ directory and then run sysupgrade -v -n /tmp/openwrt*.bin. For devices with stock firmware, flashing method depends on the device. There are multiple ways of flashing most devices, some are more relevant than others in various contexts. More at OpenWrt's wiki: Installing OpenWrt (generic)