• FooCabFirmwareHowTo

WORK-IN-PROGRESS, FOLLOW WITH CAUTION!

FooCabFirmwareHowTo describes how we create firmware images for the Soekris net45xx and net48xx, PC Engines Alix, Meraki MR24, Netgear WGT634U, Accton MR3201A, Netgear WNDR3800, Buffalo WZR600DHP, TP-Link WDR3600, and similar devices we have been working on. This document is primarily for describing what we do. This is the document to read if you want to try building the software yourself!

Historical Context

• The AlixCab and WgtCab are named after the CloneArmyBox and subsequent NewCloneArmyBox (or NuCab) they are meant to replace. These predecessors were based on recycled PCs that we obtained for free and, running Debian GNU/Linux, repurposed as routers/gateway devices for our network installations. The AlixCab and WgtCab are based on the Alix single-board computer (from PC Engines in Zurich Switzerland) and the Netgear WGT634U, respectively. PersonalTelco is (as of late June 2009) part way through a purchase of 30 Alix boards. We already own or have deployed a number of Netgear WGT634Us. PersonalTelco has some Soekris board (typically as MetrixKits) deployed about as well, running OpenWrt firmware. The same firmware (or slight permutations) as described here can also run on the Soekris devices. Note that, as of the summer of 2014 (and for some time), the Netgear WGT634U is no longer stable running our kit of software, mostly from memory pressure. That might be correctable through some debugging and development work, it's not clear. We have been moving away from the WGT634U devices, and would like to retire them from gateway duty. They seem to work fine in "dumb-AP" roles.

OpenWrt/LEDE Project

• The firmware images are based on the LEDE Project. LEDE is a "reboot" of the OpenWrt project that launched in May 2016. It is mostly compatible still. LEDE is getting significantly more work than OpenWrt going forward, so it makes sense to adopt it for our node purposes, however, LEDE still lacks a wiki, so there are still pointers to the OpenWrt wiki documentation, which is still pretty informative. As of Summer 2017, the OpenWrt and LEDE projects are in the process of re-merging. The LEDE project branch will be adopted for the re-merged project. In order to take advantage of the newest and shiniest features and bugfixes, as well as to add our own packages to images, we build the firmware ourselves rather than relying on specific binary image releases of LEDE. If you have (or can get easily) a Linux box with a few development tools, you can do this yourself without much difficulty. You will also need an internet connection. LEDE is built using a Makefile system called a buildroot. The Makefile infrastructure handles all the details of downloading source code, building the cross-compiling toolchain, compiling the packages and composing image files suitable for flashing (or otherwise writing) onto the device. The buildroot system you obtain from LEDE is itself not very large. Instead, it has pointers to the source packages for everything it is going to build, and construction implies retrieving them to a local archive if you haven't already. We are typically building the most recent (or very nearly) version from LEDE's git tree.

• OpenWrt Buildroot – About

• OpenWrt Buildroot – Installation

• OpenWrt Feeds

• OpenWrt Buildroot – Usage

Bleeding Edge

• Building the current version of a developing code base can sometimes not work. Don't be afraid to report non-workingness. Over time, bit by bit, you learn about how the buildroot works and can begin to repair problems yourself and submit patches. Don't be afraid. Try, learn, develop. The advantage of building from the main development trunk over one of the releases is that we can help ensure that we find bugs in the main development branch so that our firmware continues to work in the future. If we hug too closely to "safety" we risk a longer delay between breakage and detection of that breakage, meaning much more work to correct the breakage. So far, this has worked out well. We do sometimes encounter breakage, but if we notice and report it, and work to help fix it, it gets fixed faster and everyone is happy.

Checkout

• There are often many ways of doing things and this is no exception. We'll show how we do it. If you like another way better, and it works for you, feel free to deviate. First thing to do is to check out the LEDE master branch. In a directory where you ordinarily build software (we'll use ~/src in the examples here) and where you have at least several gigabytes of free disk space:

  • cd ~/src
    git clone git://git.lede-project.org/source.git lede

  Then, change directories into your shiny new buildroot, what is known in LEDE as $TOPDIR, or in our case, ~/src/lede:

  • cd lede

Configuration

• The first thing to do after checking out the buildroot is to set up any feeds you want to use. LEDE is using the same feeds as OpenWrt, and they work the same and compatibly. A feed is basically a URL to a tree of packages that can be built using the buildroot. The primary feed is one called "packages". You almost certainly want to enable that one. Feeds are configured in a file called $TOPDIR/feeds.conf. We are using significantly different feeds from the defaults, so create a file feeds.conf. This is what we use:

  • src-git packages https://github.com/openwrt/packages.git
    src-git luci https://github.com/openwrt/luci.git
    src-git routing https://github.com/openwrt-routing/packages.git
    src-git telephony https://github.com/openwrt/telephony.git
    src-git management https://github.com/openwrt-management/packages.git
    src-git targets https://github.com/openwrt/targets.git
    src-git oldpackages http://git.openwrt.org/packages.git
    src-git batman git://git.open-mesh.org/openwrt-feed-batman-adv.git
    src-git ptpcab git://github.com/RussellSenior/ptpcab.git

  Update and install the feeds, thusly:

  • scripts/feeds update -a
    scripts/feeds install -a

  The LEDE buildroot uses a linux kconfig-style configuration interface and stores the package configuration in a file called .config. Minimized versions of these files can be found in ptpcab feed, which is checked out into $TOPDIR/feeds/ptpcab. See, for example: $TOPDIR/feeds/ptpcab/ptp-node/files/.config-*. There is a version for each router. Currently the following routers are supported:
  • airrouter
  • alix2
  • apu
  • ath25
  • dir860l
  • mikrotik
  • net45xx
  • net48xx
  • routerstation
  • wndr3800
  • wdr3600
  • wzr600dhp
  • wgt634u
  To apply a hardware profile, run a command like the following, replacing ${machinetype} with one of the identifiers from the list above.

  • cp feeds/ptpcab/ptp-node/files/.config-${machinetype} .config
    make defconfig

  If you're using a different hardware platform that isn't on the list above, contact RussellSenior for porting information. If you want to look around or add/remove packages, do:

  • make menuconfig

  Note that selecting <M> for a package will build the package but not include the package in the image. Selecting <*> for a package builds the package and includes it in the image. Non-included packages can be added later using the opkg utility from the running image.

Node-specifics and Customization

• OpenWrt/LEDE provides a mechanism for overriding the vanilla builds by overlaying custom files in the image by placing them in $TOPDIR/files/. These files are copied in to the filesystem tree before the image is constructed, but after any package files are copied. This allows you to override package default configurations. There is not a mechanism at the present time for *removing* files.
• To generate a suitable tree of files, check out the configurator:

  [Note: before the clone, it is recommended that you check your umask. The recommended value is 0022, which will produce user writable files but not group writable files. Git apparently only tracks the execute bit, not the group bit.]

  • cd /src
    git clone git://github.com/personaltelco/ptp-openwrt-files.git
    cd ptp-openwrt-files

  The splash page files are constructed using node.js software. However, unless you modify the source code of these, you needn't build that, a static versions of those files is included in the ptp-openwrt-files git repository.
• If necessary, add (or arrange to have added) your node's information to the PTP API manager. The script you just checked out pulls data from the API to populate a templating system to create the proper (or very nearly proper) $TOPDIR/files overlay.

  NOTE: the configurator now (as of July 2017) assumes some "key" infrastructure that third parties probably won't have. (A TODO item is probably to have it do something sensible in the absence of the key infrastructure):

  • generating a root password hash for the device, pulls the value from a local file which the builder must populate
  • obtaining a dropbear hostkey from a local repository, which the builder must populate, pointed at with the HOSTKEYDIR shell environment variable (we are building the hostkeys on the build host, because of the better entropy on a non-embedded system);

  • obtaining a key/crt pair and a ca.crt for use with the Personal Telco OpenVPN infrastructure, which the builder must populate, pointed at with the PKI shell environment variable.

  Then run the configurator:

  • perl FOOCAB.pl --node <mynodename>

  or

  • perl FOOCAB.pl --host <myhostname>

  Then move the generated tree into the OpenWrt buildroot tree:

  • mv output $TOPDIR/files/

• If you want to be part of the PersonalTelcoVPN (and why wouldn't you?), your key needs to be signed by a PTP certification authority. Generate a csr and send it to Russell or someone else with a sub-ca signing key, and they will give you back a ${hostname}.crt and a ca.crt. We are using the easyrsa3 toolkit these days.
• As generated, the $TOPDIR/files tree contains some things that aren't quite production ready. You may want to edit them by hand before building. If so, it's advisable to keep a version as-generated so that it's easy to compare with a recursive diff.

  • cp -a $TOPDIR/files $TOPDIR/files-orig

• One thing you will almost certainly want to change, when you are ready, is the SSID set on the wifi. It will be generated as "test.personaltelco.net/$DEVICE" where $DEVICE is something like ALIX or WDR3600. It is generated this way because in Russell's build environment he doesn't want his testing infrastructure to be confused with his active node. The file to change is $TOPDIR/files/etc/uci-defaults/ptp.defaults.
• If you have a private network configured on the network, you can look in $TOPDIR/files/etc/uci-defaults/ptp.private.defaults.

• There is a system for reporting IP addresses back to the mothership for the purpose of updating dynamic DNS. This system uses a cryptographic hash which relies on a shared secret for validation purposes (i.e. to prevent hijacking dynamic DNS). Please contact RussellSenior to coordinate on that shared secret.

• Lastly, be sure that a valid ssh public key for you has been added to $TOPDIR/files/etc/dropbear/authorized_keys, and that the permissions on that file and the $TOPDIR/files/etc/dropbear directory are not group writable. If you don't know the root password and dropbear refuses your public key, you will need help or a serial console to get back into the device.

Building

• As part of the initial build process, the make system will download the necessary source tarballs, git checkouts or what-have-you, for the software you are building, and it will place them in a directory $(TOPDIR)/dl. If you have a persistent location for these, you may want to create a symlink in $TOPDIR. That way, if you mess up the git checkout tree, you can blow it away and not waste all that downloading effort. E.g. something like this (assuming that directory exists):

  • ln -s ../package-source dl

  To build the cross-compiling toolchain, packages and image, execute the following command (adjust the -j value as appropriate, I've had best results matching the number of cores, ignoring hyperthreading):

  • make -j16 BUILD_LOG=1 IGNORE_ERRORS=m V=99

  If things go reasonably well, the last thing you'll see is a list vaguely similar to this (filenames will vary by target device):

  • [...]
    Generating index for package ./wireless-tools_29-4_mipsel.ipk
    Generating index for package ./zlib_1.2.3-5_mipsel.ipk
    make[2]: Leaving directory `~/src/lede'
    make[1]: Leaving directory `~/src/lede'

  If things don't go well, you can use my one-liner to try to identify the package that failed to build:

  • for i in $(find logs -name compile.txt) ; do tail $i | grep -H --label=$i 'Error ' ; done | grep -v ignored | awk -F: '{ print $1 }' | sort -u

  then look at the relevant build log file and try to deduce what went wrong. If a package build fails consistently, you can try a clean/compile sequence, like this:

  • make -j16 BUILD_LOG=1 IGNORE_ERRORS=m V=99 package/<pkgname>/{clean,compile}

  if you are desperate, you can rebuild everything relevant to the target device like so:

  • make -j16 BUILD_LOG=1 IGNORE_ERRORS=m V=99 dirclean world

  On a quad-core Intel Core i7, the WgtCab build finishes in about 17 minutes, your mileage may vary. You can look in $TOPDIR/bin/ and find the images and packages that were built.

Flashing

For updating an exiting OpenWrt or LEDE installation, you should scp your image in $TOPDIR/bin/targets/$arch/$profile/lede-*-squashfs-sysupgrade.bin to the device's /tmp/ directory and then run sysupgrade -v -n /tmp/lede*.bin. For devices with stock firmware, flashing method depends on the device. There are multiple ways of flashing most devices, some are more relevant than others in various contexts. More at OpenWrt's wiki: Installing OpenWrt (generic)