WORK-IN-PROGRESS, FOLLOW WITH CAUTION!

FooCabFirmwareHowTo describes how we create firmware images for the AlixCab, WgtCab, and AcctonCab devices we've been working on. It should be suitable for following along at home. This is the document to read if you want to build your own node!

Historical Context

• The AlixCab and WgtCab are named after the CloneArmyBox and subsequent NewCloneArmyBox (or NuCab) they are meant to replace. These predecessors were based on recycled PCs that we obtained for free and, running Debian GNU/Linux, repurposed as routers/gateway devices for our network installations. The AlixCab and WgtCab are based on the Alix single-board computer (from PC Engines in Zurich Switzerland) and the Netgear WGT634U, respectively. PersonalTelco is (as of late June 2009) part way through a purchase of 30 Alix boards. We already own or have deployed a number of Netgear WGT634Us. PersonalTelco has some Soekris board (typically as MetrixKits) deployed about as well, running OpenWrt firmware. The same firmware (or slight permutations) as described here can also run on the Soekris devices.

  Note that the Netgear WGT634U is a somewhat special device. The device has 8 megabytes of flash storage and has 32 megabytes of RAM, which is rare these days. The image produced with this HowTo will not fit on a 4 megabyte flash device.

OpenWrt

• The respective firmware images for the AlixCab and WgtCab are both based on OpenWrt.

  In order to take advantage of the newest and shiniest features and bugfixes, as well as to add our own packages to images, we build the firmware ourselves rather than relying on specific binary image releases of OpenWrt. If you have (or can get easily) a Linux box with a few development tools, you can do this yourself without much difficulty. You will also need an internet connection.

  OpenWrt is built using a Makefile system called a buildroot. The Makefile infrastructure handles all the details of downloading source code, building the cross-compiling toolchain, compiling the packages and composing image files suitable for flashing (or otherwise writing) onto the device. The buildroot system you obtain from OpenWrt is itself not very large. Instead, it has pointers to the source packages for everything it is going to build, and construction implies retrieving them to a local archive if you haven't already.

  We are typically building the most recent (or very nearly) version from OpenWrt's subversion (svn) tree. We are currently (2012) using a git version that synchronizes with the svn tree every 10 minutes.

• OpenWrt Buildroot – About

• OpenWrt Buildroot – Installation

• OpenWrt Feeds

• OpenWrt Buildroot – Usage

Bleeding Edge

• Building the current version of a developing code base can sometimes not work. Don't be afraid to report non-workingness, using the Trac ticket system or on the freenode Irc channel #openwrt-devel. Over time, bit by bit, you learn about how the buildroot works and can begin to repair problems yourself and submit patches. Don't be afraid. Try, learn, develop. The advantage of building from the main development trunk over one of the "more stable" trees (such as Backfire) is that we can help ensure that we find bugs in the main development branch so that our firmware continues to work in the future. If we hug too closely to "safety" we risk a longer delay between breakage and detection of that breakage, meaning more work to correct the breakage. So far, this has worked out well. We do sometimes encounter breakage, but if we notice and report it, and work to help fix it, it gets fixed faster and everyone is happy.

Checkout

• There are often many ways of doing things and this is no exception. We'll show how we do it. If you like another way better, and it works for you, feel free to deviate. First thing to do is to check out OpenWrt's current "trunk" buildroot verion. In a directory where you ordinarily build software (we'll use /src in the examples here) and where you have at least several gigabytes of free disk space:

  • cd /src
    git clone git://nbd.name/openwrt.git

  Then, change directories into your shiney new buildroot, what is known in OpenWrt as $(TOPDIR), or in our case, /src/openwrt:

  • cd openwrt

Configuration

• The first thing to do after checking out the buildroot is to set up any feeds you want to use. A feed is basically a URL to a tree of packages that can be built using the buildroot. The primary feed is one called "packages". You almost certainly want to enable that one. Feeds are configured in a file called $(TOPDIR)/feeds.conf. The newly checked out tree will contain a file called feeds.conf.default, which looks like this:

  • src-svn packages svn://svn.openwrt.org/openwrt/packages
    src-svn xwrt http://x-wrt.googlecode.com/svn/trunk/package
    src-svn luci http://svn.luci.subsignal.org/luci/branches/luci-0.8/contrib/package
    #src-svn phone svn://svn.openwrt.org/openwrt/feeds/phone
    #src-svn efl svn://svn.openwrt.org/openwrt/feeds/efl
    #src-svn desktop svn://svn.openwrt.org/openwrt/feeds/desktop
    #src-svn xfce svn://svn.openwrt.org/openwrt/feeds/xfce

  We aren't using anything from the xwrt or luci feeds currently, and we'll change the packages feed to use git instead of svn. So, create a file feeds.conf containing this:

  • src-git packages git://nbd.name/packages.git
    src-git batman http://git.open-mesh.org/openwrt-feed-batman-adv.git
    src-git ptpcab git://github.com/RussellSenior/ptpcab.git

  Install the feeds, thusly:

  • scripts/feeds update -a
    scripts/feeds install -a

  The OpenWrt buildroot uses a linux kconfig-style configuration interface and stores the package configuration in a file called .config. Here is a minimal example for the Netgear WGT634U.

  • wget http://www.personaltelco.net/~russell/wgtcab-2012-09-10.config
    mv wgtcab-2012-09-10.config .config
    make oldconfig

  If you want to look around or add/remove packages, do:

  • make menuconfig

  Note that selecting <M> for a package will build the package but not include the package in the image. Selecting <*> for a package builds the package and includes it in the image. Non-included packages can be added later using the opkg utility from the running image. Lastly, custom files can be added to the image by placing them in $(TOPDIR)/files/. These files are copied in to the filesystem tree before the image is constructed, but after any package files are copied. This allows you to override package default configurations. There is not a mechanism at the present time for *removing* files. To generate a suitable tree of files, check out the configurator:

  • cd /src
    git clone git://git.personaltelco.net/files-master
    cd files-master

  Read the README. If it is necessary to add your node's information to the nodedb.txt (tab-delimited flat file ... make certain you use real tabs!), it is a good idea to create a git branch for your changes:

  • git checkout -b my-new-node
    vi nodedb.txt
    git commit -a 

  If you make changes to the nodedb.txt file for your node, be sure to submit those back to the main tree so that your network allocation is reserved for you. Either prepare a patch using git's excellent tools, or arrange for write access to the git repository and push your changes back. Then run the configurator:

  • perl FOOCAB.pl --node <mynodename>

  or

  • perl FOOCAB.pl --host <myhostname>

  Then move the generated tree into the OpenWrt buildroot tree:

  • mv output $(TOPDIR)/files/

  If you want to be part of the PersonalTelcoVPN (and why wouldn't you?), arrange to get an OpenVpn key from a NetworkOperationsTeam member and add that to your $(TOPDIR)/files tree. To create an openvpn key, log into donk, and run:

  • mkvpnclient <myhostname>

  Answer the default to everything except "Organizational Unit Name (eg, section) []:" (say "WgtCab" or "AlixCab" as appropriate), and for "Common Name" use the hostname with no domain. Then "y" for "Sign the certificate?" and "commit?" This results in two files: myhostname.key and myhostname.crt, which should be copied to $(TOPDIR)/files/etc/openvpn/keys/.

  Finally, because the WgtCabs lack a realtime clock, it is quite possible that the clock won't get sync'd up at boot. In that case, you'll find that OpenVpn will gag on its key if the clock is wrong. The tunnel won't come up until the time range of the key becomes valid. The way we've dealt with this problem is with an init script (/etc/init.d/initclock) that sets to time to within the time range where the key is valid. The convention has been to use the first of the month following the one that the key was created to ensure validity. This will allow the tunnel to come up so that we can log in remotely and fix the clock manually.

Building

• To build the cross-compiling toolchain, packages and image, execute the following command (adjust the -j value as appropriate, 17 works for me on a hyperthreaded quad-core):

  • make BUILD_LOG=1 IGNORE_ERRORS=m V=99 -j17

  If things go reasonably well, the last thing you'll see is a list of:

  • [...]
    Generating index for package ./wireless-tools_29-4_mipsel.ipk
    Generating index for package ./zlib_1.2.3-5_mipsel.ipk
    make[2]: Leaving directory `/src/openwrt'
    make[1]: Leaving directory `/src/openwrt'

  On a quad-core Intel Core i7, the WgtCab build finishes in about 17 minutes, your mileage may vary. You can look in $(TOPDIR)/bin/brcm47xx/ and find the images that were built, and in $(TOPDIR)/bin/brcm47xx/packages/ you'll find the package files you just built.

Flashing

Flashing method depends on the device. There are multiple ways of flashing most devices, some more relevant than others in various contexts. For example, WGT634U from a previous version of OpenWrt is going to probably have an easier method than if you start with Netgear stock firmware. More at OpenWrt's wiki: Installing OpenWrt (generic)

Updating r18781 FooCab firmware on WGT634U, Special Case

• Netgear WGT634U - Installation of OpenWrt

Modern WGT634U images can use the generic sysupgrade utility to flash a new TRX image. However, the state of sysupgrade on WgtCab of the previous generation was not fully functional (with regard to process shutting down, rootfs pivoting, etc). Also, the "mtd -r write <newimage.trx> linux" method is dangerous because some processes are still periodically looking on the flash for bits, and this will result in a kernel panic mid-flash (and a "brick") if you don't take precautions. Through trial and error, we have discovered the steps to flash an r18781 in a reasonably safe manner. Here are the steps:

• ssh root@<device-to-flash>

• cd /tmp

• scp <yourusername>@<yourbuildhost>:<path-to-trx-image> /tmp/

• /etc/init.d/snmpd stop
• /etc/init.d/cron stop
• /etc/init.d/nocatauth-gateway stop

Optionally, if you can connect directly to a public ipv4 address, you can shutdown the VPN. This doesn't seem to be necessary, and might be dangerous if you are not on-site. If there is no public interface that doesn't go through the VPN tunnel, DO NOT DO THIS. It's done simply in the interest of shutting down as many programs as possible.

• /etc/init.d/olsrd stop
• /etc/init.d/openvpn stop

Then, flash the new image using the mtd utility:

• mtd -r write openwrt-brcm47xx-squashfs.trx linux

Watch the upgrade process. When the flash is complete, it will announce it is rebooting. Wait a few (3-5) minutes and try ssh'ing back into the device. Complete any necessary configuration tweaks, e.g. the SSID. It might also be necessary to restart SNMPD as it seems to start up before the VPN tunnel has been built on the first boot and so not all the interfaces will be reported.

• /etc/init.d/snmpd restart

If you mess up and the WGT panics or is otherwise interrupted during the flash, you will need a serial console to recover the device. RussellSenior has a pile of cell-phone data cables he got from China for $3/each that can be hacked for use as a USB-serial console cable. They took nearly 3 months to arrive, so don't try that at home if you are in a hurry. Contact Russell for help getting the serial console set up, or to have him flash your device for you.