WORK-IN-PROGRESS, FOLLOW WITH CAUTION!
FooCabFirmwareHowTo describes how we create firmware images for the AlixCab, WgtCab, and AcctonCab devices we've been working on. It should be suitable for following along at home. This is the document to read if you want to build your own node!
Contents
Historical Context
The AlixCab and WgtCab are named after the CloneArmyBox and subsequent NewCloneArmyBox (or NuCab) they are meant to replace. These predecessors were based on recycled PCs that we obtained for free and, running Debian GNU/Linux, repurposed as routers/gateway devices for our network installations. The AlixCab and WgtCab are based on the Alix single-board computer (from PC Engines in Zurich Switzerland) and the Netgear WGT634U, respectively. PersonalTelco is (as of late June 2009) part way through a purchase of 30 Alix boards. We already own or have deployed a number of Netgear WGT634Us. PersonalTelco has some Soekris board (typically as MetrixKits) deployed about as well, running OpenWrt firmware. The same firmware (or slight permutations) as described here can also run on the Soekris devices.
Note that the Netgear WGT634U is a somewhat special device. The device has 8 megabytes of flash storage and has 32 megabytes of RAM, which is rare these days. The image produced with this HowTo will not fit on a 4 megabyte flash device.
OpenWrt
The respective firmware images for the AlixCab and WgtCab are both based on OpenWrt.
In order to take advantage of the newest and shiniest features and bugfixes, as well as to add our own packages to images, we build the firmware ourselves rather than relying on specific binary image releases of OpenWrt. If you have (or can get easily) a Linux box with a few development tools, you can do this yourself without much difficulty. You will also need an internet connection.
OpenWrt is built using a Makefile system called a buildroot. The Makefile infrastructure handles all the details of downloading source code, building the cross-compiling toolchain, compiling the packages and composing image files suitable for flashing (or otherwise writing) onto the device. The buildroot system you obtain from OpenWrt is itself not very large. Instead, it has pointers to the source packages for everything it is going to build, and construction implies retrieving them to a local archive if you haven't already.
We are typically building the most recent (or very nearly) version from OpenWrt's subversion (svn) tree. We are currently (2012) using a git version that synchronizes with the svn tree every 10 minutes.
Bleeding Edge
Building the current version of a developing code base can sometimes not work. Don't be afraid to report non-workingness, using the Trac ticket system or on the freenode Irc channel #openwrt-devel. Over time, bit by bit, you learn about how the buildroot works and can begin to repair problems yourself and submit patches. Don't be afraid. Try, learn, develop. The advantage of building from the main development trunk over one of the "more stable" trees (such as Backfire) is that we can help ensure that we find bugs in the main development branch so that our firmware continues to work in the future. If we hug too closely to "safety" we risk a longer delay between breakage and detection of that breakage, meaning more work to correct the breakage. So far, this has worked out well. We do sometimes encounter breakage, but if we notice and report it, and work to help fix it, it gets fixed faster and everyone is happy.
Checkout
There are often many ways of doing things and this is no exception. We'll show how we do it. If you like another way better, and it works for you, feel free to deviate. First thing to do is to check out OpenWrt's current "trunk" buildroot verion. In a directory where you ordinarily build software (we'll use /src in the examples here) and where you have at least several gigabytes of free disk space:
cd /src git clone git://nbd.name/openwrt.git
Then, change directories into your shiney new buildroot, what is known in OpenWrt as $(TOPDIR), or in our case, /src/openwrt:
cd openwrt
Configuration
- The first thing to do after checking out the buildroot is to set up any feeds you want to use. A feed is basically a URL to a tree of packages that can be built using the buildroot. The primary feed is one called "packages". You almost certainly want to enable that one. Feeds are configured in a file called $(TOPDIR)/feeds.conf. The newly checked out tree will contain a file called feeds.conf.default, which looks like this:
src-svn packages svn://svn.openwrt.org/openwrt/packages src-svn xwrt http://x-wrt.googlecode.com/svn/trunk/package src-svn luci http://svn.luci.subsignal.org/luci/branches/luci-0.8/contrib/package #src-svn phone svn://svn.openwrt.org/openwrt/feeds/phone #src-svn efl svn://svn.openwrt.org/openwrt/feeds/efl #src-svn desktop svn://svn.openwrt.org/openwrt/feeds/desktop #src-svn xfce svn://svn.openwrt.org/openwrt/feeds/xfce
src-git packages git://nbd.name/packages.git
scripts/feeds update -a scripts/feeds install -a
The OpenWrt buildroot uses a linux kconfig-style configuration interface and stores the package configuration in a file called .config. Here is a minimal example for the Netgear WGT634U.
wget http://www.personaltelco.net/~russell/wgtcab-2011-08-19.config mv wgtcab-2011-08-19.config .config make oldconfig
make menuconfig
Note that selecting <M> for a package will build the package but not include the package in the image. Selecting <*> for a package builds the package and includes it in the image. Non-included packages can be added later using the opkg utility from the running image. Lastly, custom files can be added to the image by placing them in $(TOPDIR)/files/. These files are copied in to the filesystem tree before the image is constructed, but after any package files are copied. This allows you to override package default configurations. There is not a mechanism at the present time for *removing* files. To generate a suitable tree of files, check out the configurator:
cd /src git clone git://git.personaltelco.net/files-master cd files-master
git checkout -b my-new-node vi nodedb.txt git commit -a
perl FOOCAB.pl --node <mynodename>
perl FOOCAB.pl --host <myhostname>
Then move the generated tree into the OpenWrt buildroot tree:
mv output $(TOPDIR)/files/
If you want to be part of the PersonalTelcoVPN (and why wouldn't you?), arrange to get an OpenVpn key from a NetworkOperationsTeam member and add that to your $(TOPDIR)/files tree. To create an openvpn key, log into donk, and run:
mkvpnclient <myhostname>
Answer the default to everything except "Organizational Unit Name (eg, section) []:" (say "WgtCab" or "AlixCab" as appropriate), and for "Common Name" use the hostname with no domain. Then "y" for "Sign the certificate?" and "commit?" This results in two files: myhostname.key and myhostname.crt, which should be copied to $(TOPDIR)/files/etc/openvpn/keys/.
Finally, because the WgtCabs lack a realtime clock, it is quite possible that the clock won't get sync'd up at boot. In that case, you'll find that OpenVpn will gag on its key if the clock is wrong. The tunnel won't come up until the time range of the key becomes valid. The way we've dealt with this problem is with an init script (/etc/init.d/initclock) that sets to time to within the time range where the key is valid. The convention has been to use the first of the month following the one that the key was created to ensure validity. This will allow the tunnel to come up so that we can log in remotely and fix the clock manually.
Building
- To build the cross-compiling toolchain, packages and image, execute the following command (adjust the -j value as appropriate, 17 works for me on a hyperthreaded quad-core):
make BUILD_LOG=1 IGNORE_ERRORS=m V=99 -j17
[...] Generating index for package ./wireless-tools_29-4_mipsel.ipk Generating index for package ./zlib_1.2.3-5_mipsel.ipk make[2]: Leaving directory `/src/openwrt' make[1]: Leaving directory `/src/openwrt'
On a quad-core Intel Core i7, the WgtCab build finishes in about 17 minutes, your mileage may vary. You can look in $(TOPDIR)/bin/brcm47xx/ and find the images that were built, and in $(TOPDIR)/bin/brcm47xx/packages/ you'll find the package files you just built.
Flashing
Flashing method depends on the device. There are multiple ways of flashing most devices, some more relevant than others in various contexts. For example, WGT634U from a previous version of OpenWrt is going to probably have an easier method than if you start with Netgear stock firmware. More at OpenWrt's wiki: Installing OpenWrt (generic)
Updating r18781 FooCab firmware on WGT634U, Special Case
Modern WGT634U images can use the generic sysupgrade utility to flash a new TRX image. However, the state of sysupgrade on WgtCab of the previous generation was not fully functional (with regard to process shutting down, rootfs pivoting, etc). Also, the "mtd -r write <newimage.trx> linux" method is dangerous because some processes are still periodically looking on the flash for bits, and this will result in a kernel panic mid-flash (and a "brick") if you don't take precautions. Through trial and error, we have discovered the steps to flash an r18781 in a reasonably safe manner. Here are the steps:
ssh root@<device-to-flash>
- cd /tmp
scp <yourusername>@<yourbuildhost>:<path-to-trx-image> /tmp/
- /etc/init.d/snmpd stop
- /etc/init.d/cron stop
- /etc/init.d/nocatauth-gateway stop
Optionally, if you can connect directly to a public ipv4 address, you can shutdown the VPN. This doesn't seem to be necessary, and might be dangerous if you are not on-site. If there is no public interface that doesn't go through the VPN tunnel, DO NOT DO THIS. It's done simply in the interest of shutting down as many programs as possible.
- /etc/init.d/olsrd stop
- /etc/init.d/openvpn stop
Then, flash the new image using the mtd utility:
- mtd -r write openwrt-brcm47xx-squashfs.trx linux
Watch the upgrade process. When the flash is complete, it will announce it is rebooting. Wait a few (3-5) minutes and try ssh'ing back into the device. Complete any necessary configuration tweaks, e.g. the SSID. It might also be necessary to restart SNMPD as it seems to start up before the VPN tunnel has been built on the first boot and so not all the interfaces will be reported.
- /etc/init.d/snmpd restart
If you mess up and the WGT panics or is otherwise interrupted during the flash, you will need a serial console to recover the device. RussellSenior has a pile of cell-phone data cables he got from China for $3/each that can be hacked for use as a USB-serial console cable. They took nearly 3 months to arrive, so don't try that at home if you are in a hurry. Contact Russell for help getting the serial console set up, or to have him flash your device for you.