Location: NodeLuckyLabNW
Date and Time: Wednesday, August 10, 2022, 6:30pm-8:00pm
Scribe: You
Roll Call: RussellSenior,JasonBergstrom,MatthewKlug,SteveTree,TomasKuchta,Wilf
Agenda
- Pick location for next weekly
- Node notes:
NodeAlhambra -- initial gateway plugged in, 4 programmed Meraki MR24's exchanged with the node host, cable runs in progress, outdoor APs on the way.
Realtek-based switch, Engenius EWS2910P (v1 and v3) were flashed, one discovered to be new to OpenWrt and some reverse-engineering ensued to support PoE.
- Long standing question tested: Will vlan tags survive contact with an unmanaged switch? It depends:
- Netgear GS605v3: No
- Netgear FS105v2: Yes
- TP-Link SG105v6.6: Yes
- Netgear GS308v3: Yes
- intermittent uptime on ubiquiti bullet-ac (10.11.104.19)
- bad connectivity to metrix ballroom (10.11.104.2)
- no connectivity to unifi-ac-mesh on Commons (10.11.104.6) need building access to fix.
lost connectivity to moose (at NodeFreshPot)
NodeFreshPot -- down since July 30, needs a visit to investigate
NodeSuzette -- down since closure of restaurant, need to reach out to see about gear or new tenant
NodeKlickitat -- a couple of the ground-level mesh nodes are down, Russell needs to reach out to see about rebooting/repowering
NodeC3PO -- since it is shutdown now, we should see about decommissioning the backhaul radio on host's roof, or maybe repurposing it (in which case, a beefier radio is probably warranted, it is memory constrained)?
- Russell spent Monday afternoon refining the MerakiMR24 flashing instructions and also paving over 4 different versions of the Recovery firmware built into (38 of 40) the MR24's in the reserve inventory. There are additional MR24s in the field, which perhaps should have their Recovery firmware updated as well, including:
ap1 and ap2 at NodeBrunner;
ap1 through ap5 at NodeBuffaloGap;
ap1 and ap2 at NodeEcotrust;
ap1 through ap3 at NodeGooseHollow;
ap1 and ap2 at NodeGriswold;
ap1 and ap2 at NodeLuckyLab;
ap1 and ap2 at NodeLuckyLabSW;
ap1 and ap2 at NodeStreetRoots;
ap1 at NodeTibbetts;
- kernel 4.4.21 : Sat Sep 17 19:43:29 2016 -- LEDE (old u-boot-env variables);
- kernel 4.9.20 : Tue Apr 04 13:06:14 2017 -- LEDE (old u-boot-env variables);
kernel 4.14.20 : Thu Feb 22 22:40:16 2018 -- OpenWrt (old u-boot-env variables);
kernel 4.19.72 : Sat Sep 14 17:38:48 2019 -- OpenWrt (SSID: OpenWrt currrent u-boot-env variables, includes a firewall which makes it unreachable from ethernet side);
kernel 5.10.134 : Tue Aug 02 08:23:52 2022 -- OpenWrt (SSID: RECOVERY, no firewall, no password)
- 7d41197c ; 20160917-194329 ; 331 LEBs
- 94f85921 ; 20160925-073055 ; 331 LEBs
- fef78d95 ; 20170404-130614 ; 331 LEBs
- eaed259b ; 20180222-224016 ; 331 LEBs
- 44f2e744 ; 20180222-224016 ; 331 LEBs
- a46e503a ; 20190914-173848 ; 267 LEBs
- Checked in on the 5 remaining Ubiquiti ER-X gateways in the field, and none of them are showing squashfs NAND errors we were seeing previously, so it might be whatever the bug we saw earlier was magically corrected when we weren't looking
- Wiline needs a query about our machine there
Your Item Here
Notes
- Node notes:
NodeAlhambra - Russell visited Wilf on Saturday, took spare APU2, configured 4 MR24's, exchanged for unconfigured versions, CenturyLink upstream, put APU2 on basement board by ONT.
- Wilf has the blue-handled PTP/Russ's crimper
EnGenius 2910P switch - Russell has one v3, supported by OpenWRT (v1), Wilf had a v1 that worked, the v3 has a different PSE (realtek -115.2K as opposed to broadcom -19.2K) stm32 microcontroller. Protocol is also slightly different, but works.
- v1 device suggested web interface was the way to upload firmware, but did not work.
- vlan tag stripping on unmanaged switch, have chips from managed switches?
- Freshpot - can't access either locally or over network
NodeSuzette - we don't know new tenant
- NodeC3PO - limited memory on current point to point link, we could reconfig to provide wifi to Water Ave
- MR24 - Russell reflashed onsite nodes, though many are still in boxes
- Russell updated install instructions, and firmware updates on MR24 page
- Access recovery mode by holding button, or nuke active kernel
- Recovery volume has grown, need to steal space from some active partition, like kernel which is only read during boot
- Dumb APs will request another DHCP lease, challenge with VLANs
- Some node upgrades can be done remotely, but we can also swap out some APs
Russell has found 4 different firmware versions in the field, 2 more are at Russell's and at NodeAlhambra
- ERX check - no issues, ~40 days
- Wiline - needs a call about our server
- Eric Rosenberry, NWAX - now at Ziply, mutual suggestions we should follow Jared Mauch's model
StreetRoots - bought a new building on Burnside - their ad rates are also likely low
- Next week's meeting will be at Batterfish on 34th and Belmont
Postscript
Russell tried to swap out MR24 with the new recovery firmware at NodeLuckyLabNW and found it didn't work, swapped original back in
Investigating at home, discovered some changes in the OpenWrt MR24 (which we tweak) had reconfigured the lan interface (where the wifi interfaces live) as DHCP rather than static and another component turns off dhcp server on non-static interface configurations. This has affected MR24 firmwares probably since last November or so (OpenWrt commit hash abb4083fa), but we hadn't noticed it because it only affected the wireless side in Russell's test bed and that's not heavily tested. Side effect is that the recovery firmware I flashed to all the stockpile MR24s are also affected. This can be corrected relatively easily since the corrected configuration fits in the same number of LEBs (Logical Erase Blocks) so rejiggering the ubi volume sizes is not needed, just a ubiupdatevol command with the corrected firmware, and should go relatively quickly.