Location: Jitsi
Date and Time: Wednesday, February 9, 2022, 5:30pm-8:00pm
Scribe: You
Roll Call: RussellSenior,TomasKuchta,JasonBergstrom,TedBrunner,SteveTree
Agenda
- Pick location for next weekly
- Dell R630:
- Debian installed
- VMs copied from magenta
- VMs started and recovered from brokenness
- larger disks are ... slow, I think all three RAID1s are composed of 5400 rpm disks, the 4TB pair seem to stall periodically, about 60MB/s is typical write speed.
- One of the VMs experienced some stalls:
[Sun Feb 6 16:54:42 2022] rcu: INFO: rcu_sched self-detected stall on CPU [Sun Feb 6 16:54:42 2022] rcu: 1-...0: (1 ticks this GP) idle=376/1/0x4000000000000000 softirq=2783997/2783997 fqs=16 [Sun Feb 6 16:54:42 2022] (t=6597 jiffies g=5323225 q=2310) [Sun Feb 6 16:54:42 2022] NMI backtrace for cpu 1 [Sun Feb 6 16:54:42 2022] CPU: 1 PID: 524145 Comm: kworker/1:2 Tainted: G L 5.10.0-11-amd64 #1 Debian 5.10.92-1 [Sun Feb 6 16:54:42 2022] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014 [Sun Feb 6 16:54:42 2022] Workqueue: events drm_fb_helper_dirty_work [drm_kms_helper] [Sun Feb 6 16:54:42 2022] Call Trace: [Sun Feb 6 16:54:42 2022] <IRQ> [Sun Feb 6 16:54:42 2022] dump_stack+0x6b/0x83 [Sun Feb 6 16:54:42 2022] nmi_cpu_backtrace.cold+0x32/0x69 [Sun Feb 6 16:54:42 2022] ? lapic_can_unplug_cpu+0x80/0x80 [Sun Feb 6 16:54:42 2022] nmi_trigger_cpumask_backtrace+0xd7/0xe0 [Sun Feb 6 16:54:42 2022] rcu_dump_cpu_stacks+0xa2/0xd0 [Sun Feb 6 16:54:42 2022] rcu_sched_clock_irq.cold+0x1ff/0x3d6 [Sun Feb 6 16:54:42 2022] update_process_times+0x8c/0xc0 [Sun Feb 6 16:54:42 2022] tick_sched_handle+0x22/0x60 [Sun Feb 6 16:54:42 2022] tick_sched_timer+0x7c/0xb0 [Sun Feb 6 16:54:42 2022] ? tick_do_update_jiffies64.part.0+0xc0/0xc0 [Sun Feb 6 16:54:42 2022] __hrtimer_run_queues+0x12a/0x270 [Sun Feb 6 16:54:42 2022] hrtimer_interrupt+0x110/0x2c0 [Sun Feb 6 16:54:42 2022] __sysvec_apic_timer_interrupt+0x5f/0xd0 [Sun Feb 6 16:54:42 2022] asm_call_irq_on_stack+0x12/0x20 [Sun Feb 6 16:54:42 2022] </IRQ> [Sun Feb 6 16:54:42 2022] sysvec_apic_timer_interrupt+0x72/0x80 [Sun Feb 6 16:54:42 2022] asm_sysvec_apic_timer_interrupt+0x12/0x20 [Sun Feb 6 16:54:42 2022] RIP: 0010:_raw_spin_unlock_irqrestore+0x11/0x20 [Sun Feb 6 16:54:42 2022] Code: e0 4c 39 f0 76 d8 80 4d 00 04 eb 92 cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 07 [Sun Feb 6 16:54:42 2022] RSP: 0018:ffffad65404cbc80 EFLAGS: 00000246 [Sun Feb 6 16:54:42 2022] RAX: 0000000000000000 RBX: ffff989c06c45100 RCX: 0000000000000000 [Sun Feb 6 16:54:42 2022] RDX: 000000000000c040 RSI: 0000000000000246 RDI: 0000000000000246 [Sun Feb 6 16:54:42 2022] RBP: ffffad65402a700c R08: 0000000000000300 R09: ffff989c03948e20 [Sun Feb 6 16:54:42 2022] R10: ffff989c029e3980 R11: ffff989d228e5000 R12: ffff989c06c45120 [Sun Feb 6 16:54:42 2022] R13: ffffad65404cbcf8 R14: 0000000000000246 R15: 0000000000000000 [Sun Feb 6 16:54:42 2022] qxl_ring_push+0x8e/0x290 [qxl] [Sun Feb 6 16:54:42 2022] ? ttm_bo_kunmap+0x9f/0xb0 [ttm] [Sun Feb 6 16:54:42 2022] ? ttm_bo_add_mem_to_lru+0x75/0x100 [ttm] [Sun Feb 6 16:54:42 2022] qxl_push_command_ring_release+0x64/0x80 [qxl] [Sun Feb 6 16:54:42 2022] qxl_draw_dirty_fb+0x425/0x450 [qxl] [Sun Feb 6 16:54:42 2022] qxl_framebuffer_surface_dirty+0xf5/0x1c0 [qxl] [Sun Feb 6 16:54:42 2022] drm_fb_helper_dirty_work+0x175/0x1b0 [drm_kms_helper] [Sun Feb 6 16:54:42 2022] process_one_work+0x1b6/0x350 [Sun Feb 6 16:54:42 2022] worker_thread+0x53/0x3e0 [Sun Feb 6 16:54:42 2022] ? process_one_work+0x350/0x350 [Sun Feb 6 16:54:42 2022] kthread+0x11b/0x140 [Sun Feb 6 16:54:42 2022] ? __kthread_bind_mask+0x60/0x60 [Sun Feb 6 16:54:42 2022] ret_from_fork+0x22/0x30
Russell files comments on NTIA IIJA rules, see twitter thread: https://twitter.com/rssenior/status/1489508946888912897
- Noticed that portlandfiber.com, which we host, was busted. Jason recovered a mysql database (or several) and got it working again.
https://olis.oregonlegislature.gov/liz/2022R1/Downloads/MeasureDocument/HB4092/A-Engrossed:
- Section 3:
- (1) The Oregon Broadband Office shall develop a Broadband Action Plan and a Digital Equity Plan that address the requirements of the Infrastructure Investment and Jobs Act (P.L. 117-58).
- [...]
- (4) The office may develop supplemental strategies, as necessary, to achieve availability of services and adoption of services by residents, businesses, schools and libraries throughout Oregon and to support federal planning requirements, including strategies to:
- [...]
- (i) Consider the appropriateness of open access requirements for state-funded projects;
- Section 3:
- Node notes:
NodeStreetRoots -- no progress
NodeCascadia -- no progress
Your Item Here
Notes
- Dell R630 (yellow) progress
- Slow I/O on 5400 SATA drives and also stalls
- Arch (iris) install recovered
- reviewed guest OS kernel thread dumps
- agreed to pick up 2 Samsung 870 EVO 1TB SSDs to replace some quantity of SATA drives
- Russell commented on his NTIA commentary via twitter thread
- State developing bill to receive money from above, noting consideration of open-access (may be an opportunity to testify on its importance)
- portlandfiber.com has old info but died due to upgrade/transaction issue in MySQL
- personaltelco.com/blog works blog.personaltelco.com doesn't work
- Node notes:
- Progress TBD
- one node needed to have snmpd restarted
- C3PO - concern about equipment return due to new management group, also discussed Safe Rest Villages