BALUG VM was down for fair while earlier today. Has now been up again for over 7 hours now. - BALUG-Admin

3 Sep 2018


      BALUG VM was down for fair while earlier today.
Has now been up again for over 7 hours now.
Looks like there was an I/O hiccup on the physical host,
which didn't particularly impact the physical hosts, but
was enough of an interruption (delay) that the BALUG VM kernel
paniced.
Did have a 3rd hard drive testing, etc. on the physical host
at the time ... might've hit issues and possibly it did a bus
reset?  Who knows for sure.  Anyway ...
Went down sometime after:
2018-09-02T01:27:36-07:00
and was brought back up around:
2018-09-02T13:39:30-07:00
Various bits I noted in log:
$ curl -s --range 375155-378925 http://www.archive.balug.org/log.txt
2018-09-02 Michael Paoli
host crashed sometime after:
2018-09-02T01:27:36-07:00
but probably before about:
2018-09-02T01:35:00-07:00
on console, we got:
# [54894.969741] sd 0:0:0:0: [sda] tag#3 ABORT operation started
[54900.078084] sd 0:0:0:0: ABORT operation timed-out.
[54900.080312] sd 0:0:0:0: [sda] tag#2 ABORT operation started
[54905.198438] sd 0:0:0:0: ABORT operation timed-out.
[54905.200517] sd 0:0:0:0: [sda] tag#1 ABORT operation started
[54905.357128] Kernel panic - not syncing: assertion "i &&  
sym_get_cam_status(cp->cmd) == DID_SOFT_ERROR" failed: file  
"/build/linux-AcJpTp/linux-4.9.110/drivers/scsi/sym53c8xx_2/sym_hipd.c", line  
3399
[54905.357128]
[54905.367774] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.0-8-amd64  
#1 Debian 4.9.110-3+deb9u4
[54905.370776] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[54905.372768]  0000000000000000 ffffffff84f31e54 ffff9e2f75d5a300  
ffff9e2f7fc03e50
[54905.375471]  ffffffff84d7f6ad 0000000000000020 ffff9e2f7fc03e60  
ffff9e2f7fc03df8
[54905.378226]  3ea9db08406f9671 0000000100d04ae4 ffffffffc048a250  
ffffffffc0489e80
[54905.380982] Call Trace:
[54905.381867]  <IRQ> [54905.382541]  [<ffffffff84f31e54>] ?  
dump_stack+0x5c/0x78
[54905.384428]  [<ffffffff84d7f6ad>] ? panic+0xe4/0x23f
[54905.386164]  [<ffffffffc048512e>] ? sym_interrupt+0x1c9e/0x1e80 [sym53c8xx]
[54905.388543]  [<ffffffffc03aa010>] ?  
usb_hcd_poll_rh_status+0x170/0x170 [usbcore]
[54905.391102]  [<ffffffffc03a9fc9>] ?  
usb_hcd_poll_rh_status+0x129/0x170 [usbcore]
[54905.393627]  [<ffffffffc03aa010>] ?  
usb_hcd_poll_rh_status+0x170/0x170 [usbcore]
[54905.396144]  [<ffffffff84ce7562>] ? call_timer_fn+0x32/0x120
[54905.398071]  [<ffffffffc047ea4b>] ? sym53c8xx_intr+0x3b/0x70 [sym53c8xx]
[54905.400386]  [<ffffffff84cd418e>] ? __handle_irq_event_percpu+0x7e/0x1a0
[54905.402673]  [<ffffffff84cd42e0>] ? handle_irq_event_percpu+0x30/0x70
[54905.404898]  [<ffffffff84cd4359>] ? handle_irq_event+0x39/0x60
[54905.406901]  [<ffffffff84cd7870>] ? handle_fasteoi_irq+0xa0/0x170
[54905.409001]  [<ffffffff84c27faf>] ? handle_irq+0x1f/0x30
[54905.410834]  [<ffffffff852187ee>] ? do_IRQ+0x4e/0xe0
[54905.412528]  [<ffffffff85216556>] ? common_interrupt+0x96/0x96
[54905.414523]  <EOI> [54905.415216]  [<ffffffff852151f0>] ?  
__sched_text_end+0x1/0x1
[54905.417231]  [<ffffffff852154c2>] ? native_safe_halt+0x2/0x10
[54905.419235]  [<ffffffff8521520a>] ? default_idle+0x1a/0xd0
[54905.421137]  [<ffffffff84cbc7da>] ? cpu_startup_entry+0x1ca/0x240
[54905.423215]  [<ffffffff8593df5e>] ? start_kernel+0x447/0x467
[54905.425186]  [<ffffffff8593d120>] ? early_idt_handler_array+0x120/0x120
[54905.427438]  [<ffffffff8593d408>] ? x86_64_start_kernel+0x14c/0x170
[54905.429842] Kernel Offset: 0x3c00000 from 0xffffffff81000000  
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[54905.433484] ---[ end Kernel panic - not syncing: assertion "i &&  
sym_get_cam_status(cp->cmd) == DID_SOFT_ERROR" failed: file  
"/build/linux-AcJpTp/linux-4.9.110/drivers/scsi/sym53c8xx_2/sym_hipd.c", line  
3399
[54905.433484]
... also noted within that same timeframe, on physical host, there  
were some storage related events ... but no hard failues seen on that  
physical host and no outages or failures or such observed on that  
physical host:
Sep  2 01:29:04 vicki smartd[1093]: Device: /dev/sda [SAT], SMART  
Usage Attribute: 195 Hardware_ECC_Recovered changed from 63 to 69
Sep  2 01:29:04 vicki smartd[1093]: Device: /dev/sdb [SAT], SMART  
Usage Attribute: 190 Airflow_Temperature_Cel changed from 69 to 70
Sep  2 01:29:04 vicki smartd[1093]: Device: /dev/sdb [SAT], SMART  
Usage Attribute: 194 Temperature_Celsius changed from 31 to 30
Sep  2 01:29:04 vicki smartd[1093]: Device: /dev/sdb [SAT], SMART  
Usage Attribute: 195 Hardware_ECC_Recovered changed from 63 to 66
$