Rubenerd

Last night at work we started getting multiple customers reporting kernel panics in Debian 9 Stretch VMs. People were rebooting their VMs to apply updates, only to have them hang and enter a reboot loop.

Other Debian and Ubuntu versions weren’t affected, and we hadn’t made any platform changes, so we jumped on the chat platforms with our clients trying to figure out what was going on.

The one thing they all had in common was the client had unattended-upgrades on, some of which had been activated without the admin’s knowledge by a specific piece of software. Systems should not be upgrading themselves unless the sysadmin enables this and is aware of it, but that’s for another post.

I ran updates on a fresh Debian 9 VM, and noticed I got a new kernel version:

linux-image-4.9.0-14-amd64

Upgrading and rebooting the VM resulted in the same reported kernel panic. Booting from HVM to PV mode though, and it booted fine.

Around that time we realised this was a much wider spread issue. It was solved upstream in Ubuntu in September, but it appears a broken backported patch made it into Debian. This affects any guest VM booting in Xen, irrespective of any recent version.

If your VM runs on a cloud like OrionVM with a console and the ability to change virtualisation modes, change to PV and downgrade the kernel or update grub to use the previous version, then boot back to HVM if required. If you’re on AWS, you’ll probably need to use a helper VM to mount the disk.