Debian 9 Stretch breaking in Xen guests, clouds

Software

Last night at work we started getting multiple customers reporting kernel panics in Debian 9 Stretch VMs. People were rebooting their VMs to apply updates, only to have them hang and enter a reboot loop.

Other Debian and Ubuntu versions weren’t affected, and we hadn’t made any platform changes, so we jumped on the chat platforms with our clients trying to figure out what was going on.

The one thing they all had in common was the client had unattended-upgrades on, some of which had been activated without the admin’s knowledge by a specific piece of software. Systems should not be upgrading themselves unless the sysadmin enables this and is aware of it, but that’s for another post.

I ran updates on a fresh Debian 9 VM, and noticed I got a new kernel version:

linux-image-4.9.0-14-amd64

Upgrading and rebooting the VM resulted in the same reported kernel panic. Booting from HVM to PV mode though, and it booted fine.

Around that time we realised this was a much wider spread issue. It was solved upstream in Ubuntu in September, but it appears a broken backported patch made it into Debian. This affects any guest VM booting in Xen, irrespective of any recent version.

If your VM runs on a cloud like OrionVM with a console and the ability to change virtualisation modes, change to PV and downgrade the kernel or update grub to use the previous version, then boot back to HVM if required. If you’re on AWS, you’ll probably need to use a helper VM to mount the disk.

Author bio and support

Me!

Ruben Schade is a technical writer and IaaS engineer in Sydney, Australia who refers to himself in the third person in bios. Wait, not BIOS… my brain should be EFI by now.

The site is powered by Hugo, FreeBSD, and OpenZFS on OrionVM, everyone’s favourite cloud infrastructure provider.

You can leave me a comment by contacting me, and I might publish your thoughts. Please read the FAQs first though.

If you found this post helpful or entertaining, you can shout me a coffee or buy some silly merch. Thanks!