VM’s die after updating to latest kernel on XenServer
Update 29 january 2018
It seems Red Hat has released a new kernel that should fix this issue. The kernel version is:
kernel-2.6.32-696.20.1.el6.x86_64
I did quickly deploy an old CentOS version (6.4) on my XenServer 6.2 host and after that i upgraded all packages with yum update –exclude=kernel* After reboot the machine was still booting, now i did run yum update again to update to the latest kernel and kernel related packages. The new kernel indeed was version 2.6.32-696.20.1, and after installing the kernel and restarting the VM is starting as it should! Seems like it is fixed, although i havent tested it on my production VM’s yet.
Last sunday I was working on my personal VM’s running on my XenServer 6.2 (yes it is old!). As I wanted to test update one of my CentOS 6.* VM’s with the latest kernel I did run the familiar
$ yum update –exclude=kernel*
Now I made sure all packages were updated and I did test the services running on the machine. After that I updated the kernel / kernel related packages
$ yum update
After that I usually reboot my machine to make it use the new kernel, so I initiated a reboot and suddenly the VM didnt respond anymore after it went down. It tried to start but after a few seconds it got Stopped again.
After investigating the log files I was not able to find anything related. As i had a Snapshot at hand i reverted and the machine started working again. After that I tried some other VM (CentOS 7) and it worked fine untill i applied the latest kernel and rebooted the machine. Same happened and the CentOS 7 machine showed me it was in boot progress (checked the Console in the XenCenter) but then dies.
Solution, well solution…
It seems there will be no fix for this issue on older XenServer platforms. The advice i got was to update to the latest XenServer version (at this moment (january 2018) it is 7.3). After that install all the latest hotfixes as this issue is related to Meltdown and Spectre and then you should be able to update your VM’s.
How does it happen?
It seems there is a fix in the latest kernel for CentOS, but this isnt compatible with the way XenServer handles the VM. If the XenServer was up-to-date with the hotfixes the VM’s should not have any problem. I’m not sure how other OS’es like Ubuntu / Debian handle this, but it seems like the RHEL releases are affected in this.