Recently, while testing the compatibility of our low-level software across different kernel versions, a colleague encountered a strange issue: when using KVM to boot CentOS 7 with specific kernel versions, a fatal error occurred that prevented the system from starting.
The affected kernel versions were:
3.10.0-1233.10.0-229
TL;DR
This is actually a compatibility bug between AMD Family-17h architecture CPUs and older kernel versions. The bug has been fixed in newer kernel versions:
Troubleshooting Process
LVM?
After the OS in KVM failed to boot, I connected to the machine via VNC and found this error:
dracut-initqueue timeout and could not boot – warning /dev/centos/root-lv does not exist
I’ve seen this error many times, and searching for keywords brings up numerous solutions. Most cases are caused by UUID mismatches, but since I was using LVM, I suspected the kernel wasn’t recognizing the LVM partition during boot, preventing system startup.
I then tried using standard partitions instead of LVM and reinstalled the system. The error persisted - still couldn’t boot.
XFS/EXT4/EXT2?
At this point, I began to suspect the boot partition filesystem. The reinstalled system was using XFS, and I recalled that very old kernel versions only support ext2/ext3/ext4 filesystems. So I reinstalled the OS twice more, using ext4 and ext2 for the /boot partition respectively. Still no luck.
GRUB2?
Then I started suspecting GRUB 2 parameters. After switching to standard partitions, the boot disk specification became linux16 ... root=UUID=xxxx. Could this root=UUID=xxx format also be unsupported by older kernel versions? So I added GRUB_DISABLE_LINUX_UUID=true to /etc/default/grub, then ran:
# Note: KVM wasn't using EFI boot
grub2-mkconfig -o /boot/grub2/grub.cfg
After updating, I tried rebooting again…
Still didn’t work.
VIRTIO?
I started suspecting the kernel couldn’t find the device at all, so I began investigating KVM’s virtio. Virtio is the disk bus provided by KVM, which should require corresponding drivers. I ran the lsinitrd command to check for the drivers:
# sudo lsinitrd /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img | grep virtio
-rw-r--r-- 1 root root 27885 Jun 24 2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/block/virtio_blk.ko
-rw-r--r-- 1 root root 52861 Jun 24 2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/char/virtio_console.ko
-rw-r--r-- 1 root root 50501 Jun 24 2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/virtio_net.ko
-rw-r--r-- 1 root root 29125 Jun 24 2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/scsi/virtio_scsi.ko
drwxr-xr-x 2 root root 0 Nov 13 16:19 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio
-rw-r--r-- 1 root root 15797 Jun 24 2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio/virtio.ko
-rw-r--r-- 1 root root 21253 Jun 24 2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio/virtio_pci.ko
-rw-r--r-- 1 root root 25541 Jun 24 2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio/virtio_ring.ko
Result: drivers were present, no difference from newer kernels that boot successfully.
AMD Family-17h!
There was still one clue - a message that flashed by quickly during boot (I had to take screenshots after many reboots to finally read it): core perfctr but no constraints; unknown hardware!
Googling this keyword, everything became clear: this is a compatibility issue between AMD Family-17h and older kernel versions. Here’s the explanation:
In family-17h, there is no PMC-event constraint. All events, irrespective of the type, can be measured using any of the six generic performance counters.
AMD Family-17h corresponds to the Zen/Zen+/Zen2 architecture CPUs, which matches the host machine running this VM. This also explains why the same kernel version worked fine on Intel platforms.
The solution is simply to upgrade the kernel - this fix has been applied by major operating systems years ago.
