What do you do when you accidentally remove the Linux kernel and initrd? Well, this is a cautionary tale. No matter how much you think you know, be very careful, and don’t remove the kernel and initrd. Best practice would be:
- Backup the system before upgrade
- Build new Ubuntu installation before your upgrade, that way you can reinstall in place more easily.
Sometimes, its not the upgrade that gets you, however. Ubuntu did a simple update that included
linux-image-3.19.0-25-generic. This seemed to go OK until
/boot filled up, probably making the new initrd file. The update manager even told me both that there were errors, and that there was insufficient space on
Here’s where we go off the rails. I knew that Ubuntu just left older versions around, but I didn’t know to run
sudo apt-get autoremove. I was knowledgable, but not careful enough. I became root, changed directory to
/boot, and proceeded to remove the older kernels. Oops - I accidentally removed all the kernels and initrd files.
We haven’t crashed yet - the system is already running. I tried to force an install again of the newer kernel, which sort of worked:
sudo apt-get install --force linux-image-3.19.0-24-generic
I let out my breath, didn’t read the fine print, and went to deal with a wakeful child. When I got back, I immediately felt nervous, and wanted to reboot to prove it was OK.
Oops. Now, I had to fix the system. The update process of Ubuntu expects the old initramfs (initrd) to be present. I cloudn’t find my boot media from 4 years ago - this is Ubuntu, it can upgrade in place. At work, I couldn’t install tools as Administrator, and so I built a USB incorrectly - essentially the same as a dd to
/dev/sdb1 rather than
When I got home, I searched, and finally found a 13.04 DVD install. A bit more tinkering allowed me to mount my root from an LVM partition, and then my boot more directly. I then did a chroot to the system, and ran update-initramfs manually to create a new initrd.
In more detail:
- Run a Ubuntu LiveCD, but don’t re-install. Click “Try Ubuntu” instead.
- Start a terminal.
- Become root, I use:
sudo su -
- You can use
fdisk -l /dev/sdato see your partitioning. Mine is:
/dev/sda1 * 2048 499711 497664 243M 83 Linux /dev/sda2 501758 488396799 487895042 232.7G 5 Extended /dev/sda5 501760 488396799 487895040 232.7G 8e Linux LVM
- This shows that
/dev/sda1must be /boot and / is somewhere in LVM. First, scan physical volumes:
- Next, scan volume groups:
- Activate all volume groups available:
vgchange -a y
- Scan logical volumes:
- From the output, you can see what to mount:
mount /dev/ubuntu-vg/root /mnt
- Now we can mount boot:
mount /dev/sda1 /mnt/boot
- Chroot to the mounted system:
- The chroot makes sure we’re running the right software:
- Build the initrd:
update-initframfs -c -k 3.19.0-25-generic
- Reboot into the recovered system
So, that’s why paddy lost some time at work today ;)