How to recover lost initrd

What do you do when you accidentally remove the Linux kernel and initrd? Well, this is a cautionary tale. No matter how much you think you know, be very careful, and don’t remove the kernel and initrd. Best practice would be:

• Backup the system before upgrade
• Build new Ubuntu installation before your upgrade, that way you can reinstall in place more easily.

Sometimes, its not the upgrade that gets you, however. Ubuntu did a simple update that included linux-image-3.19.0-25-generic. This seemed to go OK until /boot filled up, probably making the new initrd file. The update manager even told me both that there were errors, and that there was insufficient space on /boot.

Here’s where we go off the rails. I knew that Ubuntu just left older versions around, but I didn’t know to run sudo apt-get autoremove. I was knowledgable, but not careful enough. I became root, changed directory to /boot, and proceeded to remove the older kernels. Oops - I accidentally removed all the kernels and initrd files.

We haven’t crashed yet - the system is already running. I tried to force an install again of the newer kernel, which sort of worked:

sudo apt-get install --force linux-image-3.19.0-24-generic

I let out my breath, didn’t read the fine print, and went to deal with a wakeful child. When I got back, I immediately felt nervous, and wanted to reboot to prove it was OK.

Oops. Now, I had to fix the system. The update process of Ubuntu expects the old initramfs (initrd) to be present. I cloudn’t find my boot media from 4 years ago - this is Ubuntu, it can upgrade in place. At work, I couldn’t install tools as Administrator, and so I built a USB incorrectly - essentially the same as a dd to /dev/sdb1 rather than /dev/sdb.

When I got home, I searched, and finally found a 13.04 DVD install. A bit more tinkering allowed me to mount my root from an LVM partition, and then my boot more directly. I then did a chroot to the system, and ran update-initramfs manually to create a new initrd.

In more detail:

• Run a Ubuntu LiveCD, but don’t re-install. Click “Try Ubuntu” instead.
• Start a terminal.
• Become root, I use: sudo su -
• You can use fdisk -l /dev/sda to see your partitioning. Mine is:

/dev/sda1  *      2048    499711    497664   243M 83 Linux
/dev/sda2       501758 488396799 487895042 232.7G  5 Extended
/dev/sda5       501760 488396799 487895040 232.7G 8e Linux LVM

• This shows that /dev/sda1 must be /boot and / is somewhere in LVM. First, scan physical volumes: pvscan
• Next, scan volume groups: vgscan
• Activate all volume groups available: vgchange -a y
• Scan logical volumes: lvscan
• From the output, you can see what to mount: mount /dev/ubuntu-vg/root /mnt
• Now we can mount boot: mount /dev/sda1 /mnt/boot
• Chroot to the mounted system: chroot /mnt
• The chroot makes sure we’re running the right software: which update-initramfs
• Build the initrd: update-initframfs -c -k 3.19.0-25-generic
• Reboot into the recovered system

So, that’s why paddy lost some time at work today ;)