chroot rm-ing disaster recovery

So you’re setting up your chroot, mounting or bind-mounting important parts of your filesystem (commonly /dev, /sys, /proc), you do some work, and then you realize you missed something and you have to start over.
No biggie, it’s just a chroot, let’s … rm -rf /path/to/chroot it. But you forgot to unmount mounts in the chroot!

Disaster.

From the triplet of mounts above, /proc and /sys should survive the event unscathed. But /dev is another story… And your system is crippled (no /dev/tty hurts useful tools like, say, sudo).

First of all: don’t worry, you’re not the first (this I know for a fact), and surely not the last (this I can only guess) to do this.
So now, assuming that for some reason rebooting the machine is not an option, how to recover?

Well it turns out to be relatively easy: just restart udev. How to actually do this depends on your system.
This should get you back on your feet.

But there still remains some cleanup to be done, since all running services started prior to the blunder are still referencing the “old” /dev. They all need to be restarted. You can get a list with lsof /path/to/chroot/dev: they are the ones with a (deleted) label in the output.

There may still be some missing nodes in /dev. In my case, both my encrypted block devices were missing from /dev/mapper. To recover those, I had to manually create the nodes with mknod /dev/mapper/name b major minor. Fill in the blanks using the output of dmsetup info.

Now just be more careful next time!
I, for one, took the safety precaution of adding alias rm=NOOP to the end of root’s .bashrc.

Leave a Reply

Your email address will not be published. Required fields are marked *