Rescuing a rebooting machine that’s hanging

One of my co-worker approached me today with a weird problem. Yesterday he had a disk in a 900GiB array failing which he replaced. After that, he run a rebuild/verification, fsck’ed the file system and tried to mount the volume again.

Apparently the mount produced a kernel oops (guess what, the 900GiB is running reiserfs), thus leaving the kernel tainted (or what do they call it ?). So he tried to reboot the box but it didn’t reboot. It started rebooting but then hung (as in not continuing the reboot). He tried to ssh back to the box, and it worked just fine.

This is where sysrq comes in handy.

That’ll restart the box, and cha-ching .. 😀