Yesterday one of my customer servers crashed due to server powering off unexpectedly, and there was no ssh access of course.
As it is a remote hosted server we were unable to get our hands on it. When it started again it ran an automatic fsck over sda and sdb.
As it was a slow SATA II drive with around 1.5 TB of content, it took around 3 hours to complete. At then end, it suddenly displayed this error:
UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY
This message is pretty clear, automatic fsck couldn’t fix the inconsistency and it recommends to run fsck manually.
On this situations some recommend to boot the server with a live linux image, like a SystemRescueCD or USB boot stick to get access to the root shell, however in this case the only thing needed was to type the root password and simply run the fsck manually.
When I visited my customer’s office where this server is located there was two things to do:
1.- Avoid fsck by editing kernel parameters from grub boot loader adding ‘fastspeed’ at the end, reboot and cross fings to have the system live again.
2.- Run fsck manually to fix all the unexpected data inconsistencies.
I chosen 2, and it was pretty simple in fact:
After the UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY message it was asking to enter the root password or press CTRL + D to continue, so I entered the root password and issued the command:
fsck -y /dev/sdb1
That was all, manual fsck fixed all the errors and the system was live again.