As I mentioned a couple posts ago, I accidentally deleted a directory of very important files on my server, primarily scripts and programs that I've written for various purposes. The server is running Gentoo Linux, and I'm using an ext3 filesystem on the affected partition. For those of you not familiar with filesystems, explaining it is beyond the scope of this post, but you can find a decent write-up about it on Wikipedia.
ext3 is very similar to and backwards-compatible with ext2, which is the default filesystem for Linux. It's essentially ext2 with journaling capabilities (which basically helps prevent data corruption). Now, there are quite a few methods and programs available to recover files from an ext2 filesystem. So, given that ext3 is backwards-compatible with ext2, I thought I could use those same techniques to recover my data. Sadly, that was not the case.
It turns out that ext3 handles file deletions significantly differently than ext2. According to the official ext3 FAQ:
In order to ensure that ext3 can safely resume an unlink after a crash, it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.
Your only hope is to "grep" for parts of your files that have been deleted and hope for the best.
Needless to say, this was quite disheartening. However, after some more searching I found this clever solution. I'm certainly familiar with the strings command, which outputs a list of printable strings of text from a binary file, but I probably wouldn't have considered using it against the actual hard disk device. The results were actually quite successful - I was able to fully recover and restore my most important programs, and I'm currently working through a testing and identifying a list of smaller miscellaneous scripts that I was also able to recover.
Of course, this wasn't a pleasant experience by any means, and I was extremely lucky that I only needed to recover source code. Had I deleted a directory full of pictures or documents or basically anything other than plain text files, I would've likely been completely out of luck. As soon as I completely finish my recovery process, I will implement a newer and much more thorough backup process. It's only a matter of time before a mistake like this happens again, and the next time it does I want to simply copy files over from my backup drive rather than going through this ordeal again.
I plan on writing up a more thorough guide to recovering text files should you ever find yourself in the same situation. Stay tuned.