Linux

I had an issue with free disk space (or, more appropriately, a lack there of) on my server a while back. After some investigation, I discovered that my MySQL databases had ballooned in size to nearly 10 GB. Actually, figuring out that the /var/lib/mysql directory was taking up so much space wasn't that hard, but understanding why and what to do about it took a while (yes, I'm sometimes slow about such things).

It turns out I had two issues. The first is that MySQL configuration, by default, maintains binary logs. These logs "contain all statements that update data or potentially could have updated it (for example, a DELETE which matched no rows). Statements are stored in the form of 'events' that describe the modifications. The binary log also contains information about how long each statement took that updated data."^[1] This is fine and all, but (again by default) these log files are never deleted. There is a (configurable) max file size for each log, but MySQL simply rolls over to a new log when it's reached. Additionally, MySQL rolls over to a new log file on every (re)start. After a few months of operation, it's easy to see how this can take up a lot of space, and my server had been running for nearly four years.

Complicating matters somewhat was the fact that the default name of the binary logs changed at some point (and, according to the current docs, now appears to have changed back. As a result, I have several gigabytes worth of logs using the old naming convention, as well as several gigabytes worth of logs using the newer convention. Yay.

Like I said, recognizing that MySQL was taking up a lot of space is not hard, but I'm paranoid about my data and didn't want to risk losing anything. So, I kept putting it off until I was literally running out of space on a near daily basis. At that point I began doing research and figured out all of the above information. I also found a quick an easy way to fix the problem.

Note: This is meant for a standalone MySQL server. I'm not sure how it may affect replication, so please do not follow these instructions on a replicated server without additional research.

First of all, the binary logs typically reside in /var/lib/mysql/. You can check to see how much space they're currently taking up with this one-liner: du -hcs /var/lib/mysql/*bin.* | tail -n 1. If it's more than a few hundred megabytes, you may want to continue on.

Next, check to see if you were affected by the name switch like I was. This is unlikely unless you've been running the server for at least a year or so, but it definitely doesn't hurt to check. Look at all *bin.* files. If they're all named the same, such as mysqld-bin.000001, then you're fine. If you see some with a different name, such as both mysqld-bin.000001 and hostname-bin.000001, then you have an outdated set of logs doing nothing but taking up space. Look at the timestamps of the .index file for each set. One should be very recent (such as today), the other not. Once you've identified the older set, go ahead and delete all of them; they're no longer being used.

Finally, for the current set, login to MySQL as an admin user (eg., mysql -u root -p). You'll want to run the following two commands:
mysql> FLUSH LOGS;
mysql> RESET MASTER;

That's it. Depending on the size and number of your logs, those two commands may take a while to run, but the end result is that any unsaved transactions will be flushed to the database, all older logs will be dropped, and the log index will be reset to 1. In my case, these two steps dropped my from 9.6 GB down to about 5 MB. Good stuff.

Of course, this is simply a workaround to the problem, not a proper solution. What I'd really like to do is either automate this process so that I don't have to worry about the logs getting out of control, or even better configure MySQL to automatically flush its own logs after some period of time or it reaches a certain total file size. I haven't found any way to do this just yet, though I admittedly haven't looked too hard. I'd appreciate any recommendations, though.

• ^[1]http://dev.mysql.com/doc/refman/5.0/en/binary-log.html

In my last post I mentioned that I recently had a hardware failure that took down my server. I needed to get it back up and running again ASAP, but due to a large number of complications I was unable to get the original hardware up and running again, nor could I get any of the three other systems I had at my disposal to work properly. Seriously, it was like Murphy himself had taken up residence here. In the end, rather desperate and out of options, I turned to Xen (for those unfamiliar with it, it's similar to VMware or Virtual Box, but highly geared towards server0. I'd recently had quite a bit of experience getting Xen running on another system, so I felt it'd be a workable, albeit temporary, solution to my problem.

Unfortunately, the only working system I had suitable for this was my desktop, and while the process of installing and migrating the server to a Xen guest host was successful (this site is currently on that Xen instance) it was not without it's drawbacks. For one thing, there's an obvious performance hit on my desktop while running under Xen concurrently with my server guest, though fortunately my desktop is powerful enough that this mostly isn't an issue (except when the guest accesses my external USB drive to backup files; for some reason that consumes all CPU available for about 2 minutes and kills performance on the host). There were a few other minor issues, but by far the biggest problem was that the binary nVidia drivers would not install under Xen. Yes, the open source 'nv' driver would work, but that had a number of problems/limitations:

1. dramatically reduced video performance, both in video playback and normal 2d desktop usage
2. no 3d acceleration whatsoever (remember, this is my desktop system, so I sometimes use it for gaming)
3. no (working) support for multiple monitors
4. significantly different xorg.conf configuration

In fairness, issues 1 and 2 are a direct result of nVidia not providing adequate specifications for proper driver development. Nonetheless, I want my hardware to actually work, so the performance was not acceptable. Issue 3 was a major problem as well, as I have two monitors and use both heavily while working. I can only assume that this is due to a bug in the nv driver for the video card I'm using (a GeForce 8800 GTS), as dual monitors should be supported by this driver. It simply wouldn't work, though. Issue 4 wasn't that significant, but it did require quite a bit of time to rework it, which was ultimately pointless anyway due to issue 3.

So, with all that said, I began my quest to get the binary nVidia drivers working under Xen. Some basic searches showed that this was possible, but in every case the referenced material was written for much older versions of Xen, the Linux kernel, and/or the nVidia driver. I tried several different suggestions and patches, but none would work. I actually gave up, but then a few days later I got so fed up with performance that I started looking into it again and trying various different combinations of suggestions. It took a while, but I finally managed hit on the special sequence of commands necessary to get the driver to compile AND load AND run under X. Sadly, the end result is actually quite easy to do once you know what needs to be done, but figuring it out sure was a bitch. So, I wanted to post the details here to hopefully save some other people a lot of time and pain should they be in a similar situation.

This guide was written with the following system specs in mind:

• Xen 3.2.1
• Gentoo dom0 host using xen-sources-2.6.21 kernel package
  • a non-Xen kernel must also be installed, such as gentoo-sources-2.6.24-r8
• GeForce 5xxx series or newer video card using nvidia-drivers-173.14.09 driver package

Version differences shouldn't be too much of an issue; however, a lot of this is Gentoo-specific. If you're running a different distribution, you may be able to modify this technique to suit your needs, but I haven't tested it myself (if you do try and have any success, please leave a comment to let others know what you did). The non-Xen kernel should be typically left over from before you installed Xen on your host; if you don't have anything else installed, however, you can do a simple emerge gentoo-source to install it. You don't need to run it, just build against it.

Once everything is in place, and you're running the Xen-enabled (xen-sources) kernel, I suggest uninstalling any existing binary nVidia drivers with emerge -C nvidia-drivers. I had a version conflict when trying to start X at one point as the result of some old libraries not being properly updated, so this is just to make sure that the system's in a clean state. Also, while you can do most of this while in X while using the nv driver, I suggest logging out of X entirely before the modprobe line.

Here's the step-by-step guide:

1. Run uname -r to verify the version of your currently running Xen-enabled kernel; eg., mine's 2.6.21-xen
2. verify that you have both Xen and non-Xen kernels installed: cd /usr/src/ && ls -l
   • eg., I have both linux-2.6.21-xen and linux-2.6.24-gentoo-r8
3. create a symlink to the non-Xen kernel: ln -sfn linux-2.6.24-gentoo-r8 linux
4. install the nVidia-drivers package, which includes the necessary X libraries: emerge -av nvidia-drivers
   • this will also install the actual driver, but it'll be built and installed for the non-Xen kernel, not your current Xen-enabled kernel
5. determine the specific name and version of the nVidia driver package that was just installed; this can be found by examining the output of emerge -f nvidia-drivers (look for the NVIDIA-Linux-* line)
6. extract the contents of the nVidia driver package: bash /usr/portage/distfiles/NVIDIA-Linux-x86_64-173.14.09-pkg2.run -a -x
7. change to the driver source code directory: cd NVIDIA-Linux-x86_64-173.14.09-pkg2/usr/src/nv/
8. build the driver for the currently-running Xen-enabled kernel: IGNORE_XEN_PRESENCE=y make SYSSRC=/lib/modules/`uname -r`/build module
9. assuming there are no build errors (nvidia.ko should exist), install the driver:
   • mkdir /lib/modules/`uname -r`/video
   • cp -i nvidia.ko /lib/modules/`uname -r`/video/
   • depmod -a
10. if necessary, log out of X, then load the driver: modprobe nvidia
11. if necessary, reconfigure xorg.conf to use the nvidia binary driver rather than the nv driver
12. test that X will now load properly with startx
13. if appropriate, start (or restart) the display manager with /etc/init.d/xdm start

Assuming all went well, you should now have a fully functional and accelerated desktop environment, even under a Xen dom0 host. W00t. If not, feel free to post a comment and I'll try to help if I can. You should also hit up the Gentoo Forums, where you can get help from people far smarter than I.

I really hope this helps this helps some people out. It was a royal pain in the rear to get this working, but believe me, it makes a world of difference when using the system.

Occasionally I have need to copy files to/from a VMware instance. The usual process for this would involve starting up the virtual machine, loading the OS, copying the files, then shutting it down and exiting VMware. However, this adds a lot of overhead to a simply file copy process. There are some easy to find utilities for doing this under a Windows host OS, but doing so under Linux has proved a bit more difficult. After a good bit of searching, I found this VMware forum post that suggested I could use a utility called vmware-mount.pl to achieve this.

Excellent. Of course, as with many things in Linux, it's not that easy.

To begin with, I had to track down a copy of the utility. I use (and highly recommend) the excellent free VMware Player to run my guest instances. It works like a champ, but unfortunately contains only a limited selection of support tools. To get a copy of vmware-mount.pl, you'll need to download the VMware Server package for Linux. Specifically, download the file labeled "VMware Server for Linux Binary (tar.gz)". This script may also come with VMware Workstation, but VMware Server, like the Player product, is free to download and use.

Once you've downloaded VMware Server, extract the contents of the tarball ( tar xf VMware-server-1.0.3-44356). Change to the bin/ directory, and copy vmware-mount.pl and vmware-loop to the bin/ directory of your installed copy of VMware Player. In my case, this this /opt/vmware/player/bin/. Verify that both scripts are executable (chmod a+x >files< if necessary).

That takes care of installing the necessary files. In order to use these scripts, however, you need support enabled for certain kernel features. Depending on your distribution this may already be enabled, but for troubleshooting purposes I'm including the three main options that I'm aware of:

• Loopback device support (allows mounting of file objects as block devices):
  Device Drivers -> Block devices -> Loopback device support
• Network block device support (not sure exactly what this does, but it's necessary):
  Device Drivers -> Block devices -> Network block device support
• Filesystem support for guest OS partition(s) (eg., FAT32 or NTFS for Windows partitions, etc.)
  File systems -> DOS/FAT/NT Filesystems -> VFAT (substitute for needed filesystems)

Note: all of these options can be compiled as modules if preferred.

Now, time to test the script. From the command line, change to a directory containing a VMware disk image (*.vmdk file). Issue the following command (substituting the name of your own disk filename):

This should like the available partitions inside the virtual disk:

In this case, I have a single NTFS partition, labeled as partition 1. Next, we'll try mounting the partition. These commands must be run as root:

These commands will create a temporary directory for testing, then mount the first partition of my VMDK file in read-only mode. You will likely be given a warning about using this program with 2.4+ kernels. In my experience it is safe to ignore this warning, so enter Y to continue. You may also be given a warning about loading the Network Block Device driver; again, enter Y to continue. If successful, switch to another console, switch to root again, and change to the mount_test/ directory. You should see a list of files and directories from the guest OS. Switch back to the first console and enter Control-C to unmount the disk.

Assuming all went well, you can now copy files to/from your virtual disk (remove the read-only switch in the above code to write to the disk). However, mounting the disk from the command line and performing all copy operations as root is not very convenient, so let's setup a method to automatically mount the disk using your standard user account. The next part of this tip utilizes techniques from my Adding Custom Actions to KDE Context Menus article, and will only work for KDE users.

To begin, we need to make KDE recognize the VMDK format by creating a file association. Within Konqueror, select Settings, Configure Konqueror.... Select the File Associations tab, then click Add... under the Known Types pane. Select application as the Group, enter the name x-vmdk, and click OK. Click Add.. in the Filename Patterns pane, enter *.vmdk, and click OK. Add another pattern for *.VMDK so KDE will recognize both cases. Enter a description, such as VMware disk image, then click OK to close the settings window.

Next, we need to tell KDE what to do with these files. Following the directions in the Adding Custom Actions to KDE Context Menus article, create the following servicemenu file:

vmdk.desktop

As you can see by the Exec line, this requires two "support" scripts, launch.sh and mountvmdk.sh. launch.sh can be found on the Adding Custom Actions to KDE Context Menus article page, and mountvmdk.sh, which is based on my mountiso.sh script from the same page, can be download here: mountvmdk.sh script. Both of these scripts must be made executable and placed in a directory in your user's path (eg, ~/bin/ or /usr/local/bin/).

The following command from mountvmdk.sh does the magic:

echo 'y' will automatically answer the question about using a 2.4+ kernel. sudo instruct the system to run the command as root (more on that later). The 1 instructs the command to mount the first partition. This shouldn't be an issue in most cases as there really isn't much of a need to create multiple partitions on a VMware disk, but keep it in mind in case you do need to work with multiple partitions. Moving along, the '-o ro' option will mount the disk in read-only mode. I'm doing this purely for safety reasons (this is a new process for me), and it should not be needed. Finally, the uid=`uid -u` option is important. This instructs the mount command to mount the disk under your regular user's name rather than as root. The id -u command will print out your uid (User ID) and pass it to the sudo mount command. Without this, you would be unable to view the files as a non-root user.

Finally, we need to tell Linux that it's ok to do this as a regular user. This is done using the sudo command. Make sure that sudo is installed on your system, then run visudo to edit the sudoers file. Explanation of the file format is way beyond the scope of this article (this simple Google search should provide plenty of examples), but you'll need to give your regular user account permission to run the vmware-mount.pl command. Additionally, if you chose to compile either the loopback network block device drivers as modules you'll need to have permission to run the modprobe command as well. Make sure that this access is only given to trusted accounts, as it can definitely be a security risk

That should do it. Save all of the files, restart Konqueror, and then right-click on a vmdk file. Select Actions, Mount VMDK from the context menu. You should now have the contents of the vmdk file accessible in a subdirectory matching the name of the original file. When finished, simply enter Ctrl-C in the xterm window to unmount the disk image and remove the temporary directory.

Much easier. :-)

One thing I love about KDE is it's incredible breadth of configuration options. I really like to tweak my environment to best suite my needs, preferences, and habits. I know what works best for me, and prefer to have my desktop environment reflect those preferences.

Despite the vast number of preferences in the KDE Control Center, there are still quite a few options for which new GUI preference setting exists. The Hidden Configuration KDE Wiki page discusses some of these options, and is worth a read if you use KDE. In particular, I was looking for a way to disable the listing and expanding of archive files in Konqueror's sidebar. This "feature" was borrowed from Windows XP (Compressed Folders), where it always bugged the hell out of me. If I wanted view the contents of Zip files in Windows Explorer then I'd unzip the damn file.

Needless to say, I was rather dismayed and disappointed when I saw this "feature" appear in a KDE upgrade. There is no GUI preference available for disabling it, but after quite a bit of internet searching I found the above Hidden Configuration page, which discusses how to do it. It's a useful resource, and I wanted to make a note of it here both for the benefit of others as well as so I can easily find the page again next time I setup KDE. :-)

If you would like to disable this feature as well, first try entering this command (as documented in the Wiki): kwriteconfig --file konqsidebartng.rc --group General --key ShowArchivesAsFolders --type bool false. You'll need to restart Konqueror for the change to take effect. If that does not work (it didn't for me), do this instead:

1. Edit ~/.kde/share/config/konqsidebartng.rc
2. Search for the option titled ShowArchivesAsFolders
3. If you ran the kwriteconfig command, you should find it under the [General] category. Delete that under [General] and instead add ShowArchivesAsFolders=false it to the top of the file
4. If you do not already have that setting in the file, simply add it to the top of the file as described in the last step
5. Save the file and restart konqueror

You should now be rid of those annoying archive folders. Enjoy.

Oh, and if you'd like to disable this feature in Windows XP as well, you can easily to so by running the following command: regsvr32.exe /u zipfldr.dll. If you choose to reenable it, you can do so simply by running regsvr32.exe zipfldr.dll.

One thing I always liked about Windows (compared to Linux) is that it's very easy to add custom actions to the context (right-click) menu for any given file types. For example, I used this ability with Universal Extractor to add UniExtract... entries to the context menu of archive files, and I use it with Open with Arguments to add Open with arguments... to .exe and .bat files. I missed that ability for quite some time once I began using Linux as my primary OS. Something as simple as extracting Zip files, for example, would require jumping to the command line and entering an appropriate unzip command^[1]. However, a while back I stumbled across a tutorial entitled, "Creating Konqueror Service Menus", and was very pleasantly surprised to discover that this allowed me to do exactly what I had wanted for so long.

I setup a few custom actions (called "servicemenus" in KDE) a while back on my home system and pretty much forgot about it since it "just worked", but since I'm now using a new desktop system at home I'm already missing these custom actions. So, I figured I'd document them here while setting them up again. Hopefully this information will help out other Linux users. Much more thorough instructions can be found in the article referenced above - my instructions should be treated as more of a reference.

To begin, you'll need to create a new .desktop file for the action you want to perform. For the purposes of this article, I'm going to add a context menu item that will extract RAR files to the current directory. So, we'll create a new file named ~/.kde/share/apps/konqueror/servicemenus/rar.desktop. The file name is arbitrary, but it must be saved in the specified location, and must end with the .desktop extension. Next open the file in your favorite editor and add the following:

rar.desktop

This code is not very intuitive, so I'll explain each option

• ServiceTypes - specifies the type of files with which the action should be associated. The easiest way to determine this information is to run Konqueror, click Settings, Configure Konqueror, and select the File Associations section. Enter the file extension you want to associate the action with (in this case, rar, and then add the listed file types to the Service Types entry. Repeat for each extension if you want to associate with multiple types
• Actions - specifies the name of the stanza that defines the action. Multiple actions can be specified, but we'll only use one here. Just make sure that the name entered here matches the [Desktop Action xxx] defined below.
• Name - the name of the context menu entry that will appear when right-clicking on the given type of files
• Exec - the action to perform when selected; more details below. Please also see this page for a full discussion of this item, including a list of valid field codes.
• Icon - the name of the icon to associate with the context menu entry (optional). This can point to a real file if you want to use a custom icon, but you have to specify the full path and filename. In this case, I'm telling it to use the package icon from the current icon set. The easiest way (that I know of) to view these "pre-defined" icons is to right-click on any K-menu entry, select Edit Item, and click on the icon button for that item, It'll bring up an icon browser. Find the icon you like best, note the name, then close the windows and add it to the Icon entry.

Now, let's discuss the Exec entry. Ordinarily you'd probably want to call the binary directly; eg., unrar x \"%f\". In this case, however, I want to get feedback on the current progress of the operation, as well as any errors that might have occured. Since unrar is a CLI application, running it from a GUI wouldn't provide any feedback. It would simply run in the background and then exit. To work around this, I created a "wrapper" script called launch.sh that will accept arguments passed by KDE and run the command in a standalone xterm terminal^[2]. Using this method, clicking the the action in the context menu will spawn a new xterm window, which will then display the current status of the operation. It will also allow you to enter any additional information that may be necessary, such as answering an overwrite prompt or providing an archive password.

The code for the wrapper script is listed below. The only dependency is that xterm must be installed in an your $PATH.

launch.sh

That should do it. Save both of those files, make sure that launch.sh is copied to a location in your $PATH, then try right-clicking on a RAR file. Under the Actions submenu, you should now see an entry called Extract Here. Click it, and if all goes well the contents of the RAR file should be extracted to that directory.

For reference, here's a list of all KDE servicemenus that I have created:

• audacious.desktop - Enqueue and begin playing all selected audio files in Audacious (originally written for XMMS, and still contains the commented code if desired)
• iso.desktop - Mount an ISO CD-Rom image in a subdirectory of the current folder to allow file browsing and copying; press Enter when complete to unmount the ISO and remove the temporary directory. This service menu requires my mountiso.sh script.
• par.desktop - Repair damaged RAR archives using associated PAR files
• rar.desktop - Extract contents of RAR archives
• tbz.desktop - Extract contents of bzipped tarballs
• tgz.desktop - Extract contents of gzipped tarballs
• vmdk.desktop - Mount a VMware disk image in a subdirectory of the current folder to allow file browsing and copying; press Ctrl-C when complete to unmount the disk image and remove the temporary directory. This service menu requires my mountvmdk.sh script. More details can be found in the How to Mount VMware Disk Images under Linux article.
• xine.desktop - Enque and begin playing all selected video files in Xine
• zip.desktop - Extract contents of ZIP archives
• launch.sh - Wrapper script to display service menu output in an xterm window; most of my servicemenus require this script

^[1] Yes, I know that I can install a GUI archiving utility such as Ark. However, that's not really relevant here for two reasons:

1. I want to right-click and extract directly within Konqueror without first opening it in a separate utility
2. File extraction is just an easy-to-visualize simple example - there are other cases where install a separate utility is not an option or just doesn't make any sense

^[2] Yes, you could theoretically call xterm directly from the .desktop file rather than using a wrapper script, but I couldn't get it to work properly. I had issues with getting xterm and the associated command (in this case, unrar) to accept the correct path, as well as dealing with spaces in the filename. My wrapper script will handle anything that's thrown at it (so far, anyway...).

I'm a bit late posting this (I believe it's already made it to Slashdot), but Michael Stutz recently published a good article on the IBM developerWorks site entitled, "Learn 10 good UNIX usage habits." From the introduction:

When you use a system often, you tend to fall into set usage patterns. Sometimes, you do not start the habit of doing things in the best possible way. Sometimes, you even pick up bad practices that lead to clutter and clumsiness. One of the best ways to correct such inadequacies is to conscientiously pick up good habits that counteract them. This article suggests 10 UNIX command-line habits worth picking up -- good habits that help you break many common usage foibles and make you more productive at the command line in the process. Each habit is described in more detail following the list of good habits.

It contains some very useful tips. I recommend that anyone using a CLI environment, regardless of your experience level, give it a read.

Full link:

http://www-128.ibm.com/developerworks/aix/library/au-badunixhabits.html

As found via Slashdot:

microbee writes: "On LKML's periodic GPL vs. binary kernel module discussion, Andrew Morton hinted that he favors refusing to load binary modules in 12 months. Greg Kroah-Hartman then posted a patch to do exactly that. Surprisingly Linus chimed in and called it 'stupid' and a 'political agenda,' and even compared it with the RIAA's tactics. Later in the same thread Greg withdrew his patch and apologized for not having thought it through."

Linus' post is a pretty good read on this debate. If you're unfamiliar with the topic, here is a brief overview. My personal take is that FOSS drivers are definitely the way to go, but being able to actually use my hardware takes precedence. If no viable (or comparable) FOSS driver exists, then I'd rather use a binary driver then simply not have full use of my hardware.

Anyway, as I said, it's a pretty interesting read. Here's a full link to Linus' post:

http://lkml.org/lkml/2006/12/13/370

Reg Developer has recently published an article exploring the "support vacuum" created when a distribution substantially modifies an application's default configuration in its package system. From the article:

They've configured it right, but Apache ignores it because it needs some extra distro-defined magic to activate the configuration file in question. Now they're banging their head in desperation, and we can't help because we don't know the magic. And we get the blame.

The author uses Apache and Debian/Ubuntu as an example, but this is an issue with most (if not all) distributions. I've had similar problems myself trying to get Apache configured correctly under Gentoo (though the situation has improved substantially in recent versions).

Obviously a distribution will have to make certain modifications to a given application to make it fit in properly with the rest of the system. Sometimes there's just no avoiding it. But the author raises a great point; who will support the applications? If the application developers don't know what changes the distribution made to the application, and package maintainers do not know enough about the application to answer a question, who does it go to?

I hope package maintainers read this article and take it to heart. As I stated above, there will likely always be changes that need to be made here and there to provide a consistent feel across the distribution, but it's important to remember that the more drastically the application is changed, the more difficult it will be for the distribution's own users to get help.

Article link:

http://www.regdeveloper.co.uk/2006/11/04/apache_packages_support_vacuum/

As I mentioned a couple posts ago, I accidentally deleted a directory of very important files on my server, primarily scripts and programs that I've written for various purposes. The server is running Gentoo Linux, and I'm using an ext3 filesystem on the affected partition. For those of you not familiar with filesystems, explaining it is beyond the scope of this post, but you can find a decent write-up about it on Wikipedia.

ext3 is very similar to and backwards-compatible with ext2, which is the default filesystem for Linux. It's essentially ext2 with journaling capabilities (which basically helps prevent data corruption). Now, there are quite a few methods and programs available to recover files from an ext2 filesystem. So, given that ext3 is backwards-compatible with ext2, I thought I could use those same techniques to recover my data. Sadly, that was not the case.

It turns out that ext3 handles file deletions significantly differently than ext2. According to the official ext3 FAQ:

In order to ensure that ext3 can safely resume an unlink after a crash, it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.
Your only hope is to "grep" for parts of your files that have been deleted and hope for the best.

Needless to say, this was quite disheartening. However, after some more searching I found this clever solution. I'm certainly familiar with the strings command, which outputs a list of printable strings of text from a binary file, but I probably wouldn't have considered using it against the actual hard disk device. The results were actually quite successful - I was able to fully recover and restore my most important programs, and I'm currently working through a testing and identifying a list of smaller miscellaneous scripts that I was also able to recover.

Of course, this wasn't a pleasant experience by any means, and I was extremely lucky that I only needed to recover source code. Had I deleted a directory full of pictures or documents or basically anything other than plain text files, I would've likely been completely out of luck. As soon as I completely finish my recovery process, I will implement a newer and much more thorough backup process. It's only a matter of time before a mistake like this happens again, and the next time it does I want to simply copy files over from my backup drive rather than going through this ordeal again.

I plan on writing up a more thorough guide to recovering text files should you ever find yourself in the same situation. Stay tuned.

From Slashdot.org:

An anonymous reader writes "Sony has introduced three Linux-based in-car navigation and entertainment devices in Japan today. The NV-XYZ 33, 55, and 77 feature 3D map navigation technology, media players, hard drives, GPS, and PC connectivity. They are based on MIPS processors and a Linux 2.4-series kernel. They also appear to use a GateKeeper copy protection scheme, which may prevent them from being used with Linux desktops and normal media files." Still, the 3-D representations of actual buildings on the maps is pretty impressive.