Tips

warning: Creating default object from empty value in /var/www/legroom_v3/htdocs/modules/taxonomy/taxonomy.pages.inc on line 33.

Delete Old Files ONLY If Newer Files Exist

I discovered recently that one of my automated nightly backup processes had failed. I didn't discover this until about a week after it happened, and though I was able to fix it easily enough, I discovered another problem in the process: all of my backups for those systems had been wiped out. The cause turned out to be a nightly cron job that deletes old backups:

find /home/backup -type f -mtime +2 -exec rm -f {} +

This is pretty basic: find all files under /home/backup/ that are more than two days old and remove them. When new backups are added each night, this is no problems; even though all old backups get removed, newer backups are uploaded to replace them. However, when the backup process failed, the cron job kept happily deleting the older backups until, three days later, I had none left. Oops.

Fortunately, this didn't end up being an issue as I didn't need those specific backups, but nevertheless I wanted to fix the process so that the cleanup cron job would only delete old backups if newer backups exist. After a bit of testing, I cam up with this one-liner:

for i in /home/backup/*; do [[ -n $(find "$i" -type f -mtime -3) ]] && find "$i" -type f -mtime +2 -exec rm -f {} +; done

That line will work great as a cron job, but for the purpose of discussion let's break it down a little more:

1. for i in /home/backup/*; do
2.     if [[ -n $(find "$i" -type f -mtime -3) ]]; then
3.         find "$i" -type f -mtime +2 -exec rm -f {} +
4.     fi
5. done

So, there are three key parts involved. Beginning with step 2 (ignore the for loop for now), I want to make sure "new" backups exist before deleting the older ones. I do this by checking for any files that are younger than the cutoff date; if at least one or more files are found, then we can proceed with step three. The -n test verifies that the output of the find command is "not null", hence files were found.

Step 3 is pretty much exactly what I was doing previously, ie., deleting all files older than two days. However, this time it only gets executed if the previous test was true, and only operates on each subdirectory of /home/backup instead of the whole thing.

This brings us neatly back to step 1. In order for this part to make sense, you must first understand that I backup multiple systems to this directory, each under their own directory. So, I have:

/home/backup/server1
/home/backup/server2
/home/backup/server3
etc.

If I just use steps 2 and 3 operate on /home/backup directly, I could still end up losing backups. Eg., let's say backups for eveery thing except server1 began failing. New backups for server1 would continue to get added to /home/backup/server1, which means a find command on /home/backup (such as my test in step 2) would see those new files and assume everything just dandy. Meanwhile, server2, server3, etc. have not been getting any new backups, and once we cross the three day threshold all of their backups would be removed.

So, in step one I loop through each subdirectory under /home/backup, and then have the find operations run independently for each server's backups. This way, if all but server1 stops backing up, the test in step 2 will succeed on server1/, but fail on server2/, server3, etc,, thus retaining the old backups until new backups are generated.

And there you go: a safer way to cleanup old files and backups.

Port Testing (and Scanning) with Bash

Posts on my site have been rather... slow, to be generous. To try to change that, I'm going to begin posting neat tips and tricks that I discover as I go about my daily activities. Normally I just mention these to whoever happens to be on IM at the time, but I figure I can post here instead to share the information with a much wider audience and breathe some life back into my site. So, it's a win-win for everyone. :-)

I should note that many of these tips will likely be rather technical, and probably heavily Linux-focused, since that's my primary computing environment. Today's tip definitely holds true on both counts.

One of the neat features supported by Bash is socket programming. Using this, you can connect to any TCP or UDP port and any remote system. Of course, this is of rather limited usefulness as Bash won't actually do anything once connected unless specific protocol instructions are sent as well. As a relatively simple example of how this works:

exec 3<>/dev/tcp/www.google.com/80
echo -e "GET / HTTP/1.1\n\n">&3
cat <&3

(Note: Example taken from Dave Smith's Blog.)

This will establish a connection to www.google.com on port 80 (the standard HTTP port), send an HTTP GET command requesting the home page, and then display the response on your terminal. The &3 stuff is necessary to create a new file descriptor used to pass the input and output back and forth. The end result is that Google's home page (or the raw HTML for it, at least), will be downloaded and displayed on your terminal.

That's pretty slick, but like I said above, it's of rather limited usefulness. Not many people would be interested in browsing the web in this manner. However, we can use these same concepts for various other tasks and troubleshooting, including port scanning.

To get started, try running this command:

[ echo >/dev/tcp/www.google.com/80 ] && echo "open"

This will attempt to send and empty string to www.google.com on port 80, and if it receives a successful response it will display "open". Conversely, if you attempt to connect to a server/port that is not open, Bash will respond with a connection refused error.

Let's expand this a bit into a more flexible and robust function:

# Test remote host:port availability (TCP-only as UDP does not reply)
    # $1 = hostname
    # $2 = port
function port() {
    (echo >/dev/tcp/$1/$2) &>/dev/null
    if [ $? -eq 0 ]; then
        echo "$1:$2 is open"
    else
        echo "$1:$2 is closed"
    fi
}

Now, we can run port www.google.com 80 and get back "www.google.com:80 is open". Conversely, try something like port localhost 80. Unless you're running a webserver on your local computer, you should get back "localhost:80 is closed". This can provide a quick and dirty troubleshooting technique to test whether a server is listening on a given port, and ensure you can reach that port (eg., traffic is not being dropped by a firewall, etc.).

To take this another step further, we can use this function as a basic port scanner as well. For example:

for i in $(seq 1 1023); do port localhost $i; done | grep open

This will check all of the well-known ports on your local computer and report any that are open. I should not that this will be slower and more inefficient than "real" port scanners such as Nmap. However, for one-off testing situations where Nmap isn't available (or can't be installed), using bash directly can really be quite handy.

Additional information on Bash socket programming can be found in the Advanced Bash-Scripting Guide.

I hope you find this tip useful. Future tips will likely be shorter and more to the point, but I figured some additional explanation would be useful for this one. Feel free to post and questions or feedback in the comments.

Bash (Shell) Aliases and Functions

I started using Linux 10 years ago this month (actually, my very first Linux install would've been around 10 years ago today, though I'm not sure of the exact date). Throughout all those years, I've compiled a number of useful Bash functions and aliases that I use on a daily basis to save me time and help get things done. I figure that some of these would be useful to others as well, so I'm posting a list of them here, along with commentary where appropriate.

For those of you that either don't know what I'm talking about, or just aren't very familiar familiar with this topic, Bash stands for the "Bourne-again shell", and is the standard command line interface for Linux. There are plenty of other shells available, but Bash is the most common and is the default on most Linux distributions. Bash aliases and functions allow you to defined shortcuts for longer or more complicated commands.

Bash aliases are used for substituting a long/complicated string for a much shorter one that you type on the command line. As a simple example, consider the following alias that is defined by default on most distributions:

alias ls='ls --color'

This simply means that anytime you use the command ls, bash will automatically substitute ls --color for you. So, if you entered the command ls /home, bash will treat this as ls --color /home.

Bash functions provide the same essential concept, but allow for allow for much more complicated functionality through the use of shell scripting. Here's an example of a function:

function l() { locate "$1" | grep --color "$1"; }

This defines a function named l that will:

  1. Pass the supplied argument (search term) to the locate command, then
  2. pipe the output to the grep command to highlight the matched results

So, if you entered the command l filename, bash would actually run locate "filename" | grep --color "filename", This will search for all files on your computer named "filename", then use grep to highlight the word "filename" in the results. These are two fairly simple examples of aliases and functions, but when used frequently they can lead to significant time savings.

I'm including a full list of my personal aliases and functions below. Note: Some of these commands are rather obscure, but I'm including them anyway just for reference. At the very least, it may inspire similar shortcuts that make sense for you.

To use any of these, simply add them to your ~/.bashrc file.

#########
# Aliases
#########

# Show filetype colors and predictable date/timestamps
alias ls="ls --color=auto --time-style=long-iso"

# Highlight matched pattern
alias grep='grep --color'

# Common shortcuts and typos
alias c=clear
alias x=startx
alias m=mutt
alias svi='sudo vim'
alias ci='vim'
alias reboot='sudo /sbin/reboot'
alias halt='sudo /sbin/halt'

# Clear and lock console (non-X) terminal
alias lock="clear && vlock -c"

# If in a directory containing a symlink in the path, change to the "real" path
alias rd='cd "`pwd -P`"'

# Useful utility for sending files to trash from command line instead of
#   permanently deleting with rm - see http://code.google.com/p/trash-cli/
alias tp='trash-put'

# Generic shortcut for switching to root user depending on system
#alias root='su -'
#alias root='sudo -i'
alias root='sudo bash -l'

# Compile kernel, install modules, display kernel version and current date
#   useful for building custom kernels; version and date are for the filename
alias kernbuild='make -j3 && make modules_install && ls -ld ../linux && date'

# Shortcut for downloading a torrent file on the command line
alias bt='aria2c --max-upload-limit=10K --seed-time=60 --listen-port=8900-8909'

# Only show button events for xev
alias xevs="xev | grep 'keycode\|button'"

# Launch dosbox with a preset configuration for Daggerfall
alias daggerfall='dosbox -conf ~/.dosbox.conf.daggerfall'


###########
# Functions
###########

# Search Gentoo package database (portage) using eix
#   $1 = search term (package name)
function s() { eix -Fc "$1"; }     # Search all available; show summary
function sd() { eix -FsSc "$1"; }  # Search all available w/ desc.; show summary
function se() { eix -F "^$1\$"; }  # Search exact available; show details
function si() { eix -FIc "$1"; }   # Search installed; show summary


# Search Debian package database (apt) using dpkg
#   $1 = search term (package name)
#function s() { apt-cache search "$1" | grep -i "$1"; }  # search all available

# Search Arch package database using pacman
#   $1 = search term (package name
#function s() {
#    echo -e "$(pacman -Ss "$@" | sed \
#        -e 's#^core/.*#\\033[1;31m&\\033[0;37m#g' \
#        -e 's#^extra/.*#\\033[0;32m&\\033[0;37m#g' \
#        -e 's#^community/.*#\\033[1;35m&\\033[0;37m#g' \
#        -e 's#^.*/.* [0-9].*#\\033[0;36m&\\033[0;37m#g' ) \
#        \033[0m"
#}

# Mount/unmount CIFS shares; pseudo-replacement for smbmount
#   $1 = remote share name in form of //server/share
#   $2 = local mount point
function cifsmount() { sudo mount -t cifs -o username=${USER},uid=${UID},gid=${GROUPS} $1 $2; }
function cifsumount() { sudo umount $1; }

# Generate a random password
#   $1 = number of characters; defaults to 32
#   $2 = include special characters; 1 = yes, 0 = no; defaults to 1
function randpass() {
    if [ "$2" == "0" ]; then
        cat /dev/urandom | tr -cd '[:alnum:]' | head -c ${1:-32}
    else
        cat /dev/urandom | tr -cd '[:graph:]' | head -c ${1:-32}
    fi
    echo
}

# Display text of ODF document in terminal
#   $1 = ODF file
function o3() { unzip -p "$1" content.xml | o3totxt | utf8tolatin1; }

# Search all files on system using locate database
#   $1 = search term (file name)
function li() { locate -i "$1" | grep -i --color "$1"; }  # case-insensitive
function l() { locate "$1" | grep --color "$1"; }         # case-sensitive

# View latest installable portage ebuild for specified package
#   $1 = package name
function eview() {
    FILE=$(equery which $1)
    if [ -f "$FILE" ]; then
        view $FILE
    fi
}

# View portage changelog for specified package
#   $1 = package name
function echange() {
    PACKAGE="$(eix -e --only-names $1)"
    if [ "$PACKAGE" != "" ]; then
        view /usr/portage/$PACKAGE/ChangeLog
    fi
}

# Displays metadata for specified media file
#   $1 = media file name
function i() {
    EXT=`echo "${1##*.}" | sed 's/\(.*\)/\L\1/'`
    if [ "$EXT" == "mp3" ]; then
        id3v2 -l "$1"
        echo
        mp3gain -s c "$1"
    elif [ "$EXT" == "flac" ]; then
        metaflac --list --block-type=STREAMINFO,VORBIS_COMMENT "$1"
    else
        echo "ERROR: Not a supported file type."
    fi
}

# Sets custom Catalog Number ID3 tag for all MP3 files in current directory
#   $1 = catalog number
function cn() { for i in *.mp3; do id3v2 --TXXX "Catalog Number":"$1" "$i"; done; }

Definition Lists in OpenOffice Writer

This is a random tip that I wanted to publish here because I think it's useful, it's not very obvious (at least not to me), and other people can benefit from it.

A definition list, for those unfamiliar with the term, is essentially a list of terms and descriptions. You can also think of it as a glossary, similar to those found in various school text books or technical books.

For example, here's a simple definition list as supported by HTML:

Legroom.net
A very cool website that you should visit more frequently
OpenOffice
An excellent free and open source office suite

As you can see, definition lists provide a convenient way to present a list of terms and definitions in a structured format. (They can also be styled to be more visually appealing, such as bolding the terms increasing the description indentation, but that's beyond the scope of this post.)

Unfortunately, OpenOffice does not seem to support definition lists by default. At this point you may be asking yourself, "what's the big deal? Just type a term, hit enter, hit tab, then type the definition. Done." While that's true, it has to be manually and explicitly done for every term. If you have multiple definition lists in different locations, you need to make sure you use the same format for all of them. Worse, if you want to change the way it looks (such as bolding the terms), you need manually manually apply the new change to each and every one of the previous terms you've defined.

This is a needlessly tedious task that I'd rather to avoid. :-) Instead, I prefer to use a style that automatically applies all of the formatting for me. Additionally, if I ever wanted to change the formatting, I can simply update the style and OpenOffice will automatically update all previous definitions for me, which I like because it lets me be lazy.

As an aside, if you're unfamiliar with using styles in OpenOffice, you should really look into them. Once you get familiar with using them to control formatting, they make creating, and even more importantly maintaining and updating, documents much easier than using manual formatting. The OpenOffice documentation website provides a very thorough introduction to using styles and templates.

So, after much web searching and cursing, I decided to to create my own style for this. To do this, you'll (obviously) need to be using the Styles and Formatting feature of OpenOffice. I'm going to assume you're already familiar with this, so if you're not please hit up the OpenOffice documentation link above.

We'll need to define two new styles, one for the term and one for the description, and we'll create the definition term style first. To get started, right-click on the Default style in the Styles and Formatting window and select New. Enter a descriptive name, such as Definition List Term. I like to have my terms bolded, so click on the Font tab and select Bold under the Typeface column. That's all you need for a basic term, so click OK to save the style.

Next, right-click on the Default style and select New once again. This time we're creating the definition description style, so enter something like Definition List Description. In the same window, change Next Style to Definition List Term, then click on the Indents & Spacing tab. Set the Before Text indent to something like 0.30". I suggest also setting the a Below Paragraph spacing to help distinguish the entries from each other. A value of 0.10" should be sufficient. Click OK to save this style.

Finally, we need to make one more tweak to the definition term style. Right-click on the Definition List Term style and click Modify. Under the Organizer tab, set the Next Style to Definition List Description. Click OK to save once more.

Now, you're ready to use your shiny new definition lists! Select the Definition List Term style (double-click it in the Styles and Formatting window) and enter a term. The text should show up in bold. Next, hit enter to drop to the next line and start typing the description. The style should have automatically been switched to Definition List Description when you hit enter, which means your description should not be automatically indented. Hit enter again and style should be switched back to Definition List Term to allow you to enter a new term and repeat. Enter one or two more definitions just to get a feel for it. Once you're done, hit enter to go to a new line, and then select the Default style to get back to "normal".

At this point you should see how handy using a style like this can be, but let's take this one step further to really drive home the point. Let's say you decided that you want your descriptions to be italicized in addition to indented. Rather than modifying each description we've already typed, we can simply edit the description style and let OpenOffice do it for us. Right click on the Definition List Description style and click Modify. Select the Font tab, then select Italic under the Typeface column and click OK. Boom! All your previous descriptions should not be italicized.

I hope you find this little tip useful. Also, if you've never used OpenOffice before and this post is piqued your curiosity, you can download it for free and try it out today.

Flush and Reset MySQL Binary Logs

I had an issue with free disk space (or, more appropriately, a lack there of) on my server a while back. After some investigation, I discovered that my MySQL databases had ballooned in size to nearly 10 GB. Actually, figuring out that the /var/lib/mysql directory was taking up so much space wasn't that hard, but understanding why and what to do about it took a while (yes, I'm sometimes slow about such things).

It turns out I had two issues. The first is that MySQL configuration, by default, maintains binary logs. These logs "contain all statements that update data or potentially could have updated it (for example, a DELETE which matched no rows). Statements are stored in the form of 'events' that describe the modifications. The binary log also contains information about how long each statement took that updated data."[1] This is fine and all, but (again by default) these log files are never deleted. There is a (configurable) max file size for each log, but MySQL simply rolls over to a new log when it's reached. Additionally, MySQL rolls over to a new log file on every (re)start. After a few months of operation, it's easy to see how this can take up a lot of space, and my server had been running for nearly four years.

Complicating matters somewhat was the fact that the default name of the binary logs changed at some point (and, according to the current docs, now appears to have changed back. As a result, I have several gigabytes worth of logs using the old naming convention, as well as several gigabytes worth of logs using the newer convention. Yay.

Like I said, recognizing that MySQL was taking up a lot of space is not hard, but I'm paranoid about my data and didn't want to risk losing anything. So, I kept putting it off until I was literally running out of space on a near daily basis. At that point I began doing research and figured out all of the above information. I also found a quick an easy way to fix the problem.

Note: This is meant for a standalone MySQL server. I'm not sure how it may affect replication, so please do not follow these instructions on a replicated server without additional research.

First of all, the binary logs typically reside in /var/lib/mysql/. You can check to see how much space they're currently taking up with this one-liner: du -hcs /var/lib/mysql/*bin.* | tail -n 1. If it's more than a few hundred megabytes, you may want to continue on.

Next, check to see if you were affected by the name switch like I was. This is unlikely unless you've been running the server for at least a year or so, but it definitely doesn't hurt to check. Look at all *bin.* files. If they're all named the same, such as mysqld-bin.000001, then you're fine. If you see some with a different name, such as both mysqld-bin.000001 and hostname-bin.000001, then you have an outdated set of logs doing nothing but taking up space. Look at the timestamps of the .index file for each set. One should be very recent (such as today), the other not. Once you've identified the older set, go ahead and delete all of them; they're no longer being used.

Finally, for the current set, login to MySQL as an admin user (eg., mysql -u root -p). You'll want to run the following two commands:
mysql> FLUSH LOGS;
mysql> RESET MASTER;

That's it. Depending on the size and number of your logs, those two commands may take a while to run, but the end result is that any unsaved transactions will be flushed to the database, all older logs will be dropped, and the log index will be reset to 1. In my case, these two steps dropped my from 9.6 GB down to about 5 MB. Good stuff.

Of course, this is simply a workaround to the problem, not a proper solution. What I'd really like to do is either automate this process so that I don't have to worry about the logs getting out of control, or even better configure MySQL to automatically flush its own logs after some period of time or it reaches a certain total file size. I haven't found any way to do this just yet, though I admittedly haven't looked too hard. I'd appreciate any recommendations, though.

Running Binary nVidia Drivers under Xen Host

In my last post I mentioned that I recently had a hardware failure that took down my server. I needed to get it back up and running again ASAP, but due to a large number of complications I was unable to get the original hardware up and running again, nor could I get any of the three other systems I had at my disposal to work properly. Seriously, it was like Murphy himself had taken up residence here. In the end, rather desperate and out of options, I turned to Xen (for those unfamiliar with it, it's similar to VMware or Virtual Box, but highly geared towards server0. I'd recently had quite a bit of experience getting Xen running on another system, so I felt it'd be a workable, albeit temporary, solution to my problem.

Unfortunately, the only working system I had suitable for this was my desktop, and while the process of installing and migrating the server to a Xen guest host was successful (this site is currently on that Xen instance) it was not without it's drawbacks. For one thing, there's an obvious performance hit on my desktop while running under Xen concurrently with my server guest, though fortunately my desktop is powerful enough that this mostly isn't an issue (except when the guest accesses my external USB drive to backup files; for some reason that consumes all CPU available for about 2 minutes and kills performance on the host). There were a few other minor issues, but by far the biggest problem was that the binary nVidia drivers would not install under Xen. Yes, the open source 'nv' driver would work, but that had a number of problems/limitations:

  1. dramatically reduced video performance, both in video playback and normal 2d desktop usage
  2. no 3d acceleration whatsoever (remember, this is my desktop system, so I sometimes use it for gaming)
  3. no (working) support for multiple monitors
  4. significantly different xorg.conf configuration

In fairness, issues 1 and 2 are a direct result of nVidia not providing adequate specifications for proper driver development. Nonetheless, I want my hardware to actually work, so the performance was not acceptable. Issue 3 was a major problem as well, as I have two monitors and use both heavily while working. I can only assume that this is due to a bug in the nv driver for the video card I'm using (a GeForce 8800 GTS), as dual monitors should be supported by this driver. It simply wouldn't work, though. Issue 4 wasn't that significant, but it did require quite a bit of time to rework it, which was ultimately pointless anyway due to issue 3.

So, with all that said, I began my quest to get the binary nVidia drivers working under Xen. Some basic searches showed that this was possible, but in every case the referenced material was written for much older versions of Xen, the Linux kernel, and/or the nVidia driver. I tried several different suggestions and patches, but none would work. I actually gave up, but then a few days later I got so fed up with performance that I started looking into it again and trying various different combinations of suggestions. It took a while, but I finally managed hit on the special sequence of commands necessary to get the driver to compile AND load AND run under X. Sadly, the end result is actually quite easy to do once you know what needs to be done, but figuring it out sure was a bitch. So, I wanted to post the details here to hopefully save some other people a lot of time and pain should they be in a similar situation.

This guide was written with the following system specs in mind:

  • Xen 3.2.1
  • Gentoo dom0 host using xen-sources-2.6.21 kernel package
    • a non-Xen kernel must also be installed, such as gentoo-sources-2.6.24-r8
  • GeForce 5xxx series or newer video card using nvidia-drivers-173.14.09 driver package

Version differences shouldn't be too much of an issue; however, a lot of this is Gentoo-specific. If you're running a different distribution, you may be able to modify this technique to suit your needs, but I haven't tested it myself (if you do try and have any success, please leave a comment to let others know what you did). The non-Xen kernel should be typically left over from before you installed Xen on your host; if you don't have anything else installed, however, you can do a simple emerge gentoo-source to install it. You don't need to run it, just build against it.

Once everything is in place, and you're running the Xen-enabled (xen-sources) kernel, I suggest uninstalling any existing binary nVidia drivers with emerge -C nvidia-drivers. I had a version conflict when trying to start X at one point as the result of some old libraries not being properly updated, so this is just to make sure that the system's in a clean state. Also, while you can do most of this while in X while using the nv driver, I suggest logging out of X entirely before the modprobe line.

Here's the step-by-step guide:

  1. Run uname -r to verify the version of your currently running Xen-enabled kernel; eg., mine's 2.6.21-xen
  2. verify that you have both Xen and non-Xen kernels installed: cd /usr/src/ && ls -l
    • eg., I have both linux-2.6.21-xen and linux-2.6.24-gentoo-r8
  3. create a symlink to the non-Xen kernel: ln -sfn linux-2.6.24-gentoo-r8 linux
  4. install the nVidia-drivers package, which includes the necessary X libraries: emerge -av nvidia-drivers
    • this will also install the actual driver, but it'll be built and installed for the non-Xen kernel, not your current Xen-enabled kernel
  5. determine the specific name and version of the nVidia driver package that was just installed; this can be found by examining the output of emerge -f nvidia-drivers (look for the NVIDIA-Linux-* line)
  6. extract the contents of the nVidia driver package: bash /usr/portage/distfiles/NVIDIA-Linux-x86_64-173.14.09-pkg2.run -a -x
  7. change to the driver source code directory: cd NVIDIA-Linux-x86_64-173.14.09-pkg2/usr/src/nv/
  8. build the driver for the currently-running Xen-enabled kernel: IGNORE_XEN_PRESENCE=y make SYSSRC=/lib/modules/`uname -r`/build module
  9. assuming there are no build errors (nvidia.ko should exist), install the driver:
    • mkdir /lib/modules/`uname -r`/video
    • cp -i nvidia.ko /lib/modules/`uname -r`/video/
    • depmod -a
  10. if necessary, log out of X, then load the driver: modprobe nvidia
  11. if necessary, reconfigure xorg.conf to use the nvidia binary driver rather than the nv driver
  12. test that X will now load properly with startx
  13. if appropriate, start (or restart) the display manager with /etc/init.d/xdm start

Assuming all went well, you should now have a fully functional and accelerated desktop environment, even under a Xen dom0 host. W00t. If not, feel free to post a comment and I'll try to help if I can. You should also hit up the Gentoo Forums, where you can get help from people far smarter than I.

I really hope this helps this helps some people out. It was a royal pain in the rear to get this working, but believe me, it makes a world of difference when using the system.

How to Mount VMware Disk Images under Linux

Occasionally I have need to copy files to/from a VMware instance. The usual process for this would involve starting up the virtual machine, loading the OS, copying the files, then shutting it down and exiting VMware. However, this adds a lot of overhead to a simply file copy process. There are some easy to find utilities for doing this under a Windows host OS, but doing so under Linux has proved a bit more difficult. After a good bit of searching, I found this VMware forum post that suggested I could use a utility called vmware-mount.pl to achieve this.

Excellent. Of course, as with many things in Linux, it's not that easy.

To begin with, I had to track down a copy of the utility. I use (and highly recommend) the excellent free VMware Player to run my guest instances. It works like a champ, but unfortunately contains only a limited selection of support tools. To get a copy of vmware-mount.pl, you'll need to download the VMware Server package for Linux. Specifically, download the file labeled "VMware Server for Linux Binary (tar.gz)". This script may also come with VMware Workstation, but VMware Server, like the Player product, is free to download and use.

Once you've downloaded VMware Server, extract the contents of the tarball ( tar xf VMware-server-1.0.3-44356). Change to the bin/ directory, and copy vmware-mount.pl and vmware-loop to the bin/ directory of your installed copy of VMware Player. In my case, this this /opt/vmware/player/bin/. Verify that both scripts are executable (chmod a+x >files< if necessary).

That takes care of installing the necessary files. In order to use these scripts, however, you need support enabled for certain kernel features. Depending on your distribution this may already be enabled, but for troubleshooting purposes I'm including the three main options that I'm aware of:

  • Loopback device support (allows mounting of file objects as block devices):
    Device Drivers -> Block devices -> Loopback device support
  • Network block device support (not sure exactly what this does, but it's necessary):
    Device Drivers -> Block devices -> Network block device support
  • Filesystem support for guest OS partition(s) (eg., FAT32 or NTFS for Windows partitions, etc.)
    File systems -> DOS/FAT/NT Filesystems -> VFAT (substitute for needed filesystems)

Note: all of these options can be compiled as modules if preferred.

Now, time to test the script. From the command line, change to a directory containing a VMware disk image (*.vmdk file). Issue the following command (substituting the name of your own disk filename):

vmware-mount.pl -p WinXP_Pro_SP2.vmdk

This should like the available partitions inside the virtual disk:

--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter
Version: 1.0 build-44356
Copyright 1998 VMware, Inc. All rights reserved. -- VMware Confidential
--------------------------------------------

Nr      Start       Size Type Id Sytem
-- ---------- ---------- ---- -- ------------------------
 1         63   12562767 BIOS  7 HPFS/NTFS

In this case, I have a single NTFS partition, labeled as partition 1. Next, we'll try mounting the partition. These commands must be run as root:

mkdir mount_test
vmware-mount.pl WinXP_Pro_SP2.vmdk 1 -o ro mount_test

These commands will create a temporary directory for testing, then mount the first partition of my VMDK file in read-only mode. You will likely be given a warning about using this program with 2.4+ kernels. In my experience it is safe to ignore this warning, so enter Y to continue. You may also be given a warning about loading the Network Block Device driver; again, enter Y to continue. If successful, switch to another console, switch to root again, and change to the mount_test/ directory. You should see a list of files and directories from the guest OS. Switch back to the first console and enter Control-C to unmount the disk.

Assuming all went well, you can now copy files to/from your virtual disk (remove the read-only switch in the above code to write to the disk). However, mounting the disk from the command line and performing all copy operations as root is not very convenient, so let's setup a method to automatically mount the disk using your standard user account. The next part of this tip utilizes techniques from my Adding Custom Actions to KDE Context Menus article, and will only work for KDE users.

To begin, we need to make KDE recognize the VMDK format by creating a file association. Within Konqueror, select Settings, Configure Konqueror.... Select the File Associations tab, then click Add... under the Known Types pane. Select application as the Group, enter the name x-vmdk, and click OK. Click Add.. in the Filename Patterns pane, enter *.vmdk, and click OK. Add another pattern for *.VMDK so KDE will recognize both cases. Enter a description, such as VMware disk image, then click OK to close the settings window.

Next, we need to tell KDE what to do with these files. Following the directions in the Adding Custom Actions to KDE Context Menus article, create the following servicemenu file:

vmdk.desktop
[Desktop Entry]
ServiceTypes=application/x-vmdk
Actions=mountvmdk

[Desktop Action mountvmdk]
Name=Mount VMDK
Exec=launch.sh %d mountvmdk.sh \"%f\"
Icon=binary

As you can see by the Exec line, this requires two "support" scripts, launch.sh and mountvmdk.sh. launch.sh can be found on the Adding Custom Actions to KDE Context Menus article page, and mountvmdk.sh, which is based on my mountiso.sh script from the same page, can be download here: mountvmdk.sh script. Both of these scripts must be made executable and placed in a directory in your user's path (eg, ~/bin/ or /usr/local/bin/).

The following command from mountvmdk.sh does the magic:

echo 'y' | sudo vmware-mount.pl $1 1 -o ro,uid=`id -u` $DIR

echo 'y' will automatically answer the question about using a 2.4+ kernel. sudo instruct the system to run the command as root (more on that later). The 1 instructs the command to mount the first partition. This shouldn't be an issue in most cases as there really isn't much of a need to create multiple partitions on a VMware disk, but keep it in mind in case you do need to work with multiple partitions. Moving along, the '-o ro' option will mount the disk in read-only mode. I'm doing this purely for safety reasons (this is a new process for me), and it should not be needed. Finally, the uid=`uid -u` option is important. This instructs the mount command to mount the disk under your regular user's name rather than as root. The id -u command will print out your uid (User ID) and pass it to the sudo mount command. Without this, you would be unable to view the files as a non-root user.

Finally, we need to tell Linux that it's ok to do this as a regular user. This is done using the sudo command. Make sure that sudo is installed on your system, then run visudo to edit the sudoers file. Explanation of the file format is way beyond the scope of this article (this simple Google search should provide plenty of examples), but you'll need to give your regular user account permission to run the vmware-mount.pl command. Additionally, if you chose to compile either the loopback network block device drivers as modules you'll need to have permission to run the modprobe command as well. Make sure that this access is only given to trusted accounts, as it can definitely be a security risk

That should do it. Save all of the files, restart Konqueror, and then right-click on a vmdk file. Select Actions, Mount VMDK from the context menu. You should now have the contents of the vmdk file accessible in a subdirectory matching the name of the original file. When finished, simply enter Ctrl-C in the xterm window to unmount the disk image and remove the temporary directory.

Much easier. :-)