Published on (
 See this if you're having trouble printing code examples

Big Scary Daemons

BSD Tricks: Linux Compatibility, the Hard Way


As a consultant, I'm required to be familiar with a variety of operating systems. Rather than keep three computers on hand, I just put a 12-gig hard drive in my laptop, split off a meager five gigs for Windows and Linux, and dedicated the rest to FreeBSD.

This works well enough, and any number of people run their systems in this configuration. It's always irritated me to have two Linux installs on my system, however: one under /linux, and one under /usr/compat/linux. It seems it would be possible to run programs in Linux mode using the existing Linux partition, rather than installing linux_base.

This column heads off into uncharted, unsupported territory. Don't play with this on a production system. Bring aspirin, patience, and lots of Band-Aids. Here be dragons. Bridge Out Ahead. Beware of Postal Workers. You have been warned.

I'm running FreeBSD-current on /dev/ad0s2. Red Hat Linux 6.2 uses /dev/ad0s3. The Linuxulator is based on Red Hat 6.1, meaning that the Linux userland I have is out of sync with the Linux kernel compatibility that FreeBSD provides. This turns out to be the least of our problems.

First of all, if you try to mount the Linux ext2fs partition read-write, you'll get a "permission denied" error. Your error log will show something like:

Oct 5 11:10:19 turtledawn /boot/kernel/kernel: WARNING: R/W mount of #ad/0x40002 denied due to unsupported optional features

A bit of wandering around in the mailing list archives shows that the problem is the new "sparse superblocks" that recent Linux kernels use. This increases the amount of storage space on large hard drives that use ext2fs. You can change this by using Linux's tune2fs and e2fsck.

Do not attempt to use these programs in Linux mode. Linux mode is powerful, but it doesn't address things like system nodes and directly addressing hardware. If you're lucky, you'll panic your machine; if not, you'll silently reboot it. Boot into Linux and log in as root.

First, drop to single user mode with:

shutdown -t 1 now

You need to unmount all of your filesystems, and mount them read-only. You can do this with:

umount -r /

At this point, your file systems will be mounted read-only. Linux's /etc/mtab will still show your root as being mounted read-write, but don't let that bother you; after all, the successful read-only mount means that /etc/mtab can't be updated. You can confirm that each filesystem is read-only; a simple touch /etc/test will give you a "permission denied" error if everything's OK.

You're ready to turn off sparse superblocks. Get your device names from Linux's /etc/fstab. For each device, do a:

tune2fs -s off /dev/hda3

After tuning all the file systems, check them.

e2fsck -y /dev/hda3

When complete, reboot into FreeBSD.

You can now mount your ext2 partition read-write with

mount_ext2fs /dev/ad0s3 /linux

Or you can jump right in. You can safely move the Linux userland without unloading linux.ko.

mv /compat/linux /compat/linux-pkg
mkdir /usr/compat/linux
mount_ext2fs /dev/ad0s3 /usr/compat/linux

Also edit /etc/fstab, listing the Linux partition like this:

/dev/ad0s3   /usr/compat/linux   ext2fs  rw,noauto   0   0

(Why noauto? For now, trust me.)

For a first test I tried that beast of an office suite, StarOffice 5.2. To my surprise, it worked on the first try.

Deeper checks unveiled problems. The files in /compat/linux/home/mwlucas were owned by uid and gid 500, which is what the Linux account "mwlucas" uses. I worked around user files by editing /compat/linux/etc/passwd and /compat/linux/etc/group, assigning myself my BSD uid and gid of 1001 instead of the Linux uid and gid of 500. A few runs of chown -R mwlucas.mwlucas made the problem go away for good, no matter what OS I boot.

More seriously, RealPlayer wouldn't run, failing with complaints about being unable to access the audio device. Linux programs looked in my home directory under /compat/linux/home/mwlucas, rather than under /usr/home/mwlucas. These problems all had a common cause; the Linuxulator looks for files under /compat/linux first, and only checks the rest of the system if it can't find something in its own playground. In the new configuration, dotfiles appeared left and right in the Linux home directory, while Linux programs ignored settings in my .cshrc and other BSD home directory configuration files.

Here I had a difficult situation. Not only was it doing something wrong, I had to figure out what "right" meant in this circumstance. I could set the Linux home directory to /bsd/usr/home/mwlucas, but the Linux environment is purely experimental; I don't want to damage my production environment with Linux errors. (The astute among you might notice that this entire process is endangering my production environment; you're welcome to wait in the hall for the rest of the article.) I want the Linuxulator to see files in both my Linux home directory and my BSD home directory. I decided that I'd prefer that programs run in Linux mode work with my BSD home directory first, only falling back to the Linux home directory if necessary.

FreeBSD has the ability to do this; it's called unionfs. Dire warnings fill mount_union(8). Since I'm already doing strange things, we'll charge on.

Unionfs mounts one section of the filesystem "over" another section, allowing both sections to appear to be one directory. For example, you could mount /tmp over /cdrom. An ls /cdrom would show all the files from /tmp intermingled with the files from /cdrom, as if they were one directory.

The layers are referred to as the "upper" and "lower" layers. When you alter a file, changes are made in the upper layer. If files in both the upper and lower layer have the same name, the upper file is visible. In the example above, /cdrom would appear to be writable. The files would actually be written and changed in /tmp, however. If you edited a file from /cdrom, the file would actually be copied to /tmp and edited there.

I don't really care about altering the Linux home directory -- I'm in Linux rarely enough that I can copy things from my BSD home directory should they become important. I want the BSD home /usr/home/mwlucas to be the upper layer, mounted over /usr/compat/linux/home/mwlucas. Do this with:

mount_union /usr/home/mwlucas /usr/compat/linux/home/mwlucas

Similarly, I want the Linuxulator to check for entries in /dev before /compat/linux/dev -- this will make RealPlayer access the correct /dev/audio. I had some concern about accessing entries in /dev, but Linux mode's default behavior does this, so what the heck.

mount_union /dev /usr/compat/linux/dev

To my surprise, this works as well. My Linux apps run, and things seem stable. When I'm finished, my filesystem table looks like this.

Don't worry about the "capacity" column on union-mounted partitions; I'm not sure how df calculates the amount of space used on a unionfs partition, but it's blatantly bogus. Note that when you add in union-mounted partitions, this drive supposedly has roughly 22 gigs of space on a 12-gig hard drive.

When you have a problem with the Linuxulator in this configuration, the first thing to do is restore the original configuration. If you can reproduce the problem with linux_base, it's an actual issue. If you can't, well, have fun.

If you go with the obvious default settings of /usr/compat/linux automounting via /etc/fstab, and LINUX_ENABLE="YES" in /etc/rc.conf, your system won't boot. FreeBSD mounts filesystems before loading linux.ko. If /usr/compat/linux is an ext2 partition, and is mounted before linux.ko is loaded, your system will shut down. It won't panic, it won't run any shutdown scripts, and it won't unmount your file systems; you'll just see a couple notices about waiting for core processes to stop, and be back at the BIOS screen.

The solution is to load linux.ko before mounting the file system. I use the following shell script ( in /usr/local/etc/rc.d to mount /usr/compat/linux and run the assorted mount_union commands.


/sbin/mount /usr/compat/linux

/sbin/mount_union /usr/home/mwlucas /usr/compat/linux/home/mwlucas
/sbin/mount_union /dev /usr/compat/linux/dev

You might also want to union mount /tmp over /usr/compat/linux/tmp.

When (not if) your system shuts down unnaturally, the reboot fsck will not clean the ext2 file system. When you try to mount it, you'll get a "permission denied" error and announcements in /var/log/messages. Do not attempt to use Linux's e2fsck under Linux mode; you will panic the system, and the folks up on -hackers don't want to hear about it.

To fix this, reboot into Linux. Let e2fsck do its work, then boot back into FreeBSD.

This is how I run Linux emulation. I can't say I recommend it for everyone's general use, but it might meet your needs. The surprising thing about running Linux mode in this way isn't that it works at all, but that it's so stable. Once I stopped trying to run Linux system programs in FreeBSD, it worked just as solidly as the standard FreeBSD linux_base package. This isn't because of my skill so much as it is a tribute to the FreeBSD developers; even when you do things clearly marked "dangerous," it still probably runs.

Note that "probably" doesn't mean that it's recommended.

Michael W. Lucas

Read more Big Scary Daemons columns.

Discuss this article in the Operating Systems Forum.

Return to the BSD DevCenter.


Copyright © 2009 O'Reilly Media, Inc.