Building a Web Cluster with FreeSBIE

What is FreeSBIE? Quoting from its developers:

What is FreeSBIE? Simply: It's a live system on compact disk (CD), or an operating system that is able to load directly from a bootable CD, without any installation process, without any hard disk. It's based on the FreeBSD operating system.

Great news! There were a lot of different so-called LiveCD projects based on FreeBSD, but as far as I know, none has released a stable and public version. FreeSBIE 1.0 was the first one, so it sounds like a good choice.

Is there much need for FreeSBIE? Currently, the price of hard disk drives is pretty low, so it's easy to equip all our servers and workstations with hard disks. The answer is lifecycle. A regular hard drive will break after an indefinite period of time; this is unavoidable. Hard drives work for a lot of read/write operations with a heavy load. The CD's lifecycle is much longer. It normally works only for read operations, was designed as a better storage alternative, and it's also worth mentioning that the cost of one blank compact disk is hundred times less than the cost of a hard disk drive.

There are a lot of ways that running a UNIX operating system from a standalone CD can save you time and money. I'd like to describe one situation where the use of LiveCD is a very easy and cost-effective solution — a clustering solution for diskless stations.

A Clustering Example

Suppose that we need to build a cluster of web servers to serve HTTP and HTTPS connections. Why do we need a cluster? First of all, our web services are heavily loaded; having one or two CPU systems and a lot of RAM is not enough anymore. Secondly, our services need 24x7 availability and this requires an excellent fail-over backup system, which should be completely transparent for customers and normal web surfers. Nobody cares how many servers and sites we have; everyone only wants to see the requested web page.

It's also much easier to kill one server with Denial of Service (DoS) attacks, but having multiple servers will keep us safe longer. More than that, we can not rely on only one hardware server. That would be stupid. Hardware faults can happen anytime and we should not risk our business because of this. To crown it all, I should say there are a lot of different arguments why a cluster can make sense.

Related Reading BSD Hacks 100 Industrial Tip & Tools By Dru Lavigne Table of Contents Index Read Online--Safari Search this book on Safari:   Only This Book All of Safari Code Fragments only

Our clustering solution includes:

• Load-balancing hardware or software. For hardware, we can use F5 BIG-IP or anything of this kind. This hardware can handle HTTP connections and route them to the correct server. It also knows many tricks and can cache HTTPS. Also, I like it because it runs BSD OS as a managing OS. I'll call it lb01.

  There are a lot of different load-balancing systems. As a rule they are high configurable but it's usually easy to understand their syntax and write your own configuration. Our example needs to cache SSL and forward plain HTTP to our servers. It should choose the server with the least load and should remove broken ones from the list. All modern load balancers know how to do this well.

• Data storage. Personally, I like the NetApp NFS server from Network Appliance. It runs BSD, supports the NFS protocol, and works very well. I'll call it as nfs01.
• Diskless workstations, equipped only with network adapters and CD-ROM drives. Since we'll only use them for web services, I'll call them web01-10. They will run the Apache Web Server and whatever else needed.

Before I start exploring the build process, I'd like to explain why I chose this cluster configuration. There are many different possible cluster configurations. The most popular configuration involves equipping the web servers with only network adapters and making them boot through PXE-BOOT. This configuration uses an additional DHCP server. Personally I think it's a good approach, but it has some weaknesses. For starters, it can be somewhat expensive. It also has a potential quality-of-service drawback; all cluster machines depend on the DHCP server, which may be heavily loaded and has the potential of network-killing hardware failure. On the other hand, this configuration requires less managing time.

Building the FreeSBIE CD

Let's start from the FreeSBIE building process. To build a FreeSBIE LiveCD, we need a FreeBSD station. Preferably, it should run the same hardware configuration as all our cluster machines. In my case, this is an AMD Duron 1200 CPU with 128 MB of RAM. The build station also has a Maxtor 6Y120P0 UDMA 100 hard drive. Our cluster machines do not need this.

Finally, the build station also needs a CD-RW or DVD-RW disk drive to burn our CD ISO image, after we build a system, and spare, blank CD-R disks. (CD-RW may be better until you're familiar with the building and burning processes).

FreeSBIE has a FreeBSD port at /usr/ports/sysutils/freesbie. I used the version from 7 February 2004, freesbie-20040207.tar.bz2, with a size of 151,799 bytes. The FreeSBIE port is "a collection of scripts that help a user create CDs/DVDs containing a complete operating system based on FreeBSD. It is used as 'live-CD' and boots straight from CD." FreeSBIE uses compression to store a lot of software on the LiveFS file system.

The FreeSBIE site has pre-built LiveCD ISO images. When I wrote this article they had the following:

Latest ISO: FreeSBIE-1.0-i386

If the default functions and configuration is OK for you, you can use these images. For our configuration we'll need to customize the defaults.

The first step is to build an ISO image to burn to a blank CD-R or CD-RW disk. The install process requires the following files:

Installation is as simple as:

$ cd /usr/ports/sysutils/freesbie
$ make
$ su
# make install

The main part of this package is the collection of scripts that live under /usr/local/share/freesbie. The README suggests to run a user interface for the scripts, built using Savio Lam's dialog program. Let's run that ./freesbie. Start it from /usr/local/share/freesbie, since all the scripts use relative, not absolute, paths.

The first run of the script, shown in Figure 1, proposes a startup configuration.

Figure 1: FreeSBIE startup configuration.

We need to set paths and file names about the LiveFS file system to create. Let's put it in /usr/local/livefs. If the directory doesn't exist, the installer will prompt you to create it:

ATTENTION PLEASE!

The path you have entered does not seems a valid path or the directory does not
exist.

Do you wish to create it?   [ Yes ] No

We also need to set FreeSBIE's home directory, where the installer will create the kernel configuration file used to build a system. Use the current directory, /usr/local/share/freesbie.

Next, set the path to the ISO image to create. The default is /usr/local/share/freesbie/FreeSBIE.iso. I personally would rather not build a custom system in the system-wide directories, though. This will lead you to the main menu, shown in Figure 2.

Figure 2: The installer main menu.

The first section, Configure, we have already passed in the previous block. So we can move forward. Unfortunately, if you need a help system and press F1 as it advises at the bottom of the screen, it'll take you to a shell. You probably don't need the help, but beware.

Let's tackle the menu options in order.

Rmdir - Clean the FreeSBIE FS

This options runs the 0.rmdir.sh script to clean the LiveFS file system directory.

Mkdir - Create a New FreeSBIE FS

This options starts the 1.mkdir.sh script, which tries to create a directory structure for the new file system. After this happens, the /usr/local/livecd directory will resemble:

drwxr-xr-x   2 root  wheel   512 23 Feb 17:44 cdrom
drwxr-xr-x   2 root  wheel   512 23 Feb 17:44 home
drwxr-xr-x   2 root  wheel   512 23 Feb 17:44 mfs
drwxr-xr-x  37 root  wheel  1024 23 Feb 17:44 mnt
drwxr-xr-x   2 root  wheel   512 23 Feb 17:44 stand
drwxr-xr-x   3 root  wheel   512 23 Feb 17:44 usr
drwxr-xr-x   3 root  wheel   512 23 Feb 17:44 var

./mnt contains the directory structure for the various supported file systems:

drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.1
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.2
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.3
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.4
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.ext.1
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.ext.2
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.ext.3
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 dos.ext.4
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.1
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.2
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.3
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.4
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.ext.1
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.ext.2
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.ext.3
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ext2fs.ext.4
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 floppy
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.1
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.2
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.3
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.4
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.ext.1
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.ext.2
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.ext.3
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ntfs.ext.4
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 temp
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 tmp
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.1
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.2
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.3
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.4
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.5
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.6
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.7
drwxr-xr-x  2 root wheel  512 23 Feb 17:44 ufs.8

The whole directory structure takes about 94 Kbytes now.

World - BuildWorld - Create a New World

This step runs the 2.buildworld.sh script. It requires that you've installed the src-all package (FreeBSD sources for everything). I suggest that you perform a full upgrade for the whole system through CVSup. It can save you a lot of time and nerves instead of spending hours figuring out why something is not compiling. After the world building has finished, you can look at the results in the default /usr/obj/usr/src directory. In our case it takes up about 419 Mbytes of space.

Install - InstallWorld - Install New World

This option runs the 3.installworld.sh script to install the freshly built world. This step will take less time than the build process. The /usr/local/livefs will then resemble:

-rw-r--r-- 2   root  wheel   802 23 Feb 19:41 .cshrc
-rw-r--r-- 2   root  wheel   251 23 Feb 19:41 .profile
-r--r--r-- 1   root  wheel  4735 23 Feb 19:41 COPYRIGHT
drwxr-xr-x 2   root  wheel  1024 23 Feb 19:42 bin
drwxr-xr-x 3   root  wheel   512 23 Feb 19:43 boot
drwxr-xr-x 2   root  wheel   512 23 Feb 19:43 bootstrap
drwxr-xr-x 2   root  wheel   512 23 Feb 17:44 cdrom
drwxr-xr-x 16  root  wheel  2048 23 Feb 19:41 etc 
drwxr-xr-x 3   root  wheel 18432 23 Feb 19:41 dev
drwxr-xr-x 2   root  wheel   512 23 Feb 17:44 home
drwxr-xr-x 2   root  wheel   512 23 Feb 17:44 mfs
drwxr-xr-x 37  root  wheel  1024 23 Feb 17:44 mnt
drwxr-xr-x 2   root  wheel   512 23 Feb 19:41 modules
dr-xr-xr-x 2   root  wheel   512 23 Feb 19:41 proc
drwxr-xr-x 2   root  wheel   512 23 Feb 19:41 root
drwxr-xr-x 2   root  wheel  2048 23 Feb 19:42 sbin
drwxr-xr-x 5   root  wheel  1024 23 Feb 19:43 stand
lrwxr-xr-x 1   root  wheel    11 23 Feb 19:41 sys -> usr/src/sys
drwxrwxrwt 2   root  wheel   512 23 Feb 19:41 tmp
drwxr-xr-x 13  root  wheel   512 23 Feb 19:41 usr
drwxr-xr-x 20  root  wheel   512 23 Feb 19:41 var

Now our LiveFS file system is 131 Mbytes in size.

Kernel - Build & Install Kernel - Create and Install Kernel

This option runs the 4.kernel.sh script to compile and install a kernel to our LiveFS file system. The default FreeSBIE kernel configuration file is /usr/local/share/freesbie/FREESBIE. It's based on the GENERIC kernel configuration file with a lot of added options, so you should customize this file. If you're not familiar with the kernel configuration procedure, back up the original file as FREESBIE.orig.

The default configuration produces a kernel of about 5 Mbytes in size, which is definitely too much. After removing support for SCSI, RAID, missing network cards, and ISDN, my kernel size decreased to 2.2 Mbytes. You can do better.

I advise you to do a kernel compile and install using the /usr/local/share/freesbie/kernel.sh script, but not the visual dialog interface. If any errors appear during the compilation or installation it's hard to see because dialog clears all debug information too fast.

After you've finished this step, copy the kernel to the /usr/local/livefs kernel directory.

FIXIT - PatchFiles - Copy Cool and Useful Files

This option runs 5.patchfiles.sh, which adds a lot of different scripts to the LiveFS file system, especially in the scripts/ directory:

Patching files in /etc directory...
Making vnodes in /dev...
Copying cool scripts (from sysutils/livecd)
cp: /usr/local/share/freesbie/files/livecd/scripts/lang is a directory
(not copied).

/usr/local/share/freesbie/files/livecd/scripts/create_vnodes.sh
	-> /usr/local/livefs/scripts/create_vnodes.sh
/usr/local/share/freesbie/files/livecd/scripts/lang/vnodes_En
-> /usr/local/livefs/scripts/lang/vnodes_En
Copying FloppyBackup and cardDetect scripts

Packages - Packages - Choose Packages to Install

This options runs the 6.packages.sh script to install additional packages to your LiveFS file system. Figure 3 shows the menu of packages to choose.

Figure 3: The package menu.

The script generates this package list with information from pkg_info. In other words, LiveFS can install only those packages already installed on your build system. If you have installed some of your software manually without using the ports or pkg_add utility, you won't see it in the generated list. As Figure 4 shows, this operation is very similar to using the /stand/sysinstall utility for tuning your FreeBSD installation.

Figure 4: Package selection.

While choosing necessary packages, keep in mind that you're limited by the space of your storage medium. A CD-R or CD-RW normally holds between 650 and 800 Mbytes. If you use a DVD, you'll have much more room for your software.

User - Custom User - Install Shell Script Files and Others

This option runs 7.customuser.sh. This copies user profiles and software configurations from /usr/local/share/freesbie/files/foobar to the LiveFS file system. The defaults are:

Installing customized files...
/usr/local/share/freesbie/files/foobar/irssi_config ->
	/usr/local/livefs/root/.irssi/config
.muttrc
.mutt/aliases
.mutt/auto_views
.mutt/bindings
.mutt/colours
.mutt/gpg
.mutt/headers
.mutt/macros
cp: /usr/local/share/freesbie/files/foobar/.. is a directory (not copied).
cp: /usr/local/share/freesbie/files/foobar/. is a directory (not copied).
/usr/local/share/freesbie/files/foobar/.xcustom.sh ->
	/usr/local/livefs/root/.xcustom.sh
/usr/local/share/freesbie/files/foobar/.vimrc ->
	/usr/local/livefs/root/.vimrc
/usr/local/share/freesbie/files/foobar/.tcshrc.complete ->
	/usr/local/livefs/root/.tcshrc.complete
/usr/local/share/freesbie/files/foobar/.tcshrc.bindkey ->
	/usr/local/livefs/root/.tcshrc.bindkey
/usr/local/share/freesbie/files/foobar/.tcshrc ->
	/usr/local/livefs/root/.tcshrc
/usr/local/share/freesbie/files/foobar/.shrc ->
	/usr/local/livefs/root/.shrc
/usr/local/share/freesbie/files/foobar/.rhosts ->
	/usr/local/livefs/root/.rhosts
/usr/local/share/freesbie/files/foobar/.xsession ->
	/usr/local/livefs/root/.xsession
/usr/local/share/freesbie/files/foobar/.mailrc ->
	/usr/local/livefs/root/.mailrc
/usr/local/share/freesbie/files/foobar/.mail_aliases ->
	/usr/local/livefs/root/.mail_aliases
/usr/local/share/freesbie/files/foobar/.login_conf ->
	/usr/local/livefs/root/.login_conf
/usr/local/share/freesbie/files/foobar/.login ->
	/usr/local/livefs/root/.login
/usr/local/share/freesbie/files/foobar/.custom.sh ->
	/usr/local/livefs/root/.custom.sh
/usr/local/share/freesbie/files/foobar/.cshrc ->
	/usr/local/livefs/root/.cshrc
/usr/local/share/freesbie/files/foobar/.Xauthority ->
	/usr/local/livefs/root/.Xauthority
/usr/local/share/freesbie/files/foobar/.profile ->
	/usr/local/livefs/root/.profile
/usr/local/share/freesbie/files/foobar/.xinitrc ->
	/usr/local/livefs/root/.xinitrc
/usr/local/share/freesbie/files/foobar/mutt.tar.bz2 ->
	/usr/local/livefs/root/mutt.tar.bz2
/usr/local/share/freesbie/files/foobar/irssi_config ->
	/usr/local/livefs/root/irssi_config

This is not very useful for server configuration.

Related Reading The Complete FreeBSD Documentation from the Source By Greg Lehey Table of Contents (PDF) Index (PDF) Sample Chapter

Build - Build ISO - Create .iso File

This option runs the 8.mkiso.sh script to create the ISO image file using the already configured path. In our case, this will generate the file /usr/local/share/freesbie/FreeSBIE.iso. We are almost done!

There are two different ways to customize the build. The first way is to edit all files in the /usr/local/share/freesbie directory before starting ./freesbie. The second way is to create a LiveFS file structure and then edit necessary files. Certainly, you can combine both of them, but I find the second approach much more sensible.

Customizing the System

Now that you're familiar with the build process, I can describe how to customize the system.

First, we need to customize the /etc directory on the LiveFS. The /etc/rc.conf file holds hostname and interface configuration. We have no easy way to keep this file on an NFS drive, except for patching the startup scripts. Hopefully GUFI, the main Italian FreeBSD User Group developing FreeSBIE, will simplify this eventually. We can either prepare a different configuration for each server in web01-10 or edit the scripts to mount /etc/rc.conf from nfs01. You can also try to follow the /usr/share/examples/diskless/clone_root instructions of cloning a diskless configuration, although I believe it'll also require a lot of hacking. I have not looked at it closely, but it seemed more oriented to diskless configurations.

Once we've set a system hostname and have connected to the network, we must start our services. This means we'll need to edit /etc/fstab:

nfs01:/startup/usr/local/etc/rc.d/cluster
/usr/local/etc/rc.d/cluster nfs ro 0 0

The /usr/local/etc/rc.d/cluster directory holds a startup script for local services. FreeBSD's /etc/rc script mounts NFS file systems before it starts local software, so we won't need any hacks there. Also we need to mount a data directory and software configuration, too:

nfs01:/storage/var/web /var/web nfs Ro 0 0

The /var/web directory holds data for our web services. This could be a directory with a complex file structure. Though we shouldn't normally make changes there, we can do this if we mount it in rw mode.

nfs01:/startup/usr/local/etc /usr/local/etc nfs ro 0 0

The /usr/local/etc directory holds our software configuration files.

To crown it all, nfs01 must store:

/startup/usr/local/etc
/startup/usr/local/etc/rc.d
/storage/var/web

This isn't a big deal, except for the data at /storage/var/web. This won't be a big load for our NetApp NFS server.

When we've finished customizing our system, we can prepare an ISO image to burn. It took me no more than two hours to reach this point from the very beginning. I hope it normally would take no more than four or five hours for any professional familiar with FreeBSD to prepare a completely new, highly customized cluster system.

You can now burn the ISO image to disk using any operating system or software you like. Under FreeBSD I like the burncd one and on the Windows platform I use Ahead Nero. Here's how to burn the system on a CD-RW disc under FreeBSD:

# burncd -s max -f /dev/acd0a blank

This will make a blank image. To burn a CD:

# burncd -s max -f /dev/acd0a /usr/local/share/freesbie/FreeSBIE.iso fixate

Updates

It's also worth mentioning how to keep your systems up-to-date. It's both easy and difficult. I believe that the most correct way to stay current is to keep your build system updated. As a rule, your production system needs infrequent updates — only when FreeBSD has important changes. So, while your build system is up-to-date, you can pass through the build process from the very beginning to create an image for the most recent version.

Conclusion

Now you can take a freshly burned CD disk and put it into the CD-ROM drive of your cluster machines. Boot it, then share your happiness at using such an easy and cost-effective solution with others!

Alexander Prohorenko is a certified professional, who holds Sun Certified System Administrator and Sun Certified Java Programmer certifications.

Return to the BSD DevCenter.

Copyright © 2007 O'Reilly Media, Inc.