Building a Web Cluster with FreeSBIE
Pages: 1, 2

World - BuildWorld - Create a New World

This step runs the 2.buildworld.sh script. It requires that you've installed the src-all package (FreeBSD sources for everything). I suggest that you perform a full upgrade for the whole system through CVSup. It can save you a lot of time and nerves instead of spending hours figuring out why something is not compiling. After the world building has finished, you can look at the results in the default /usr/obj/usr/src directory. In our case it takes up about 419 Mbytes of space.

Install - InstallWorld - Install New World

This option runs the 3.installworld.sh script to install the freshly built world. This step will take less time than the build process. The /usr/local/livefs will then resemble:

-rw-r--r-- 2   root  wheel   802 23 Feb 19:41 .cshrc
-rw-r--r-- 2   root  wheel   251 23 Feb 19:41 .profile
-r--r--r-- 1   root  wheel  4735 23 Feb 19:41 COPYRIGHT
drwxr-xr-x 2   root  wheel  1024 23 Feb 19:42 bin
drwxr-xr-x 3   root  wheel   512 23 Feb 19:43 boot
drwxr-xr-x 2   root  wheel   512 23 Feb 19:43 bootstrap
drwxr-xr-x 2   root  wheel   512 23 Feb 17:44 cdrom
drwxr-xr-x 16  root  wheel  2048 23 Feb 19:41 etc 
drwxr-xr-x 3   root  wheel 18432 23 Feb 19:41 dev
drwxr-xr-x 2   root  wheel   512 23 Feb 17:44 home
drwxr-xr-x 2   root  wheel   512 23 Feb 17:44 mfs
drwxr-xr-x 37  root  wheel  1024 23 Feb 17:44 mnt
drwxr-xr-x 2   root  wheel   512 23 Feb 19:41 modules
dr-xr-xr-x 2   root  wheel   512 23 Feb 19:41 proc
drwxr-xr-x 2   root  wheel   512 23 Feb 19:41 root
drwxr-xr-x 2   root  wheel  2048 23 Feb 19:42 sbin
drwxr-xr-x 5   root  wheel  1024 23 Feb 19:43 stand
lrwxr-xr-x 1   root  wheel    11 23 Feb 19:41 sys -> usr/src/sys
drwxrwxrwt 2   root  wheel   512 23 Feb 19:41 tmp
drwxr-xr-x 13  root  wheel   512 23 Feb 19:41 usr
drwxr-xr-x 20  root  wheel   512 23 Feb 19:41 var

Now our LiveFS file system is 131 Mbytes in size.

Kernel - Build & Install Kernel - Create and Install Kernel

This option runs the 4.kernel.sh script to compile and install a kernel to our LiveFS file system. The default FreeSBIE kernel configuration file is /usr/local/share/freesbie/FREESBIE. It's based on the GENERIC kernel configuration file with a lot of added options, so you should customize this file. If you're not familiar with the kernel configuration procedure, back up the original file as FREESBIE.orig.

The default configuration produces a kernel of about 5 Mbytes in size, which is definitely too much. After removing support for SCSI, RAID, missing network cards, and ISDN, my kernel size decreased to 2.2 Mbytes. You can do better.

I advise you to do a kernel compile and install using the /usr/local/share/freesbie/kernel.sh script, but not the visual dialog interface. If any errors appear during the compilation or installation it's hard to see because dialog clears all debug information too fast.

After you've finished this step, copy the kernel to the /usr/local/livefs kernel directory.

FIXIT - PatchFiles - Copy Cool and Useful Files

This option runs 5.patchfiles.sh, which adds a lot of different scripts to the LiveFS file system, especially in the scripts/ directory:

Patching files in /etc directory...
Making vnodes in /dev...
Copying cool scripts (from sysutils/livecd)
cp: /usr/local/share/freesbie/files/livecd/scripts/lang is a directory
(not copied).

/usr/local/share/freesbie/files/livecd/scripts/create_vnodes.sh
	-> /usr/local/livefs/scripts/create_vnodes.sh
/usr/local/share/freesbie/files/livecd/scripts/lang/vnodes_En
-> /usr/local/livefs/scripts/lang/vnodes_En
Copying FloppyBackup and cardDetect scripts

Packages - Packages - Choose Packages to Install

This options runs the 6.packages.sh script to install additional packages to your LiveFS file system. Figure 3 shows the menu of packages to choose.

Figure 3: The package menu.

The script generates this package list with information from pkg_info. In other words, LiveFS can install only those packages already installed on your build system. If you have installed some of your software manually without using the ports or pkg_add utility, you won't see it in the generated list. As Figure 4 shows, this operation is very similar to using the /stand/sysinstall utility for tuning your FreeBSD installation.

Figure 4: Package selection.

While choosing necessary packages, keep in mind that you're limited by the space of your storage medium. A CD-R or CD-RW normally holds between 650 and 800 Mbytes. If you use a DVD, you'll have much more room for your software.

User - Custom User - Install Shell Script Files and Others

This option runs 7.customuser.sh. This copies user profiles and software configurations from /usr/local/share/freesbie/files/foobar to the LiveFS file system. The defaults are:

Installing customized files...
/usr/local/share/freesbie/files/foobar/irssi_config ->
	/usr/local/livefs/root/.irssi/config
.muttrc
.mutt/aliases
.mutt/auto_views
.mutt/bindings
.mutt/colours
.mutt/gpg
.mutt/headers
.mutt/macros
cp: /usr/local/share/freesbie/files/foobar/.. is a directory (not copied).
cp: /usr/local/share/freesbie/files/foobar/. is a directory (not copied).
/usr/local/share/freesbie/files/foobar/.xcustom.sh ->
	/usr/local/livefs/root/.xcustom.sh
/usr/local/share/freesbie/files/foobar/.vimrc ->
	/usr/local/livefs/root/.vimrc
/usr/local/share/freesbie/files/foobar/.tcshrc.complete ->
	/usr/local/livefs/root/.tcshrc.complete
/usr/local/share/freesbie/files/foobar/.tcshrc.bindkey ->
	/usr/local/livefs/root/.tcshrc.bindkey
/usr/local/share/freesbie/files/foobar/.tcshrc ->
	/usr/local/livefs/root/.tcshrc
/usr/local/share/freesbie/files/foobar/.shrc ->
	/usr/local/livefs/root/.shrc
/usr/local/share/freesbie/files/foobar/.rhosts ->
	/usr/local/livefs/root/.rhosts
/usr/local/share/freesbie/files/foobar/.xsession ->
	/usr/local/livefs/root/.xsession
/usr/local/share/freesbie/files/foobar/.mailrc ->
	/usr/local/livefs/root/.mailrc
/usr/local/share/freesbie/files/foobar/.mail_aliases ->
	/usr/local/livefs/root/.mail_aliases
/usr/local/share/freesbie/files/foobar/.login_conf ->
	/usr/local/livefs/root/.login_conf
/usr/local/share/freesbie/files/foobar/.login ->
	/usr/local/livefs/root/.login
/usr/local/share/freesbie/files/foobar/.custom.sh ->
	/usr/local/livefs/root/.custom.sh
/usr/local/share/freesbie/files/foobar/.cshrc ->
	/usr/local/livefs/root/.cshrc
/usr/local/share/freesbie/files/foobar/.Xauthority ->
	/usr/local/livefs/root/.Xauthority
/usr/local/share/freesbie/files/foobar/.profile ->
	/usr/local/livefs/root/.profile
/usr/local/share/freesbie/files/foobar/.xinitrc ->
	/usr/local/livefs/root/.xinitrc
/usr/local/share/freesbie/files/foobar/mutt.tar.bz2 ->
	/usr/local/livefs/root/mutt.tar.bz2
/usr/local/share/freesbie/files/foobar/irssi_config ->
	/usr/local/livefs/root/irssi_config

This is not very useful for server configuration.

Related Reading The Complete FreeBSD Documentation from the Source By Greg Lehey

Build - Build ISO - Create .iso File

This option runs the 8.mkiso.sh script to create the ISO image file using the already configured path. In our case, this will generate the file /usr/local/share/freesbie/FreeSBIE.iso. We are almost done!

There are two different ways to customize the build. The first way is to edit all files in the /usr/local/share/freesbie directory before starting ./freesbie. The second way is to create a LiveFS file structure and then edit necessary files. Certainly, you can combine both of them, but I find the second approach much more sensible.

Customizing the System

Now that you're familiar with the build process, I can describe how to customize the system.

First, we need to customize the /etc directory on the LiveFS. The /etc/rc.conf file holds hostname and interface configuration. We have no easy way to keep this file on an NFS drive, except for patching the startup scripts. Hopefully GUFI, the main Italian FreeBSD User Group developing FreeSBIE, will simplify this eventually. We can either prepare a different configuration for each server in web01-10 or edit the scripts to mount /etc/rc.conf from nfs01. You can also try to follow the /usr/share/examples/diskless/clone_root instructions of cloning a diskless configuration, although I believe it'll also require a lot of hacking. I have not looked at it closely, but it seemed more oriented to diskless configurations.

Once we've set a system hostname and have connected to the network, we must start our services. This means we'll need to edit /etc/fstab:

nfs01:/startup/usr/local/etc/rc.d/cluster
/usr/local/etc/rc.d/cluster nfs ro 0 0

The /usr/local/etc/rc.d/cluster directory holds a startup script for local services. FreeBSD's /etc/rc script mounts NFS file systems before it starts local software, so we won't need any hacks there. Also we need to mount a data directory and software configuration, too:

nfs01:/storage/var/web /var/web nfs Ro 0 0

The /var/web directory holds data for our web services. This could be a directory with a complex file structure. Though we shouldn't normally make changes there, we can do this if we mount it in rw mode.

nfs01:/startup/usr/local/etc /usr/local/etc nfs ro 0 0

The /usr/local/etc directory holds our software configuration files.

To crown it all, nfs01 must store:

/startup/usr/local/etc
/startup/usr/local/etc/rc.d
/storage/var/web

This isn't a big deal, except for the data at /storage/var/web. This won't be a big load for our NetApp NFS server.

When we've finished customizing our system, we can prepare an ISO image to burn. It took me no more than two hours to reach this point from the very beginning. I hope it normally would take no more than four or five hours for any professional familiar with FreeBSD to prepare a completely new, highly customized cluster system.

You can now burn the ISO image to disk using any operating system or software you like. Under FreeBSD I like the burncd one and on the Windows platform I use Ahead Nero. Here's how to burn the system on a CD-RW disc under FreeBSD:

# burncd -s max -f /dev/acd0a blank

This will make a blank image. To burn a CD:

# burncd -s max -f /dev/acd0a /usr/local/share/freesbie/FreeSBIE.iso fixate

Updates

It's also worth mentioning how to keep your systems up-to-date. It's both easy and difficult. I believe that the most correct way to stay current is to keep your build system updated. As a rule, your production system needs infrequent updates — only when FreeBSD has important changes. So, while your build system is up-to-date, you can pass through the build process from the very beginning to create an image for the most recent version.

Conclusion

Now you can take a freshly burned CD disk and put it into the CD-ROM drive of your cluster machines. Boot it, then share your happiness at using such an easy and cost-effective solution with others!

Alexander Prohorenko is a certified professional, who holds Sun Certified System Administrator and Sun Certified Java Programmer certifications.

---

Return to the BSD DevCenter.