ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Building a Unix Server

by Dru Lavigne
08/26/2004

Yes, I know. I promised that this article would continue to cover printing and the CUPS utility. But things didn't work out that way and at this point a second printing article wouldn't do the justice that a Unix GUI printing utility deserves.

Instead, I found myself spending the last few weeks installing servers for a small startup. As I did, I remembered the myriad details needed to create servers optimized for both performance and security. While it would easily take a book to explain all the details, this article can certainly cover some of the common pitfalls to watch out for and the logical approach necessary to do the job correctly. I'll demonstrate for a FreeBSD system, but the same logic applies to your operating system of choice.

Planning the Install

If you're like the typical open-source user, you spend a lot of time experimenting by installing, reinstalling, and uninstalling software. In the beginning, that software usually includes the OS itself as you discover exactly what you can and can't do to a system! While this is a fantastic learning experience, it's a luxury you can't afford when you move on to servers in a working environment. Enter "production systems," "no downtime," and the raised eyebrows and disparaging looks (or worse) from your superiors when something doesn't work.

It's no exaggeration to think of a server install as 99% preparation and 1% configuration. Sure, you can finish quickly by installing the operating system and applications with their default settings. There isn't a sysadmin out there who hasn't had to fix the fallout from such an install — usually done by someone else who left town without a forwarding number.

Related Reading

The Complete FreeBSD
Documentation from the Source
By Greg Lehey

You can save yourself a lot of future grief if you start by clarifying your superiors' needs. Put all of the cards on the table so that you understand exactly what they want and they understand what is technically feasible. You don't want to spend a week configuring and testing Apache 2 only to find out that the web-mail program your manager has his heart set on only works with Apache 1.x. Or to hear, "What do you mean, I have to download all of my email?" after configuring a POP3 server even though you'd originally tried to sell management on the greater security provided by an IMAP server. Yes, there's often a wide gulf between management's software needs and the technical details a sysadmin concerns himself with; it may take all of your communication skills and patience to work through this preparation phase.

Finally, if your manager is the type who is always discovering new software and constantly asking you to install "just one more thing," make sure she understands that servers are special; once you've started, you'll only install what you've agreed on and written down.

The Install

After you've agreed on the software to install, you'll naturally progress to the design phase. Will you install all of the software on one server or will you spread different services out among different servers? The answer will depend upon the company's security policy, the available hardware, and the budget available for acquiring additional hardware.

For each server, go out and buy yourself a binder. You'll need thorough documentation that you create as you go along. I can hear you groaning now — or at least thinking, "I'll write down what I did later when I have more time." You won't. Even if you do, you won't remember half the stuff you did, especially the stuff you did at 3 a.m. because something still wasn't working and the server needed to be live by 6 a.m.

Next, decide whether to install using a CD or the two floppies and an Internet connection. If the system is not behind a firewall, buy or burn yourself a CD. The Number One rule when installing any server operating system is NEVER expose it to the Internet until you have secured the OS and applications. This means it needs a firewall. It also means that you don't start creating rules on the firewall to let connections in to the server until you're satisfied the server is secure. (Instead, start with temporary firewall rules that only allow connections in from a specific testing system.)

When you reach the partitioning portion of the install, pause to consider the purpose of the server you're installing. The default partition sizes are usually fine for a desktop but rarely so for a server. For example, when I chose a for automatic on my 5.2.1 desktop system, I received:

256 MB		/
622 MB		swap
256 MB		/var
256 MB		/tmp
13261 MB	/usr

Every partition (except swap) received 256 MB with the balance of the disk going to /usr. This is totally out of whack for a server. If you start installing web, ftp, or mail servers, you want to log their activities. Logs go in /var where 256 MB of space won't cut it. Things are even worse on a mail server, with mail stored in /var/mail until the user picks it up. Depending upon the type of server, /usr may also need to be fairly big as this partition contains user directories and installed software.

Unless experience has taught you otherwise, a safer assumption is to keep /, swap, and /tmp as-is and divvy up the rest of your disk space between /usr and /var. I usually press a for automatic, then d to delete /usr and /var. I can then use c to create more reasonable sizes. Ideally, I like to use two hard disks where /usr is the rest of disk 1 and /var takes over disk 2.

The other thing to keep in the back of your mind is inodes, those entries the file system uses to keep track of your files. You have a limited number of inodes. If you run out, you can't create any more files on your file system unless you delete files or repartition. Running out of inodes usually isn't an issue unless your partition will store a huge amount of very small files. The periodic script /etc/periodic/daily/400.status-disks emails root each night with the output from the df (disk free) command. After the installation, edit that script to add the -hi switch after df. This will change the output to human-readable with inode information. That way you can monitor both disk and inode usage on a daily basis.

Finally, the installer will ask you what to install. For servers, I fall into the "install the bare necessities then add what you need" group of folks as I find it easier than "installing more than you need than taking out what you don't." To me this means no docs, manpages, or games as I always have an Internet-attached system nearby should I need to look something up or take a break. I do, however, choose src so I can recompile the kernel and rebuild the world. (Yes, you'll be recompiling the kernel to optimize it for the needs of a server.)

I also believe that XFree86 does not belong on a server. Servers should be lean, mean, performance machines without the prettiness, bloat, or security implications provided by a GUI. If other admins or technical support staff will administer the server, I'll instead install /usr/ports/sysutils/webmin and /usr/ports/sysutils/usermin. These applications have configuration options to allow each support staff to access only the services they need to administer, with the added bonus of providing a GUI interface they can access from the comfort of their web browser.

You will most likely be installing software on the server and will want to keep that software up-to-date. While I'm probably the biggest fan of the ports collection out there, I don't install it on my servers as I can make better use of the 500 MB or so of /usr real estate needed to maintain it. This means I also won't use portupgrade to keep up-to-date; instead I'll demonstrate how to use porteasy to make a leaner equivalent.

Post-Install Configuration

When the installer finishes, follow through the post-install configuration menus. When it asks to create a user, make sure to create an account for yourself with a good password. Create an excellent password for the super-user account. When asked to view the ports collection, I say "yes" so I can install cvsup-without-gui and porteasy.

One of the first tasks I do after rebooting into the new system -- before I even begin installing the required server applications -- is to cvsup all of the changes to the operating system that have occurred since its release.

However, the very first thing I do once the install is complete is to grab my server binder. Up to this point I've taken only rough notes on my network settings and partition sizes. Even I have trouble deciphering my handwriting, so it's time to create some printable documentation. This is where scp (secure copy) comes in handy. I make sure to have sshd running on another system with a configured printer, then copy over my hardware information:

% scp /var/run/dmesg.boot dru@192.168.1.10:/usr/home/dru

The above command will access the SSH daemon running on 192.168.1.10, login as the user dru and copy /var/run/dmesg.boot from this system to dru's home directory on the other system. As you can see, scp works just like the cp command, but allows the source or destination file to be on another system.

I'll then send the output of my NIC settings:

% ifconfig > nic_settings && scp nic_settings dru@192.168.1.10:/usr/home/dru

Default gateway settings:

% netstat -rn > gateway && scp gateway dru@192.168.1.10:/usr/home/dru

DNS settings:

% scp /etc/resolv.conf dru@192.168.1.10:/usr/home/dru

And partition and swap settings:

% df -h > disk_usage && scp disk_usage dru@192.168.1.10:/usr/home/dru
% swapinfo > swap_usage && scp swap_usage dru@192.168.1.10:/usr/home/dru

From the system running sshd, I can now print the copied files and add them to my binder.

Securing the OS

Now it's time to start securing the system. First, I create a cvs-supfile:

more /root/cvs-supfile
*default host=cvsup.ca.freebsd.org
*default base=/usr/local/etc/cvsup
*default prefix=/usr
*default tag=RELENG_5_2_1_RELEASE
*default release=cvs delete use-rel-suffix compress
src-all

Choose a host= geographically close to you and make sure that the tag= matches your OS. (See the cvsup section of the FreeBSD Handbook for details.)

I'll then create the base directory and download the changed source:

# mkdir /usr/local/etc/cvsup
# cvsup -L 2 /root/cvs-supfile

While the download continues, I start my hardening routine. I covered many of the steps in Securing FreeBSD. I'll also create an SSH banner on the server and use the AllowUsers option to limit SSH access to myself and other authorized staff.

When the download finishes, it's time to rebuild the world and the generic kernel:

# cd /usr/src
# make buildworld
# make buildkernel KERNCONF=GENERIC
# make installkernel KERNCONF=GENERIC
# make installworld

After rebooting into the up-to-date OS, it's time to strip the kernel. I'll carefully review each line in /usr/src/sys/i386/conf/GENERIC to remove the hardware and options that aren't relevant to the server. I'll then read through NOTES (or LINT) to see if there are additional options that will increase the security or performance of the server. Avleen Vig's Tuning FreeBSD for different applications has some recommendations to get you started.

Next, I install and reboot into the new kernel, printing out a copy of the kernel configuration file with comments explaining why I added the options I did, for my server book. I then copy my kernel config file to another location such as /usr/local/etc. At this point, it's a design decision whether to remove /usr/src from the system. Removing it frees up about 400 MB of space; however, /usr/src is sometimes necessary to implement the solution to a security advisory.

Installing Software

Now that you have an up-to-date OS and an optimized kernel, it's time to start installing software. While using pkg_add -r to install pre-compiled binaries is quick and convenient, it isn't the best choice for a server. The same goes for installing a port without first reading its Makefile and combing through the installation instructions at the application's web site. Server applications come with make options which influence the application's behavior and performance. Be aware of these options before you compile the binary. This brings us back to the "99% preparation, 1% configuration" truism.

It also brings us back to your server documentation. As you install a service, carefully note the make options you used. For example, here are two entries from one of my server installs:

#installed from /usr/ports/www/apache2
#use "make show-options" to see available make options
#use "make show-modules" to see available modules
#use anon auth and disable SSI and autoindexing:
make -DWITHOUT_AUTH -DWITHOUT_MODULES="autoindex" \ 
-DWITHOUT_MODULES="include" install clean

#installed from /usr/ports/ftp/pure-ftpd
#install as stand-alone server with privilege separation
make -DWITH-PRIVSEP -DWITHOUT-INETD install clean

Knowing what options you used to compile the binary will greatly assist in troubleshooting future configuration issues. You'll also be able to repeat these options when you eventually upgrade the software.

How did I get those port directories to cd into when I didn't install the ports collection? This is where porteasy comes into play: it downloads just the ports skeletons you need. To do so, first set up your environment:

# setenv CVSROOT :pserver:anoncvs@anoncvs.at.FreeBSD.org/home/ncvs
# touch /root/.cvspass

Then, as you need a ports skeleton, tell porteasy what you want:

# porteasy -u www/apache2

That's it — all the convenience of the ports collection without maintaining the entire collection.

Keeping Software Up-to-Date

porteasy will also assist in keeping your software up-to-date. I create a script like this:

echo "Updating installed ports skeletons"
porteasy -uI

echo "The following ports need upgrading:"
porteasy -s |grep "<"

Note that this script will keep my ports skeleton up-to-date and inform me of out-of-date ports; however, it won't upgrade them for me. This is actually ideal for a server as you never want to upgrade applications blindly. Instead, carefully plan the upgrade, research any changes to the new version and their impact on your current configurations, and schedule the upgrade for a time that will least impact users on the off-chance that the upgrade results in an unforeseen glitch. Yes, I'm talking about more preparation. Most major server applications have an upgrade section to their documentation; all applications have an UPDATING or README that comes with the new version. Read them all.

A Final Word on Configuration

The actual configuration of an application will definitely depend on the application. Fortunately, most of the major server products provide excellent documentation at their web sites. If anything, a poor sysadmin might suffer from information overload!

If it's your first time plunging into a product, especially something as complicated as a web or mail server, take the time to skim through all of the documentation before you install anything. Much of it won't apply, but it will give you a good idea of what options you have and what the configuration will entail. You'll probably also find sections that you'll want to print out and add to your server binder until you're more familiar with the product.

I always print out a copy of the original configuration file(s) that come with an application and store it in my server binder. I find it convenient to pencil in the changes that I made with comments to myself reminding me why I did so. An alternate approach is to carefully comment your changes as you make them and to print out the final result. Either way, you do want a hard copy to refer to. (It goes without saying that you'll have at least one software backup copy of both the original and modified configuration files.)

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.


Read more FreeBSD Basics columns.

Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.