Building a FreeBSD Build Systemby Bjorn Nelson
Updating software--many fear this task. It risks interruption of service. It introduces a big, gray unknown. Even with a test system and usage reports, it may bring doom to your applications. With great confidence and trust in his abilities, the system administrator must tread down this uncertain path. Sometimes this requires persuading, as many people don't see the need. Trying to quantify the risk of a compromise is not easy, especially for those who have never experienced an attack. The process of rebuilding software can sometimes be lengthy, and merging configuration files can be tedious. It's a thankless task, and not many users will recognize the effort.
Fortunately, FreeBSD has a pretty good track record of not breaking during updates--or maybe I underestimate my sysadmin skills. Unfortunately, keeping FreeBSD current is a very time-consuming job. I started seeing how daunting this is while watching our 20-plus FreeBSD servers age and fall into legacy mode. I noticed that Red Hat took very little time to stay up-to-date, and I thought there had to be a way to do something similar on FreeBSD.
I set out on a search for what was available. I tried binary updating from bsdupdates.com, which was pretty easy to set up and was great for security updates, but it didn't really help with updating configuration files--and upgrades cost money. I then attempted
radmind, but quite honestly, waiting for it to traverse my machine and build an image didn't give me the sense of time savings I wanted.
Eventually I put it on hold for year until I could find an official document or article describing best practices and procedures for updating a bunch of servers. Then, while glancing at the FreeBSD Handbook, I found what I had been looking for: tracking for multiple machines. I read about it and rejoiced. With great guidance and direction, I proceeded to build the ultimate updater: the FreeBSD build system.
The section in the handbook is a great start, but it will leave you with a system that is far from ultimate status. Never fear; when you finish this article, you will have an unbeatable update system. Even
mergemaster will work faster. You will have an update system in which a machine update/upgrade will take less than 10 minutes.
To set up a FreeBSD build system, you need three components. A build server is the first requirement. It should be either a fairly beefy uniprocessor or a lesser SMP-based machine. The second component is a staging server, which is basically a test machine where you can test the build without potentially destroying a production box. This doesn't have to be a machine with much fanfare, but it should be as close as possible to the rest of your machines to ensure an accurate test platform. The third component, called the build set, consists of all the clients to which you want to install the updates. These are your production machines.
Most of the scripts I've included here have optimizations for parallel processing, so it would be wise to use a build server that supports SMP of some form. The hardware I used is a quad PIII 700MHz with 4GB of RAM (although it would probably run fine on 512MB, 1.7GB inactive and 1.6GB free on normal load). My
buildkernel commands take not much more than 45 minutes each.
One nice aspect to running a source-based operating system is the ability to configure all compiles to follow the same behavior. The BSD
make uses a centralized /etc/make.conf file that needs to match on all the machines that will use the build server. For my build server I set:
This is a fairly small, centralized
make configuration. Most people disable parts of the core system such as X and games. For the most part I do too, but sometimes you will run across a requirement that needs things like X. I haven't run into any harm building more than what the build server requires, and restricting the build set by enabling variables such as
NO_GAMES on the clients. As always, test on the staging server before installing on the build set.
Synchronize the source
Next, you need a way to synchronize usr/src with the latest code from an official FreeBSD repository on a regular schedule. Many methods of doing this are documented in the handbook. I used
cvsup. Set it up by installing it from ports:
% cd /usr/ports/net/cvsup && make install
Then add a command to your /etc/crontab, which will start this process every morning:
# cvsup local sources and ports 0 7 * * * root /usr/local/bin/cvsup -g -L 2 /etc/cvsupfile \ >> /var/log/cvsup-client.out
Obviously, the schedule you use is your decision. You may want to slow the interval from daily to weekly or monthly to allow yourself more time for testing and to ensure that you can update all your machines using the same update baseline.
The next step is to build your /etc/cvsupfile. I set my host to
localhost because I use a local cvsup-mirror:
*default host=localhost *default prefix=/usr *default base=/usr/local/etc/cvsup *default release=cvstag=RELENG_6_0 *default delete use-rel-suffix src-all *default tag=. ports-all doc-all
I track the
RELENG_6_0 tag, which gives me security updates and bug fixes for 6.0. Then, when a later version comes out, I will switch on my own schedule and start updating all our servers.
Add to your /etc/make.conf:
SUP_UPDATE=true SUP=/usr/local/bin/cvsup SUPFLAGS=-g -L 2 SUPFILE=/etc/cvsupfile PORTSSUPFILE=/etc/cvsupfile DOCSUPFILE=/etc/cvsupfile
and start a synchronization:
% cd /usr/ports && make update