Behind DragonFly BSDby Federico Biancuzzi
For years we had FreeBSD for performance, NetBSD for portability, and OpenBSD for security. Now a new project, focused on innovation, has worked very hard and is ready to release its first version: DragonFly BSD 1.0. Federico Biancuzzi interviewed some of these innovators to discuss their interesting points of view.
Matthew, could you give us a short history of the steps that brought you from being a FreeBSD developer to becoming the DragonFly BSD project creator?
Matthew Dillon: Well, while at Berkeley (1985-1989) I used CSRG's BSD 4.3 quite a lot and even did a little driver work for the Perkin-Elmer we had to play with. I had already done considerable work in the embedded world. Working with Berkeley's UNIX really piqued my interest in UNIX and operating systems in general. After graduating I began using Linux, and contributed to some of the early Linux kernel work. But once the USL lawsuit was resolved and the BSDs started gaining momentum I returned to my roots and began actively using FreeBSD. Later on I began using it on BEST Internet's machines, which of course resulted in a large number of bug and patch submissions, which eventually led to a commit bit as a FreeBSD committer. However, that relationship had always been quite fractious, primarily due to the inability for some FreeBSD committers to accept the work on its face. For example, I had to fight tooth and nail to get the VM changes between FreeBSD 3.x and 4.x into the tree.
However, while committer relations have always been an issue, DragonFly split off from FreeBSD-5 over major architectural differences, not anything else. We really do feel that FreeBSD-5 is taking the wrong approach to SMP and building something that is so complex that it will ultimately not be maintainable. We think we have a better way.
Hiten, Jeffrey and Joerg, could you introduce yourself?
Joerg Sonnenberger: I'm a 20-year-old student of mathematics and spare-time programmer. I have investigated different fields of CS in the past years and got a bit bored last year. That's why I'm here.
Jeffrey Hsu: I was one of the early FreeBSD committers 10 years ago, working in the kernel and on many of the compiler and language ports. I did the first JDK port to FreeBSD. Recently I've done a lot of work on the FreeBSD networking stack, mainly adding SMP locking and improving TCP. When Matt started DragonFly, he invited me to join and I've been here ever since. I find the friendly project atmosphere a refreshing change.
Hiten Pandya: I am a 17-year-old and I live in the United Kingdom (UK). I have been working on FreeBSD for a good five years, since 1999, and as a committer since summer of 2003. I was aware of DragonFly and Matt's plans at about the end of May 2003, i.e., before it was publicly announced, and I have worked on it since its inception in early June 2003.
I still work with FreeBSD people often but my primary interest is in the advancement of DragonFly because of the adopted methodologies. I really do not have anything against FreeBSD, because without that, we would not have DragonFly. For me it is just a matter of preference and I like to work with a smaller (committer) team because it is easier to get work coordinated and actioned without too many interruptions. At the end of day, it's all about advancement of technology and ideas. Politics has little to do with it. My interests roughly scale from Operating Systems to Clustering. I have worked on various projects, including Apache HTTP, Jakarta Tomcat, xMach, IBM's Journal File System for Linux and OS/2, DCE UUIDs, SCTP network protocol, and such. Over the last five years, I have written numerous amounts of technical documentation, device drivers, subsystems and embeddable systems on *BSD and Linux. In the past few years, I have also worked extensively on Bioinformatics applications such as BLAST and Phylogeny and Annotation tools, thus my interest in developing SSI-type technology among others, like Checkpointing.
What is your role in the project?
JS: I'm doing a lot of the driver and low-level hardware work e.g. by investigating and backporting the various changes of FreeBSD CURRENT. Another field is keeping the system compiler up-to-date.
JH: I do mostly the same things as before: adding MP support to the networking stack and improving the TCP algorithms.
HP: My role in DragonFly is hard to describe in a few words. I try to do whatever is necessary for the Project like coordinating developer effort, help with features, liaise with BSD and Linux developers, try to keep peace and clear any misconceptions held by others, stop people from unwanted ranting on the lists, etc., etc. If there is work that needs to be done but no one has picked up the ball due to constraints, I generally tend to them. Basically, anything and everything; starting from core kernel work to documentation, web site updates, anything.
I have worked in these areas many times before so most of them aren't a problem. I usually read CS papers to see how DragonFly's approach compares to others, what interesting things we can take from NetBSD, OpenBSD, etc. Additionally, I try to stay aware of developments in the CS industry from time to time.
While we are on the topic of who's who, I would like to thank the following people: the DragonFly Team — especially David Rhodus — for providing support whenever possible. I typed this up on a notebook donated to me for DragonFly work by Chris Beuchler and Scott Ullrich.
Who do you think should try DragonFly BSD?
MD: Anyone at all.
JS: Anybody with a certain understanding of computers and UNIX. I'm not sure whether I want to use DragonFly for a productive system now, but it runs fine most of the time and has a decent performance. I'm re-commenting it to Linux and other BSD users and haven't received negative replies yet.
JH: Anyone who likes Unix(tm) should give DragonFly a try. We're based on FreeBSD 4, the most stable and the fastest FreeBSD version we've ever released as a project (the next closest one in terms of quality was FreeBSD 2.2.5). In DragonFly, we work hard to maintain backward and forward compatibility with FreeBSD 4. We even have two compilers in our system, gcc2 and gcc3, and all the infrastructure in place around them to generate either FreeBSD 4 binaries or the newer gcc3-format binaries. In fact, I do much of my DragonFly kernel development on a FreeBSD userland and just compile and copy the kernel over and reboot.
HP: Anyone can try DragonFly, whether it fits their needs or not is up to them to decide. Currently, the DragonFly community is pretty much a mix of various types of people, some are taking the "hmm, interesting, let's wait-and-see" approach, while others are dissatisfied with SMP methodologies adopted by FreeBSD or are interested in upcoming features of DragonFly.
Note, we are not trying to vouch that DragonFly is very stable right now, because there are still a lot of unstable changes to be made. SSI is definitely going to be one of many, although most people have found it reasonably stable for their work.
What main advantages does DragonFly BSD provide against Net/Open/FreeBSD?
MD: Well, our primary competition (if competition is even the right word) is FreeBSD. FreeBSD had always been on the forefront of implementing new ideas and if DragonFly is about anything it's about implementing new ideas. We are not trying to support every platform under the sun (NetBSD) and while security is important to us, it isn't a focus like it is in OpenBSD.
What DragonFly aims to provide is a SMP-capable infrastructure that is inherently easy to understand and develop for. Our primary advantage over FreeBSD is in the methodologies we have chosen to follow in the development of the kernel's infrastructure. We feel that in the long term our infrastructure will be easier to develop and maintain and lead to levels of stability that rival and surpass the mid 4.x releases of FreeBSD.
We have some long term goals, such as SSI (Single System Image) support, which are going to be years in the making, and we have chosen to develop a number of models, such as a new threading, messaging, and IPI messaging model, which we feel will be superior to the other BSDs, and Linux, particularly when used in an SMP environment.
JS: At the moment, we have the speed of FreeBSD 4 and the ongoing work to support SMP. Ask me at the end of the year and DragonFly might be the IA32 reference for performance.
JH: We have the most advanced networking stack and have even more improvements in the pipeline, so the disparity will only get larger as time goes by. In addition, our MP support is a departure from the outdated SMP model. Today's MP hardware is not symmetric, so if you were to add MP support to an OS today, why not use techniques that are more suitable to today and tomorrow's technology? Our networking stack is designed to scale to large number of processors, more than the 2 or 4 that SMP hardware can effectively run on today.
HP: It would not be right to say FreeBSD is a competition because our code-base started with that as the root; but I will not deny that we have a very practical and sane goal-set in mind.
Our advantage from my point of view is that, we are blending in technology and code from other BSDs and Linux (not directly taking code in the latter case). Net/Open/FreeBSD have their own individual advantages, portability, security and implementing new ideas, respectively.
For e.g., NetBSD has made some interesting advancements in their VM system and in its network stack, we occasionally take a peak at them and see how those changes can be better integrated with our model. Most of the times, it would mean tailoring the original code to a large extent, but at the end of the day, the result would be much better code and performance.
Pretty much what Matt said about long term goals, the end result should be practical and not losing out on single-processor performance. We plan to clean out the unnecessary items that we have inherited and create smaller, faster and efficient replacements. We believe that creating a powerful abstraction (like LWKT messaging) is much more important than just implementing a subsystem because everyone else in the neighborhood is doing it.
It seems that you have chosen a development cycle more similar to OpenBSD than FreeBSD, why?
MD: I wasn't comparing development cycles between the BSDs so it is coincidence. But possibly it is coincidence borne out of necessity. We do not have the developer resources that FreeBSD has and so our release cycle is going to be slower (perhaps twice a year instead of every 4 months), and we cannot maintain half a dozen release branches in parallel so end-users will have to upgrade to keep up with developments. We do plan on having a stable tag that at least tracks security changes between releases, however.
JS: We don't have the resources to follow a CURRENT + STABLE + RELEASE system like FreeBSD does. It would also slow down development a lot, which is not very useful in the current state of DragonFly.
I don't think we will announce releases every half year, but that isn't decided yet. It puts pressure on us as team, which is both positive to get stuff ready, but also means overhead.
HP: Not by choice as Matt said, but the OpenBSD development and release cycle is good to keep most bugs out of the system, in my opinion. Secondly, we lack the resources and hardware to keep up with tons of release cycles. Not a bad time to say, the team members would appreciate to have different sorts of hardware if possible.
Right now, the only people with any powerful hardware is David Rhodus and Matt. Both have been very generous in providing resources when possible to help development.