BSD DevCenter
oreilly.comSafari Books Online.Conferences.


Confessions of a Recovering NetBSD Zealot
Pages: 1, 2, 3

Recently NetBSD defined a set of rules and time slots to manage the release engineering process, and also a new numbering scheme. Do you think this will solve any of the problems you have just outlined?

Charles M. Hannum: Not at all. What we see now is just a mad push to release something, no matter what it is. The gold standard has moved entirely from technical excellence to the time on the clock. This isn't a good solution either, and actually it risks losing a lot of users by putting out really buggy releases. There needs to be some balance.

NetBSD is the only BSD project that kept using XFree86 after the license change in 4.4.0. What is your opinion about it?

Charles M. Hannum: This relates to something I mentioned before. The fact that X is part of the "OS," and uses a custom build process, makes it much harder to update--it's really a substantial amount of work. If it was a separate package, I think we would have seen in use a long time ago--someone would have just spent the few hours to make the package and people would have started using it. It's this method that is really the core asset of the "bazaar" idea.

Do you think that we are going towards a universal package system, something like pkgsrc or OpenPKG, or that in the end every OS/distro will develop its own?

Charles M. Hannum: Well, look at the current situation. If you look at just the major options in the Linux community, you have apt, rpm+up2date and rpm+yast. They have all converged on pretty much the same feature set, but they have different UIs, and the packages themselves are maintained by completely different communities (with their own versioning, patches, and selected options). There's no good reason that they need to have different underlying formats (.deb versus .rpm), but they do.

I think fundamentally what leads to a lot of forks in the Linux distros is NIH syndrome. There's no reason to think this won't apply to package systems.

Oddly enough, though FreeBSD ports, NetBSD pkgsrc, and OpenBSD packages have all forked their maintenance and build structures long ago, and they are now quite dissimilar, the prebuilt package format is still quite similar. I think it would be a good idea to at least merge the format and the utilities, if for no other reason than it would enable sysadmins to learn one set of tools for all three systems. Some work on package upgrading and patching also needs to be done; all three systems handle this poorly.

What do you like in the management and code development process of other BSD projects? Which things would you define as "best practices" and would you try to repeat in the NetBSD Project?

Charles M. Hannum: As I alluded to in my original statement, I think FreeBSD has similar problems, and this is a large part of why Dragonfly was created. OpenBSD does have strong leadership, but unfortunately it's a little too strong, and that has caused a lot of people to not contribute. I don't have a guide to what works best, but I can definitely tell you a few things that do not work.

There are quite a few open source projects that I classify as successful--though what that means could be another debate. Two examples I'll give are KDE and These projects are making good progress and doing really good things.

From your point of view what are the advantages and disadvantages of the Linux development process?

Charles M. Hannum: Well, first we have to define what the "Linux development process" actually is. I think ESR got it wrong. Back then, we had different distros all doing their own thing, making their own sets of kernel patches, and integrating totally different versions of other software. Linux compatibility was terrible--it got so bad that at least one company ported NetBSD's libc and statically linked everything so that their software would actually run on multiple distros.

However, some things have changed. Compatibility has gotten better--though there are still problems. The base Linux kernel distribution has gotten far enough, and complete enough, that there are fewer changes maintained by distros. I think this is in large part because there is a more open and effective (though sometimes seemingly arbitrary) method for getting things into the base kernel.

The thing that has not changed, and that carries through within most distros, is that the individual packages are maintained in fiefdoms. Many of those fiefdoms are impenetrable and inscrutable; it is as if the gate is down, the moat is poisoned, and they occasionally toss a cow over the wall to feed the peasants. In short, I liken the "Linux process" not to a bazaar, but to a set of independent fortresses, all doing their own thing. (BTW, I use such a metaphor only to keep in the spirit of ESR's metaphor.)

This has been a subject of some debate for many years. Back when I was hitting the conference circuit, I sat in on discussions about the problems with this model and how to fix it. The biggest complaint seemed to be the lack of transparency and accountability for upstream and package maintainers, partly because most of them used no version control system at all, or kept all the history private. Because of this, it's much more work to pinpoint bugs, and where they came from. It's also hard to decide whether the maintainer is doing a good job, and whether a fork is mandated, if you have no idea what they're doing.

(I know some people argue that "there's no point in assigning blame," but I'm not talking about blame. Sometimes it's important to understand why a change was committed in the first place; e.g., maybe it was supposed to fix a security hole, but they made a mistake. Obviously you don't want to put the security hole back in.)

In short, I don't see the Linux model as a panacea.

How should open source project leaders assign or remove the commit bit?

Charles M. Hannum: I think the first thing to do is have a firm set of commit standards, as I mentioned. Maybe schedule a particular window during the development process for committing non-functional (i.e. whitespace, function prototype, etc.) changes, so that people can merge branches around them only once rather than a zillion times. Switch to subversion (or git, if that's your choice of Kool-Aid) and make people batch similar commits. Be hard-nosed about removing changes that don't meet standards or break things.

The Linux kernel effectively has such standards now because everything is filtered through a small set of people with reasonable taste (and they're not totally overloaded the way Linus used to be). That's not to say I agree with all of their choices, but they seem to do a good job of enforcing their standards and principles, and I have to respect that.

NetBSD today does a very poor job of setting and meeting standards. I created the mythos of NetBSD having "clean" code, and even I don't buy it any more. By "clean," I certainly never meant "no trailing whitespace"--the idea was that changes were publicly reviewed and cooperatively developed, and we didn't do quick minimalist hacks. This isn't how things are done now; in fact, at least one major architectural change recently was developed entirely in secret and committed wholesale, with zero public review. Despite its rather different beginnings, even the Linux community is more open than that now.

I'm not the only one who was affected by the recent termination of accounts. Of particular note are two other names: Lennart Augustsson and Darren Reed. Lennart single-handedly wrote the entire USB stack in NetBSD, and many of the device class drivers; Darren wrote ipfilter, which is the only fully supported firewall and NAT solution in NetBSD (though there has been some ongoing work to integrate OpenBSD's pf). These two have rather obviously made significant contributions, and their termination is a major loss for the project.

The announcement of this action also contained several factual errors.

After your email made it to the mailing lists of every BSD project (including OpenSolaris), what has happened? What do you think will happen in the near future?

Charles M. Hannum: The overwhelming majority of feedback I've gotten privately (and there has been a lot) is positive. I've heard from many former users and developers who abandoned NetBSD for reasons that I stated. Even comments from Linux and FreeBSD developers have generally been positive. It's been interesting to read. There have been quite a few people calling for a fork, and even offering help, but I'm not going to state an opinion on that at this point.

Not surprisingly, there has been quite a bit of flamage on the NetBSD mailing lists--mostly people trying to stand their ground and pretend that there is nothing wrong.

I think the most interesting part is that several users have specifically said they "didn't know anything was wrong [with the organization]." This is at least partly my fault; I stepped out for a while and waited to see how the circus proceeded without me. All I can say is that it's been an unmitigated train wreck--they've repeatedly grabbed more authority and kicked people out of various positions, made people sign Draconian agreements, done almost everything in secret, and failed to actively promote the project. I find it hard to imagine that an open source project could have worse management.

Federico Biancuzzi is a freelance interviewer. His interviews appeared on publications such as,,,,,,, the Polish print magazine BSD Magazine, and the Italian print magazine Linux&C.

Return to the BSD DevCenter.

Sponsored by: