Let's say that you're a programmer working on a wildly successful open source project in your free time. You'll typically go home from work and check your inbox, find that a new patch was committed to the Subversion repository for the project, and review it carefully. You might post to a thread on a mailing list about a new feature that's being planned, voting against the feature because you feel it will cause serious design problems down the road and suggesting a better way to solve the problem. In general, you feel strongly that nobody should commit a broken build; that new contributors should be able to easily install the software; and that it's important to get as many people reviewing the code as possible. You're not alone. Many of the most successful open source projects are run this way, and the vast majority of open source developers feel that this is the right way to run a project.
If you're also a professional developer at your day job, then you may have tried to bring some of those great practices to work with you. If you have, then these excuses will sound very familiar: "We have a business to run." "Those ideas might work in a perfect world, but we need to concentrate on our code." "It would be great to do the project like that, but we just don't have time." It would be frustrating to contribute to a successful open source software project as a hobby, only to work on a challenged one professionally.
When people talk about the benefits of open source software, one of the first things to come up is "more eyeballs": when there are more people carefully looking at the code, it leads to better software. Open source projects rely heavily on open discussion of every aspect of the project: what features will (and won't!) be written, the best way to implement those features, how to keep the design flexible and maintainable for later additions, and how to address problems when they come up. When there are disputes, they are resolved often democratically and always in an open manner--and, generally, anyone can take part in the discussion, regardless of how many years of experience they have or their role on the project. And the most successful open source projects always seem to be the ones that adhere most closely to these ideas.
The GIMP project makes an excellent case study. The developer site lays out guidelines for every aspect of software development. It describes how the project is managed and who coordinates it. The site describes how developers use CVS, submit patches, and create documentation. There is a process for defect reporting, triage, repair, and verification. There are standards for building, packaging, testing, and releasing new versions. There are mailing lists for announcements, developers, users, documentation, and even for the website itself. There are tutorials and FAQs to help new developers get oriented and learn how the GIMP team works. Everything that happens on the GIMP project is planned and agreed upon by the team. And any new developers can go back and see exactly which decisions were made, who made them, and why. And the developers really do their best to observe these rules. No wonder GIMP is widely seen as one of the most successful and popular open source projects.
But the GIMP project is not alone in this. The vast majority of visible and successful open source projects are run in the same way. Some, like GnuCash, have all of this information on a wiki. Others, like KDE, drive their developer documentation from a Subversion repository. What they all have in common is extensive documentation and written policies on every aspect of how a developer is expected to interact with the rest of the team--and strong support from the entire team for maintaining and enforcing those policies. Almost every successful open source project has a similar website, and a similar commitment from the team.
Yet it is rare to find a corporate environment where the project team has anything approaching the level of planning, documentation, or review found in successful open source projects. For some reason, as soon as a budget and a deadline are involved, all of the lessons we've learned over the years and applied successfully to open source projects seem to fly out the window. Discussions of what features are in and out of scope never seem to take place. When a corporate developer tries to bring people together to discuss the design of the software or to make plans for how code is added or maintained, he's met with groans about "yet another meeting." What's worse, when a developer actually manages to get everyone on the team to agree to do design or code reviews, their managers often tell them to be careful "not to spend too much time" on it, implying that any activity other than writing code is somehow "frivolous" or "over-engineering." After all, they're "only" programmers, and their opinions only matter on issues directly related to the construction of the software. The direction that the software is taking should be "dictated by the business," and the programmers should just stick to writing code.
However, it's well known that corporate projects routinely fail to produce quality software. or even any results at all! Many important studies have repeatedly reported that fact, such as the Standish Group's annual CHAOS Report (which as recently as 2004 has shown [PDF] that less than one-third of corporate projects are considered successful).
Yet many large-scale open source projects succeed, and under much more difficult conditions: no budget, a widely geographically distributed team, and a completely volunteer workforce in which people take on only the roles and responsibilities that they feel like taking on, and put in only as much time as they want to devote. So how do open source project teams ensure success under conditions that would cause all but the most disciplined corporate project teams to immediately self-destruct? In our book, Applied Software Project Management (O'Reilly 2005; see our companion website), we introduce five basic principles that should be applied to every software development project. These principles, if adhered to, will help lead to success on any project, open source or proprietary. These principles are:
Each of these principles is important, and each of them is routinely ignored in corporate environments--to the detriment of project success. There are chronic behaviors that routinely occur on corporate projects, which violate these principles. On the other hand, by employing time-tested and accepted practices or means of collaboration, successful open source projects are very good at putting these principles into practice.
It seems straightforward to say that everyone should always be truthful, but it's actually more challenging than many people realize. Most developers who have been around for a while have been in a situation where they changed the direction of a project by stretching the truth. For example, say a user wants a feature that will be very challenging or tedious to build, or for which the implementation isn't immediately obvious. For some programmers, it's easier to say that it simply can't be done than to take the time to do the required research or analysis. (Be honest--most of us have done this at least once in our careers, the authors of this article included!) In corporate projects, many features have been scrapped over the years because developers did not want to develop them--even when the company would have been better off in the long run had the developers built the software that was requested of them. Requirements will sometimes get trimmed and schedules will sometimes get padded when developers don't feel that a feature is easy to build--with or without the necessary investigation.
It's not just developers who are tempted to stretch the truth. A vice president or CTO may announce that a product will be done in a month, when everyone (well, everyone but the customers) knows it won't. A manager will cut out quality assurance activities without telling anyone. The company will decide to fund or not fund a project without explaining why. The entire direction of a project can be changed out from underneath the team on the whim of a senior manager, without any explanation whatsoever. Most of us have been in the awkward position of being told by a superior to ship software that was simply not ready for prime time.
So how do we make sure to tell the truth all the time? Building transparency into a development organization requires open discussion--open not just to the developers, but to everyone involved in or relying on the project. In a transparent organization--one built on honesty, rather than expedience--every idea is evaluated on its merits. Alternatives are openly discussed, in a forum where everyone feels comfortable giving an honest opinion. In that forum, the right decision will get made. A solution is not rejected because it's time-consuming, tedious, or a lot of work. If that solution is genuinely going to blow the schedule or cause serious technical problems, the decision may still be to reject the idea. But only through honesty and real dialogue will the correct decision--meaning the decision that's in the best interest of the project and the organization--be made.
Open source projects can teach us a great deal about transparency. Every successful, large-scale open source project has an open, archived discussion forum. They use mailing lists, wikis, discussion web pages, newsgroups, IRC channels, and other forums to make sure that developers and other project team members have many opportunities to bring up important project issues. But transparency requires more than just installing a wiki or mailing list software; it requires a real commitment from the team to make decisions openly and to keep everyone in the loop, and this is where open source projects really excel.
In contrast, managers associated with many corporate software projects will hand down decisions to the team without consulting them at all. The most common reason for this is the schedule: for some reason, many people in that position feel that asking the developers for an opinion is just begging for a schedule delay. And this is not just a matter of heavy-handed micromanagement; in fact, often the opposite is true. If a project is being built, a senior manager, a stakeholder, or a user may actually avoid bringing up--or may even shoot down--a good idea, in order to keep from disturbing a project that seems to be making progress. Open discussion is squelched in the interest of time. Ironically, much of this discussion is actually a time-saver: the developers are almost certainly able to evaluate whether or not an idea is worth implementing more effectively than the manager is. But without an open forum and some encouragement to voice their opinions, the project team will miss this important chance to make the right decision--that is, until very late in the project, when it is much more expensive to make changes.
Many corporate environments are dysfunctional because of the company hierarchy. For example, a vice president who is two levels above the development team leader can make sweeping changes to the project at any time, with or without support from the developers. When this happens, the team is forced to scramble to make the necessary changes to the code. And once a bad decision is made far above the team in the company, it has to be accepted as law--even if it causes rework, forces bad architecture decisions, or causes the entire project to grind to a halt.
As most experienced lead developers know, getting a senior executive to change his mind once a "final" decision has been made is often next to impossible. A programmer may be highly respected on the team, and may have gathered an enormous amount of evidence that a decision handed down from above will cause harm to both the project and the balance sheet. Yet when this evidence is presented, all it seems to do is threaten the executive's sense of hierarchy and consistency. In fact, it can often cause him to dig in his heels, and stick more resolutely to the bad decision. If you're a programmer in a large corporation, it's likely that you can think of several times in your career when you've had a good project cancelled because of a senior executive's whim. Or the opposite may be true--you may recall a project that clearly should have been terminated, but which was kept alive because it was somebody's pet project.
This would never happen on an open source project. And it's not because there is no hierarchy. As Karl Fogel points out in his excellent book, Producing Open Source Software (O'Reilly 2005), many successful open source projects are run by a maintainer he refers to as a "benevolent dictator." According to Fogel, in a project like this, "final decision-making authority rests with one person, who by virtue of personality and experience, is expected to use it wisely." As Fogel explains:
Being a BD requires a combination of traits. It needs, first of all, a well-honed sensitivity to one's own influence in the project, which in turn brings self-restraint. In the early stages of a discussion, one should not express opinions and conclusions with so much certainty that others feel like it's pointless to dissent. People must be free to air ideas, even stupid ideas. [emphasis ours] It is inevitable that the BD will post a stupid idea from time to time too, of course, and therefore the role also requires an ability to recognize and acknowledge when one has made a bad decision-though this is simply a trait that any good developer should have, especially if she stays with the project a long time. But the difference is that the BD can afford to slip from time to time without worrying about long-term damage to her credibility. Developers with less seniority may not feel so secure, so the BD should phrase critiques or contrary decisions with some sensitivity for how much weight her words carry, both technically and psychologically.
A good senior executive or manager should approach a software project in exactly this way. Yet in a corporate environment, it's rare to find such a person. Instead, the corporate environment seems to breed an attitude of infallibility. This may simply be because a senior person is paid more, and he feels that he needs to justify his salary. Or he may simply believe that the mere fact that he is paid more and is higher up in the organization's hierarchy means that any lower-paid subordinate who disagrees with him is automatically wrong. ("The boss is always right!")
That's not to say that there are not good corporate projects, or that there are no good senior managers and executives. There certainly are, and those people do trust their teams. When team members have ideas, the manager listens; when the manager disagrees with a team member, he argues it on the merits of the idea, and doesn't simply pull rank. (And, more often than not, he loses the argument gracefully.)
But the corporate environment seems to be set up to pathologically encourage this problem, and to breed distrust of the team among senior managers. For a leader--whether an executive, middle manager, or project lead--to do a good job in a corporate environment, she must be aware of this shortcoming and actively work against it. If she follows the "benevolent dictator" model adopted by many open source projects, that's exactly what will happen.
Open source projects have a long history of embracing the idea of reviews. As Eric Raymond points out in The Cathedral and the Bazaar (O'Reilly 2001), "Given enough eyeballs, all bugs are shallow." He calls this "Linus's Law" (in reference to Linux creator and maintainer Linus Torvalds):
My original formulation was that every problem "will be transparent to somebody." Linus demurred that the person who understands and fixes the problem is not necessarily or even usually the person who first characterizes it. "Somebody finds the problem," he says, "and somebody else understands it. And I'll go on record as saying that finding it is the bigger challenge." That correction is important. But the key point is that both parts of the process (finding and fixing) tend to happen rapidly.
This is an excellent description of why reviews are important, and why they work so well to remove defects from software. Yet in many corporate projects, it is nearly impossible to get the time to perform reviews. Vision and scope documents--a short description of who needs the software, what the needs are, and how the project will meet those needs--are never distributed, read, or commented upon. (Often, they're not even written, and the team is never given a chance to understand or influence the scope or purpose of the project that they're working on!) Database designs, class models, and other technical documents are not inspected for mistakes or defects. And most importantly, code is committed to the source control repository without ever having been code-reviewed.
These important activities are all avoided because a manager arbitrarily decides it's inefficient to do reviews. Yet study after study (and, more importantly, the direct experience of many developers) contradicts this. It really is true that "more eyeballs" leads to better software.
So why do corporate project teams and managers see reviews as "bureaucratic," while open source projects see them as vital? There's a simple explanation: people have a natural tendency to avoid seeing their work criticized. It is rare for most professionals outside of a software engineering team to place their work under a microscope. And what's more, when a person's livelihood is attached to his work, then he may not want his "mistakes" to be shown off to the rest of the team. In essence, many people intuitively feel that reviews amount to a wholesale admission that team members got paid to make mistakes. And that just doesn't sit well with a lot of people.
But that's not enough to explain the intense animosity towards reviews that exists on many corporate teams. After all, everyone does make mistakes; catching mistakes early and building a better product is clearly the most effective way to protect everyone's professional reputation. So why the resistance?
It may simply be that the standard of quality in open source projects is higher than that of corporate projects. Many corporate teams are pressured by their senior managers to build the lowest quality software that they can get away with without losing customers. There is seemingly sound business reasoning behind building only what's necessary. This reasoning holds that what's best for corporate profits is to ship the cheapest software possible. Many corporate managers interpret this to mean building the lowest quality software that customers will still buy, even if that means shipping it with multiple and severe known defects. (Of course, that reasoning often makes sense only in the short run; eventually a reputation for building poor software can sink a company!) But even if this were always true, it would be a poor argument for eliminating reviews, because it's much cheaper to find and repair defects early on in the project than it is to build them into the software, only to have to dig them out much later. While this is a false argument, it's a highly influential one; as a result, corporate teams routinely ship poor quality software that has not been reviewed.
This is the polar opposite of the attitude of open source projects. On a well-run open source project, code is carefully reviewed and accepted before it is committed to the version control system. Project contributors feel that it's their right--and not just an obligation!--to make sure that the project starts out on track and stays there. They feel privileged to be able to review each others' work, and to be included in each design decision. They don't dismiss it as a drain on their time; they value it as a way to make the project better. The project team as a whole buys into the direction of the software and its implementation. By taking part in these decisions, all team members end up feeling greater ownership and pride in the final product, and the team builds and releases better software.
Say a junior developer is working on a bug fix, and realizes that the entire design of the part of the software he's working on is flawed. He comes up with an improvement and brings it up with his boss. If this is a typical corporate project, it's likely that this junior person will see his idea shot down almost immediately. His boss will brush him off, saying that it will take too long to implement, or that it's going to be too hard a change to make, and nobody else on the team will even hear about it. But what his boss is really thinking is, "How can this junior guy spot a mistake that none of the senior people caught? He must not understand all of the details."
Say that same junior developer later that week brings it up with a senior team member. The senior guy is in a good mood, and entertains the thought. Suddenly he realizes that there really is a serious problem with the project! He goes to the boss, but this time the boss listens. With the senior person's backing, the suggestion has a lot more credibility. Now it will be heard and acted upon.
This is no way to run a software project. Good ideas come from everyone, not just the senior team members. But corporate environments frequently discourage innovation from the "bottom of the food chain." In fact, many corporate environments discourage innovation altogether, dismissing it as risky and potentially expensive. There's often a sense that no new innovation should be adopted unless it has been successfully implemented in another project. This is yet another way that team members' opinions can be squelched on corporate projects. In the end, many bright (and often young) project team members routinely come up with new and insightful ideas, only to have them shot down because they didn't follow proper channels or have enough seniority--or because a senior manager simply didn't understand them.
This is another area in which corporate projects can learn from successful open source ones. Blake Ross, a 19-year-old hacker, started the Firefox project when he was dissatisfied with the way the Mozilla project was going. And he's not alone. In Producing Open Source Software, Karl Fogel points out that open source teams judge their contributors solely on their code contributions and their posts to mailing lists, message boards, and discussion groups. He relates the story of a respected GNU Emacs contributor who turned out to be a 13-year-old living with his parents. No corporate project would ever give either him or Ross the time of day, yet the open source teams treated them both with respect solely because of the quality of their work. Open source project teams adopt a philosophy of treating all developers equally, judging them solely on the basis of their work.
Even when corporate managers have acknowledged that many of these practices are useful, and have put together a software process that uses them, they may routinely ignore their own rules when they feel like they are under pressure. As deadlines get closer, management looks for ways to "trim the fat." They cut any practices that they feel don't actively contribute to the bottom line, often starting with reviews (the very thing that makes many open source projects so successful). If a project is running late, many important activities start to seem frivolous to many managers: that three-hour meeting that brings the programmers together to review the architecture or new patches, the important refactoring exercise that improves the design of an important object, the building of automated unit tests to catch bugs before the software is shipped. They'll argue that these things are too expensive to "waste" those programmers' hours on. Programmers are told to cut any activity that doesn't directly produce code.
When this happens, the team is forced to cut the two-hour meeting that would have saved them from wasting weeks going down a wrong path. Or they don't spend four hours writing unit tests, which would have prevented serious defects that eventually cost days of time and delayed the ship date. Thinking about how to do the work before doing it is always faster than blindly plunging forward.
Well-run projects require a lot of collaboration. Much of that collaboration is understood and planned from the very beginning. Team members need to talk about the work they will do, and they need to keep talking about it to make sure they're on track. When a programmer writes code but doesn't fully understand its purpose, he ends up with an overcomplicated and unmaintainable code base, because he inevitably goes down many wrong paths that can't be fully undone.
Many poor-quality architectural decisions have been made by "locking" a programmer in an office and telling him that the only thing that matters is the code he writes. He's told by his well-meaning boss not to waste his teammates' time with reviews, and he's told not to "overthink" the problem and to just build the most obvious solution. This is a recipe for disaster when trying to build a complex system.
The reason that most teams agree at the beginning of their project to build the software in a way that involves a lot of communication, collaboration, and review is that those are practices that work. Any time that they're circumvented, treated as frivolous, or cast aside as "overengineering," the project suffers time delays. Every mailing-list post, every team meeting to discuss a design decision, every knowledge transfer session between a senior and junior team member, and every single review saves more time than it costs. Yet when a boss feels that his project is under pressure, these are exactly the practices that are cut out in order to "save time." This inevitably leads to a great deal of rework that could have been avoided. By dropping these practices, he's showing the team that he values time spent writing bad code over time spent avoiding it.
In contrast, open source projects don't compromise on time-tested practices. They tend to have strict rules about how a new contributor joins, what privileges he or she gets, how defects are tracked, how changes get reviewed and are handled, and how changes in the direction of a project are made. The team even has rules on how to change those rules. Even when the project is facing pressure, like when there are public and embarrassing bugs that need to be addressed right away, the team does not deviate from these rules. The reason that people are so determined to stick by the rules is that everyone feels ownership of them. Team members came up with the method of working, and everyone accepts the ground rules when they join the project.
A good example of an open source team sticking to its rules happened recently (at the time of this writing) in the Firefox project. Several serious security holes were discovered in version 1.5, which had only been out for a little over a month. Security holes can be embarrassing and potentially very damaging to a popular browser, and there was enormous pressure on the team to fix them quickly. It certainly must have been tempting to try to slap together "quick" patches to seal up these problems. Yet they carefully followed their review requirements. Each defect was entered into Bugzilla, reproduced, repaired, regressed, and verified, all in a way that is completely transparent. What they did not do was apply a hotfix directly which had not been verified and reviewed--something that corporate teams do all the time. The Firefox team released the new version only after all of the defects went through their process. Maybe this is why Firefox is one of the most widely used and depended-upon pieces of software available--because it has a reputation for good security and high quality.
One important lesson corporate teams can learn from this is that the teams themselves need to be involved in setting the goals and rules around which they will work. In a company that does that, asking the team to deviate from the most efficient work processes is the same as asking them to throw away their own principles and ideas. This often serves as an effective balance against the urge to acquiesce to time pressure.
There is a lot that corporate projects can learn from their open source counterparts. But is there anything that open source project teams can learn from the way corporate projects are structured and run?
Certainly. Plenty of successful software is built by corporate teams, and some of that software is even good! That's actually an important distinction: successful software is not necessarily good software, and good software is not always successful. This seems contradictory, but it's actually pretty easy to understand.
One important shortcoming of open source projects is that oftentimes they are just done on a programmer's whim. On the other hand, the problem with many corporate projects is that they're just done on a vice president's whim. The best projects--in both worlds--are the ones that actually fulfill real needs of real users, and aren't just someone's pet project. In the open source world, this happens most often when the programmers' needs coincide with those of many other users. Apache, Firefox, Linux, Gnome, GnuCash, GIMP--these are all projects that their programmers needed, but they are also products that many non-programmers found useful. When the needs of the programmers overlap the needs of other users, it works out very well for everyone.
Unfortunately, there are few open source requirements gathering efforts. There are no open source demographic research departments or marketing teams. The fact that corporations have to go to investors or shareholders to explain how they are doing requires that they justify their projects in terms of actual user appeal. In other words, successful corporate projects must fill real needs in order to get funding. If a user has a need, he will always choose the poorly engineered, difficult-to-use, low-quality product that actually fills the need over the beautifully engineered, highly usable, stable product that doesn't. But software teams don't have to make the choice between software that's well-engineered and software that's useful. By taking the best practices from both corporate and open source worlds, it should be possible to build products that achieve both.
Andrew Stellman comes from a programming background, and has managed teams of requirements analysts, designers, and developers. He and Jennifer Greene formed Stellman & Greene Consulting in 2003, with a focus on project management, software development, management consulting, and software process improvement.
Jennifer Greene has spent the past 15 years or so building software for many different kinds of companies. She's built software test teams and helped lots of companies diagnose and deal with habitual process problems so that they could build better software. Jennifer founded Stellman & Greene Consulting with Andrew Stellman in 2003. For more information about Jennifer, Andrew Stellman, and their books, visit http://www.stellman-greene.com.
Return to ONLamp.com.
Copyright © 2009 O'Reilly Media, Inc.