Published on (
 See this if you're having trouble printing code examples

How to Decide What Bugs to Fix When, Part 2

by Scott Berkun, author of The Art of Project Management

When we last discussed bugs, we talked about essential no-frills tactics like triage, prioritization, and naked co-ed blindfolded palm reading (a lie to get you to look at part 1). But here in part 2 we're moving up. Now the stakes are higher: more investment, but greater returns.

Level 3: Exit Criteria

How do you know when you're done working on something? Some things, like chocolate chip cookies and lasagna, show on the surface that they're done. But more complex things, like software, aren't as transparent (or as tasty). You need to plan out, in advance, how you'll know when you're done. If you don't do this you'll spend hours arguing over whether code is done enough or not. If you're smart, and take the time earlier to define exit criteria, you'll set up your team to spend those hours coding instead of arguing.

An exit criteria is anything that must be true before the software can be called complete. The simplest exit criteria is time. You state an exact date and time when work will be finished, regardless of how many bugs or how much unfinished work remains. This criteria is easy to follow, but doesn't say anything about quality. You can meet the goal of shipping in a week whether you have 10 percent of the features working or 90 percent. Unless that date is built out of high-quality historic trend-influenced estimates, the date is just an arbitrary unit of time. Another obvious, and also bad, exit criteria is opinion. Your VP can be a living exit criteria, only allowing the software out the door when he feels it's ready. Even with a brilliant VP, this dependence is bad because if she gets hit by a bus, your project's fate is tied to hers. You need something written down to accelerate communication, and if it stays in someone's head there will be a long line of people outside their door.

Real exit criteria focus on ways to measure quality. How many defects, of what type, for what parts of the software, are acceptable? If you've been following along from part 1, you know about bug priority and severity, two easy ways to categorize defects. But what's still undefined is which areas of the project should be prioritized over others. Do all priority 1 bugs need to be fixed? Or only all priority 1 bugs for certain important areas? If you don't know, your team doesn't know (and you'll be arguing over each bug). It's an act of leadership to define exit criteria. It requires thought about the project as a whole and when it's done right it has a catalyzing effect on the entire organization.

Related Reading

The Art of Project Management
By Scott Berkun

The key question is this: How many defects, of which type, for which areas, are acceptable? Here are some guiding questions to help:

Any exit criteria you create should be part of any work specification. Each feature or design should include a section explaining how the team can verify that the work is finished. Write down what test cases must be passed, what performance numbers must be reached, what usability metrics must be achieved, or what kinds of bugs must be fixed. If you can't write it down, you haven't thought hard enough about the feature you're building, or the customer you're making it for.

To keep exit criteria simple, you might want to have two sets: one baseline set that applies to the entire project, and additional criteria that are feature-specific. With this approach, the only features that get customized exit criteria are ones that have higher or lower quality than the rest of the project. (Slacker hint: this is less work, and the burden for the first set can be given to one senior person).

Remember: you can always adjust exit criteria during a project. You're not going to get it all right out of the gate. But by putting them in place you both clarify for the team that quality is important and give them some tools for guiding their work towards a quality outcome. It's fine if periodically the criteria are improved (not simply changed) based on discussions with customers and the team. Even if arguments arise when these changes are discussed, people will be arguing about the project (and customer) view of quality, which is a more effective place than the bug level to debate.

Level 4: Early Planning

Now you are ready to turn the corner on bugs. If you have the first 3 levels going it's time to step out of tactics and move into strategy. To get above mediocrity in anything you have to find ways to spend your time on advanced problems, not the basics. Mastering the low-level tactics buys you just enough time to play lookout for your project and plan for problems before they become serious. So if any of the above has helped your team (or yourself) to become more efficient, carve out some of that earned time for thinking about the next few days, weeks, or months. You want to look for more fundamental things you can do to improve your team's ability to deal with bugs.

The simplest kind of early planning is to look back at your last project (or the last week of the current project). Ask yourself, and others, what didn't go well that is still hurting. Are there bad habits, poor tools, or common miscommunications? Compile a list of areas for improvement and put them in a rough order of urgency or value.

Remedies for these problems may involve buying better bug tracking tools, clarifying who has what triage authority, training your team in better engineering techniques, or improving the way bugs are tracked and managed. It might make sense to have a person dedicated to communicating with customers about bugs, or for you to define a checklist to help customers report issues, raising the quality of bugs as they're entered into the system. A common way to deliver on many of these ideas is to dedicate a full-time person to the role of quality assurance, or if you already have one, to provide them with more resources to do their job ( Another approach is to focus on the early definition of projects. If you can hire a designer or usability engineer to help organize the project from the outset, prioritizing the things that matter most, you can minimize the unneeded features and development work from the start.

If you do develop a list of areas to improve, stack rank it. Put it in the order of the greatest value to you and your project, and then commit to the first one--not the first five, not the first ten, just the first one. Until you've proven to yourself and your team that everyone is capable of making positive changes, you want everyone focused on making that one change work. Help everyone to rally around improving that one thing, including involving them in suggesting improvements and in carrying them out. Make sure to give yourself an exit criteria for the change. How will you know that the change has had the impact you wanted?

Exceptions and Notes

There are some special cases where you can skip the advice mentioned in this two-part essay. If you've been waiting to pounce on me for some horrific oversight, hopefully I've beaten you to it.

Frequently Asked Questions


Software testing: This is a big subject and I've given you a quick, dirty, and skewed introduction. Start here:

Painless bug tracking: An excellent essay from Joel on simple bug tracking.

Examples of bug triage/prioritization: Here's a sampling of how different organizations handle triage. Diversity in these links is intentional.

Software quality assurance FAQ.

In April 2005, O'Reilly Media, Inc., released The Art of Project Management.

Scott Berkun is the best selling author of Confessions of a Public Speaker, The Myths of Innovation, and Making Things Happen. His work as a writer and public speaker have appeared in the The Washington Post, The New York Times, Wired Magazine, Fast Company, Forbes Magazine, and other media. His many popular essays and entertaining lectures can be found for free on his blog at Scott Berkun.

Return to

Copyright © 2009 O'Reilly Media, Inc.