oreilly.comSafari Books Online.Conferences.


How to Decide What Bugs to Fix When, Part 2

by Scott Berkun, author of The Art of Project Management

When we last discussed bugs, we talked about essential no-frills tactics like triage, prioritization, and naked co-ed blindfolded palm reading (a lie to get you to look at part 1). But here in part 2 we're moving up. Now the stakes are higher: more investment, but greater returns.

Level 3: Exit Criteria

How do you know when you're done working on something? Some things, like chocolate chip cookies and lasagna, show on the surface that they're done. But more complex things, like software, aren't as transparent (or as tasty). You need to plan out, in advance, how you'll know when you're done. If you don't do this you'll spend hours arguing over whether code is done enough or not. If you're smart, and take the time earlier to define exit criteria, you'll set up your team to spend those hours coding instead of arguing.

An exit criteria is anything that must be true before the software can be called complete. The simplest exit criteria is time. You state an exact date and time when work will be finished, regardless of how many bugs or how much unfinished work remains. This criteria is easy to follow, but doesn't say anything about quality. You can meet the goal of shipping in a week whether you have 10 percent of the features working or 90 percent. Unless that date is built out of high-quality historic trend-influenced estimates, the date is just an arbitrary unit of time. Another obvious, and also bad, exit criteria is opinion. Your VP can be a living exit criteria, only allowing the software out the door when he feels it's ready. Even with a brilliant VP, this dependence is bad because if she gets hit by a bus, your project's fate is tied to hers. You need something written down to accelerate communication, and if it stays in someone's head there will be a long line of people outside their door.

Real exit criteria focus on ways to measure quality. How many defects, of what type, for what parts of the software, are acceptable? If you've been following along from part 1, you know about bug priority and severity, two easy ways to categorize defects. But what's still undefined is which areas of the project should be prioritized over others. Do all priority 1 bugs need to be fixed? Or only all priority 1 bugs for certain important areas? If you don't know, your team doesn't know (and you'll be arguing over each bug). It's an act of leadership to define exit criteria. It requires thought about the project as a whole and when it's done right it has a catalyzing effect on the entire organization.

Related Reading

The Art of Project Management
By Scott Berkun

The key question is this: How many defects, of which type, for which areas, are acceptable? Here are some guiding questions to help:

  • What did you do last time? Have a baseline of numbers from the previous release. It's arbitrary, yes, but it does give you a point of reference. You can decide to raise or lower quality comparatively and have everyone know exactly what you are talking about. If you've never measured this before, make up the goals (yes, that's right). Keep track of final decisions so you're set up for version 2.0. Sometimes you can look at the quality bar set by a competitor and calibrate.

  • Which areas are most important? Stack rank the areas/features of the current project. Some are more important to customers and higher quality in those areas will have more value. You can set one level of exit criteria for features A, B, and C, and a second set of lower criteria for D, E, and F. Resources are zero sum: your best use of resources is always on the features used most often.

  • What test cases are you using? Exit criteria do not have to focus on bug counts. If you are using test cases for each feature you can define exit criteria on the percentage of test cases passed. If you don't have any test cases (, now is the time to make them. They encapsulate many different quality criteria into a single, often automatable, test. If you create them early, they will not only help with exit criteria, but will focus the development work as it's done. Each check-in will be done against those test cases, warning you early when there are big problems.

  • What performance metrics must be met? Are there attributes of performance that you care about? load time, save time, download time? Focus your metrics on user observable kinds of performance--that way you'll be sure to fix things that impact customers. Performance bugs are often the most frustrating for customers, and tend to require complex work to fix (hint: you want to identify them early). If you have no idea what your performance numbers look like, reread the first point: get a baseline so next time you'll be able to make intelligent decisions.

Any exit criteria you create should be part of any work specification. Each feature or design should include a section explaining how the team can verify that the work is finished. Write down what test cases must be passed, what performance numbers must be reached, what usability metrics must be achieved, or what kinds of bugs must be fixed. If you can't write it down, you haven't thought hard enough about the feature you're building, or the customer you're making it for.

To keep exit criteria simple, you might want to have two sets: one baseline set that applies to the entire project, and additional criteria that are feature-specific. With this approach, the only features that get customized exit criteria are ones that have higher or lower quality than the rest of the project. (Slacker hint: this is less work, and the burden for the first set can be given to one senior person).

Remember: you can always adjust exit criteria during a project. You're not going to get it all right out of the gate. But by putting them in place you both clarify for the team that quality is important and give them some tools for guiding their work towards a quality outcome. It's fine if periodically the criteria are improved (not simply changed) based on discussions with customers and the team. Even if arguments arise when these changes are discussed, people will be arguing about the project (and customer) view of quality, which is a more effective place than the bug level to debate.

Level 4: Early Planning

Now you are ready to turn the corner on bugs. If you have the first 3 levels going it's time to step out of tactics and move into strategy. To get above mediocrity in anything you have to find ways to spend your time on advanced problems, not the basics. Mastering the low-level tactics buys you just enough time to play lookout for your project and plan for problems before they become serious. So if any of the above has helped your team (or yourself) to become more efficient, carve out some of that earned time for thinking about the next few days, weeks, or months. You want to look for more fundamental things you can do to improve your team's ability to deal with bugs.

The simplest kind of early planning is to look back at your last project (or the last week of the current project). Ask yourself, and others, what didn't go well that is still hurting. Are there bad habits, poor tools, or common miscommunications? Compile a list of areas for improvement and put them in a rough order of urgency or value.

Remedies for these problems may involve buying better bug tracking tools, clarifying who has what triage authority, training your team in better engineering techniques, or improving the way bugs are tracked and managed. It might make sense to have a person dedicated to communicating with customers about bugs, or for you to define a checklist to help customers report issues, raising the quality of bugs as they're entered into the system. A common way to deliver on many of these ideas is to dedicate a full-time person to the role of quality assurance, or if you already have one, to provide them with more resources to do their job ( Another approach is to focus on the early definition of projects. If you can hire a designer or usability engineer to help organize the project from the outset, prioritizing the things that matter most, you can minimize the unneeded features and development work from the start.

If you do develop a list of areas to improve, stack rank it. Put it in the order of the greatest value to you and your project, and then commit to the first one--not the first five, not the first ten, just the first one. Until you've proven to yourself and your team that everyone is capable of making positive changes, you want everyone focused on making that one change work. Help everyone to rally around improving that one thing, including involving them in suggesting improvements and in carrying them out. Make sure to give yourself an exit criteria for the change. How will you know that the change has had the impact you wanted?

Pages: 1, 2

Next Pagearrow

Sponsored by: