ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


More Test-Driven Development in Python

by Jason Diamond
02/03/2005

In the first article in this series, Test-Driven Development with Python, Part 1, I started to build an event tracking application using the principle of test-driven development. Before writing any new code or making changes to any existing code, I wrote a failing test first. When the new test started passing, I stopped and moved on. It's a simple technique, but requires discipline in order to apply it successfully.

If you haven't read part one yet, I suggest you do so before continuing on in this article. That will familiarize you with the problem I'm trying to solve and bring you up to speed on my current predicament.

A New Design Emerges

As indicated in the previous article, I'm starting to think that the single DatePattern class I have so far is responsible for too much. It's only 30 lines long at this point, but I don't mean its length when I say "too much." What I mean is, conceptually, it does four different things; I even started adding a fifth before I stopped myself.

What are its four different responsibilities? It has to determine if the year matches. That's one. The month is another, as are the day and the weekday, too. Can I break those responsibilities into four different classes? I'm sure I can, but should I?

Not only that, but for each component of the pattern, I have to check to see if it's a wild card and act appropriately. Maybe I need yet another class to encapsulate that logic.

I can't answer these questions by just thinking about them, so I will explore by writing a few tests:

class NewTests(unittest.TestCase):

    def testYearMatches(self):
        yp = YearPattern(2004)
        d = datetime.date(2004, 9, 29)
        self.failUnless(yp.matches(d))

red

Note that I've put this new test case in a new fixture. If this turns out to be a dead end, I can easily delete this fixture and continue onward with the old. If this ends up being a good idea, I can just as easily delete the old fixture (and the code it exercised, the DatePattern class).

This new test case, of course, fails. Here's the code required to make it pass:

class YearPattern:
    def __init__(self, year):
        pass

    def matches(self, date):
        return True

green

Boring. How about another test?

def testYearDoesNotMatch(self):
    yp = YearPattern(2003)
    d = datetime.date(2004, 9, 29)
    self.failIf(yp.matches(d))

red

Now to make the new test pass without breaking the old one:

class YearPattern: 
    def __init__(self, year):
        self.year = year

    def matches(self, date):
        return self.year == date.year

green

Perfect.

What's the point? Before I can show you that, I need to write a few more tests:

def testMonthMatches(self):
    mp = MonthPattern(9)
    d = datetime.date(2004, 9, 29)
    self.failUnless(mp.matches(d))

def testMonthDoesNotMatch(self):
    mp = MonthPattern(8)
    d = datetime.date(2004, 9, 29)
    self.failIf(mp.matches(d))

def testDayMatches(self):
    dp = DayPattern(29)
    d = datetime.date(2004, 9, 29)
    self.failUnless(dp.matches(d))

def testDayDoesNotMatch(self):
    dp = DayPattern(28)
    d = datetime.date(2004, 9, 29)
    self.failIf(dp.matches(d))

def testWeekdayMatches(self):
    wp = WeekdayPattern(2) # Wednesday
    d = datetime.date(2004, 9, 29)
    self.failUnless(wp.matches(d))

def testWeekdayDoesNotMatch(self):
    wp = WeekdayPattern(1) # Tuesday
    d = datetime.date(2004, 9, 29)
    self.failIf(wp.matches(d))

red

Here's the code to make these tests pass:

class MonthPattern:
    def __init__(self, month):
        self.month = month

    def matches(self, date):
        return self.month == date.month

class DayPattern:
    def __init__(self, day):
        self.day = day

    def matches(self, date):
        return self.day == date.day

class WeekdayPattern:
    def __init__(self, weekday):
        self.weekday = weekday

    def matches(self, date):
        return self.weekday == date.weekday()

green

Is it obvious where I'm going with this yet? If not, this test should make it clear:

def testCompositeMatches(self):
    cp = CompositePattern()
    cp.add(YearPattern(2004))
    cp.add(MonthPattern(9))
    cp.add(DayPattern(29))
    d = datetime.date(2004, 9, 29)
    self.failUnless(cp.matches(d))

red

What I've stumbled across here is an instance of the Composite Pattern. (Interestingly, this is the same name as my class--that wasn't intentional, I promise.)

Design Patterns

A composite is basically an object that contains other objects, where both the composite object and its contained objects all implement the same interface. Using the interface on the composite should invoke the same methods on all of the contained objects without forcing the external client to do so explicitly. Whew, that was a mouthful.

Here, that interface is the matches method, which accepts a date instance and returns a bool. Python is a dynamically typed language, so I don't need to define this interface formally (which is fine by me).

How do I implement the composite? Like this:

class CompositePattern:
    def __init__(self):
        self.patterns = []

    def add(self, pattern):
        self.patterns.append(pattern)

    def matches(self, date):
        for pattern in self.patterns:
            if not pattern.matches(date):
                return False
	return True

green

The composite pattern asks each of its contained patterns if it matches the specified date. If any fail to match, the whole composite pattern fails.

I have to confess that I cheated here. I wrote more code than I needed to pass the test! Sometimes I get ahead of myself. Sorry. It turned out OK this time because all of the tests are passing, but I need to create a test that should not match, just to be sure I have everything working correctly:

def testCompositeDoesNotMatch(self):
    cp = CompositePattern()
    cp.add(YearPattern(2004))
    cp.add(MonthPattern(9))
    cp.add(DayPattern(28))
    d = datetime.date(2004, 9, 29)
    self.failIf(cp.matches(d))

green

Cool. It passes.

It might be a little difficult to see this, but the composite contains a DayPattern that matches the 28th and I'm matching it against the 29th, which is why I expect the matches method to return False.

So I can match dates again. Big deal--I was already doing that. What about wild cards?

I'll write a test to match my anniversary with the new classes:

def testCompositeWithoutYearMatches(self):
    cp = CompositePattern()
    cp.add(MonthPattern(4))
    cp.add(DayPattern(10))
    d = datetime.date(2005, 4, 10)
    self.failUnless(cp.matches(d))

green

It just works. Why?

There's no YearPattern in the composite requiring the passed-in date to match any specific year. Wild cards now work by not specifying any pattern for a given component. Remember when I thought I might need a class to do the wild card matching? I was wrong!

Cleaning Up

At this point, I feel really good about the new approach and will just delete the old tests and code.

I'll also refactor the tests a bit. Did you notice that every one of the new tests contained a duplicate line? I did. It started to bother me, but that's what test fixtures are for:

class PatternTests(unittest.TestCase):
    def setUp(self):
        self.d = datetime.date(2004, 9, 29)

    def testYearMatches(self):
        yp = YearPattern(2004)
        self.failUnless(yp.matches(self.d))

    def testYearDoesNotMatch(self):
        yp = YearPattern(2003)
        self.failIf(yp.matches(self.d))

green

I've only shown the first two test cases (in the fixture previously known as NewTests) but now all of the test cases refer to the date as self.d instead of constructing a local date instance. It's not a huge refactoring, but it makes me feel better. You do want me to feel the best I can about my code, don't you? Of course you do.

I did have to change testCompositeWithoutYearMatches to use this date instead of my anniversary. As cute as it was to throw that date in there, I decided I'd rather have clean code without duplication than cuteness.

I also took this opportunity to add some named constants for weekdays:

MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY = range(0, 7)

Now I can use these instead of the hard-coded constants I had in the weekday tests, and also delete the comments I had explaining what the constants represented. Intention-revealing code beats comments any day of the week.

Where am I now? Before switching gears, I planned to write a test for a pattern that matched the last Thursday of every month. It's time to do that now:

def testLastThursdayMatches(self):
    cp = CompositePattern()
    cp.add(LastWeekdayPattern(THURSDAY))
    self.failUnless(cp.matches(self.d))

red

Cool. A new class to implement!

The implementation for this class is slightly more complicated than the others:

class LastWeekdayPattern:
    def __init__(self, weekday):
        self.weekday = weekday

    def matches(self, date):
        nextWeek = date + datetime.timedelta(7)
        return self.weekday == date.weekday() and nextWeek.month != date.month

red

Oops. It doesn't pass. Why?

The date I'm trying to match in the test is a Wednesday, not a Thursday! I need to fix the test (not forgetting to rename it) to add a test where I expect the match to fail (which I should have done before implementing matches):

def testLastWednesdayMatches(self):
    cp = CompositePattern()
    cp.add(LastWeekdayPattern(WEDNESDAY))
    self.failUnless(cp.matches(self.d))

def testLastWednesdayDoesNotMatch(self):
    cp = CompositePattern()
    cp.add(LastWeekdayPattern(WEDNESDAY))
    self.failIf(cp.matches(self.d))

red

Rats. The first test passes but the second one fails. The date created in setUp is the same for every test case so it will always be a Wednesday, but I need a date that's not on a Wednesday to make this test pass. Rather than creating a new date in this test case (and ignoring the one created in setUp), I'll move both of these tests into a new fixture--one specific for testing the LastWeekdayPattern class:

class LastWeekdayPatternTests(unittest.TestCase):
    def setUp(self):
        self.pattern = LastWeekdayPattern(WEDNESDAY)

    def testLastWednesdayMatches(self):
        lastWedOfSep2004 = datetime.date(2004, 9, 29)
        self.failUnless(self.pattern.matches(lastWedOfSep2004))

    def testLastWednesdayDoesNotMatch(self):
        firstWedOfSep2004 = datetime.date(2004, 9, 1)
        self.failIf(self.pattern.matches(firstWedOfSep2004))

green

The Open-Closed Principle

Isn't it nice being able to add new functionality without changing existing classes? This is part of what Bertrand Meyer called the Open-Closed Principle: "Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification." With the new approach of using the Composite pattern, I can extend the behavior of the system by writing a new class with a match method and passing an instance of that class into add. This is one of the most fundamental principles of object-oriented design that, too often, gets lost in shuffle.

Now they both pass and the tests use everything they create. Nice.

While moving these tests into a new fixture, I also noticed that I created a CompositePattern that contained only one pattern. That's kind of pointless, so I stopped doing it.

Should I move the test cases that exercise the various pattern classes into their own fixtures? That's a tendency that many, including me, often have. It's sometimes useful to resist that urge, though. As Dave Astels, author of Test-Driven Development: A Practical Guide, puts it: a fixture is "a way to group tests that need to be set up in exactly the same way." In other words, a fixture is not a container for all of the tests for a single class, or at least, it doesn't have to be.

Having said that, I prefer it when all of the test cases in a fixture exercise the same class. In harmony with Dave's definition of fixtures, I just don't require that all of the test cases that exercise the same class be in the same fixture. Make sense?

Suppose that I have four test cases for the class Foo but half require different setUp code than the other half. I'd split those cases up into two fixtures, even though they both exercise the same class. If I then started writing tests for the class Bar and discovered that some of its tests could use the same setUp code as one of the class Foo fixtures, I would not just shove those tests into one of the existing fixtures.

Wouldn't that mean I've duplicated the setUp code for the two fixtures? Duplication is evil! If I thought the duplication was enough to be a problem, I would Extract Superclass the duplicated code out of the two fixtures. Yes, you can--and should--refactor your tests, too.

When starting on a new project, I create one fixture with no setUp method and add all of my test cases to that one fixture. Eventually, I reach the point where I need to refactor the fixture, and I do it. Remember: do the simplest thing that could possibly work first. Then refactor if necessary.

What, though, is the benefit of ensuring that all of the test cases in a fixture only exercise one class? Well, besides making it more cohesive (and obeying the Single Responsibility Principle (PDF)), think about what might happen when you decide a class is no longer necessary. You'll need to delete the tests for that class, too. It's a lot easier to delete a whole test fixture than to look at each test case in a fixture to see if it exercises the class you just deleted.

Think you won't delete classes? Think again. You saw me delete the DatePattern class and all of its tests earlier, didn't you? It wasn't hard. I felt good about it, too.

More Patterns

Because it's so easy and fun, I want to add another pattern:

class NthWeekdayPatternTests(unittest.TestCase):
    def setUp(self):
        self.pattern = NthWeekdayPattern(1, WEDNESDAY)

    def testMatches(self):
        firstWedOfSep2004 = datetime.date(2004, 9, 1)
        self.failUnless(self.pattern.matches(firstWedOfSep2004))

    def testNotMatches(self):
        secondWedOfSep2004 = datetime.date(2004, 9, 8)
        self.failIf(self.pattern.matches(secondWedOfSep2004))

red

I don't have an example of this in my use cases as listed at the beginning of this article, but it's a feature that both calendar and pal support, so I expected to add it at some point.

Making these tests pass shouldn't be too hard:

class NthWeekdayPattern:
    def __init__(self, n, weekday):
        self.n = n
        self.weekday = weekday

    def matches(self, date):
        if self.weekday != date.weekday():
            return False
        n = 1
        while True:
            previousDate = date - datetime.timedelta(7 * n)
            if previousDate.month == date.month:
                n += 1
            else:
                break
        return self.n == n

green

OK, it was harder than I thought. I'm not a huge fan of the way that algorithm looks, but it's OK for now. I really should at least extract it into its own method so I can give it an intention-revealing name:

def matches(self, date):
    if self.weekday != date.weekday():
        return False
    return self.n == self.getWeekdayNumber(date)

def getWeekdayNumber(self, date):
    n = 1
    while True:
        previousDate = date - datetime.timedelta(7 * n)
        if previousDate.month == date.month:
            n += 1
        else:
            break
    return n

green

Better.

The last example I do have at the beginning of this article is the "last day of the month" case. That should match days "in reverse." I could modify the existing DayPattern class, but instead I want to add a new pattern class:

class LastDayInMonthPatternTests(unittest.TestCase):
    def testMatches(self):
        lastDayInSep2004 = datetime.date(2004, 9, 30)
        pattern = LastDayInMonthPattern()
        self.failUnless(pattern.matches(lastDayInSep2004))

red

While typing in that test, I decided that I couldn't think of a reason to support the second-to-last day in a month, or the third-to-last day, and so on. Can you? I made it easy on myself and decided to implement a class called LastDayInMonthPattern. People usually argue that writing tests up front takes too much work, but writing a test first this time actually saved me from writing code I would never use!

The implementation of this new pattern is:

class LastDayInMonthPattern:
    def matches(self, date):
        tomorrow = date + datetime.timedelta(1)
        return tomorrow.month != date.month

green

Nice.

Although I just realized I'm cheating again. Here's how the fixture should have looked (after extracting out the setUp method) before fully implementing the matches method:

class LastDayInMonthPatternTests(unittest.TestCase):
    def setUp(self):
        self.pattern = LastDayInMonthPattern()

    def testMatches(self):
        lastDayInSep2004 = datetime.date(2004, 9, 30)
        self.failUnless(self.pattern.matches(lastDayInSep2004))

    def testNotMatches(self):
        secondToLastDayInSep2004 = datetime.date(2004, 9, 29)
        self.failIf(self.pattern.matches(secondToLastDayInSep2004))

green

I feel pretty good about myself right now. That usually means it's time to refactor. Now I really want to do some renaming.

For the last pattern I implemented, I was very explicit about what it did: LastDayInMonthPattern only matches the last day in a month and there's no further clarification needed. What about NthWeekdayPattern and LastWeekdayPattern? I really want to add InMonth to the end of both of those class names. Yes, I'm that picky.

Related Reading

Learning Python
By Mark Lutz, David Ascher

I also took this time to rename a few of the test cases and reorder some of the class definitions. I won't bore you with the details, but you can see the final results for yourself if you download the code at the end of the article.

This type of tidying up may seem trivial but it's extremely important. If your code doesn't look clean, you (and others who find themselves working on your code) won't have any incentive to keep it clean. The Pragmatic Programmers call this the Broken Window Theory. If you live with broken windows, don't be surprised when your neighbors start using your lawn as a junk yard.

Conclusion

I have about 60 non-blank lines of code so far, spread across eight classes. That's not too much, but the code is very simple and yet highly flexible. I seriously doubt I would have been able to conceive of this design without writing my tests first.

What's even cooler is that I have about 90 non-blank lines of test code. Yes, I have more test code than I have "real" code. Is that wrong? Absolutely not. That's wonderful! I feel extremely confident about the quality of the code that I have so far. Is it perfect? I doubt it. When I discover a bug, though, I can add a new test to demonstrate it and fix it so that it never happens again. If I need to perform an optimization, I'll have a suite of tests I can use to verify that I didn't screw anything up while applying the optimization.

What's also interesting to note is that the design that emerged from this work. I spent zero time in front of a modeling tool trying to create a design that would both meet my needs today and still be elegant enough to (hopefully) meet all of tomorrow's needs, as well. I didn't intend for this to happen. It just magically happened that way. This isn't rare--this almost always happens when I do test-driven development.

How is this design more flexible than I originally intended? Suppose that I want to create a pattern that matches every Friday the 13th. That wasn't one of my original use cases, and I gave no thought to it while writing the tests. The classes I came up have no trouble representing that pattern:

>>> import DatePatterns
>>> fri13 = DatePatterns.CompositePattern()
>>> fri13.add(DatePatterns.WeekdayPattern(DatePatterns.FRIDAY))
>>> fri13.add(DatePatterns.DayPattern(13))
>>> import datetime
>>> aug13 = datetime.date(2004, 8, 13)
>>> aug13.strftime('%A')
'Friday'
>>> fri13.matches(aug13)
True
>>> sep13 = datetime.date(2004, 9, 13)
>>> sep13.strftime('%A')
'Monday'
>>> fri13.matches(sep13)
False

While I'm not done with the application yet, I do have a solid foundation to build on. Next, I need to add some parsing code so that I can read a file containing events in order to construct and use the patterns I implemented above. I'll visit that task next time.

The code and tests are available for download and inspection.

Jason Diamond is a consultant specializing in C++, C#, and XML, and is located in sunny Southern California.


Return to the Python DevCenter.

Copyright © 2009 O'Reilly Media, Inc.