A Primer on Python Metaclass Programming

This series of articles by David Mertz assumes readers have a familiarity with basic object-oriented programming concepts: inheritance, encapsulation, and polymorphism. We pick up where the basics leave off; how some "exotic" techniques can make applied programming tasks easier, more maintainable, and just plain more elegant. This series will primarily present examples in Python, but the concepts apply to other programming languages too.

This first installment examines metaclasses. Just as ordinary instances (objects) are built out of classes, classes themselves are built out of metaclasses. Most of the time, you do not need to worry about the implicit metaclass (named type in Python) that is used in class construction, but occasionally, deliberate control of a class' metaclass can produce powerful effects. For example, metaclasses allow "aspect oriented programming," meaning you can enhance classes with features like tracing capabilities, object persistence, exception logging, and more.

An Object-Oriented Programming Review

Related Reading Python in a Nutshell By Alex Martelli Table of Contents Index Sample Chapter Read Online--Safari Search this book on Safari:   Only This Book All of Safari Code Fragments only

In general principles, OOP works the same way across many programming languages, modulo minor syntax differences. In an object-oriented programming language, you can define classes that bundle together related data and behavior. These classes can inherit some or all of their qualities from their parents, but can also define attributes (data) or methods (behavior) of their own. Classes generally act as templates for the creation of instances (or simply objects). Different instances of the same class typically have different data but share the same "shape"--e.g., the Employee objects bob and jane both have a .salary and a .room_number, but not the same room and salary as each other.

A Metaprogramming Rejoinder

In Python (and other languages), classes are themselves objects that can be passed around and introspected. Just as regular classes act as templates for producing instances, metaclasses act as templates for producing classes.

Python has always had metaclasses. The metaclass machinery became exposed much better with Python 2.2. Specifically, with version 2.2, Python stopped being a language with just one special (mostly hidden) metaclass that created every class object. Now, programmers can subclass the built-in metaclass type and even dynamically generate classes with varying metaclasses.

You do not need to use custom metaclasses to manipulate the production of classes, however. A slightly less brain-melting concept is a class factory. An ordinary function can return a class that is dynamically created within the function body. In traditional Python syntax, you can write the following.

Example 1. An old-fashioned Python 1.5.2 class factory

Python 1.5.2 (#0, Jun 27 1999, 11:23:01) [...]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> def class_with_method(func):
...     class klass: pass
...     setattr(klass, func.__name__, func)
...     return klass
...
>>> def say_foo(self): print 'foo'
...
>>> Foo = class_with_method(say_foo)
>>> foo = Foo()
>>> foo.say_foo()
foo

The factory function class_with_method() dynamically creates and returns a class that contains the method/function passed into the factory. The class itself is manipulated within the function body before being returned. The new module provides a more concise spelling, but without the same options for custom code within the body of the class factory:

Example 2. A class factory in the new module

>>> from new import classobj
>>> Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>> Foo2().bar()
'bar'
>>> Foo2().say_foo()
foo

In all of these cases, the behaviors of the class (Foo, Foo2) are not directly written as code, but are instead created by calling functions at runtime, with dynamic arguments. Not only are the instances dynamically created, so are the classes themselves.

From Class Factories to Metaclasses

Methods (i.e., of classes), like plain functions, can return objects. In that sense, it is obvious that class factories can be classes just as easily as they can be functions. In particular, Python 2.2+ provides a special class called type that is just such a class factory. The new class type is backwards-compatible with the older function of the same name, by the way. The class type works as a class factory in the same way that the function new.classobj does:

Example 3. type as a class factory metaclass

>>> X = type('X',(),{'foo':lambda self:'foo'})
>>> X, X().foo()
(<class '__main__.X'>, 'foo')

Since type is now a (meta)class, you are free to subclass it:

Example 4. A type descendent as class factory

>>> class ChattyType(type):
...     def __new__(cls, name, bases, dct):
...         print "Allocating memory for class", name
...         return type.__new__(cls, name, bases, dct)
...     def __init__(cls, name, bases, dct):
...         print "Init'ing (configuring) class", name
...         super(ChattyType, cls).__init__(name, bases, dct)
...
>>> X = ChattyType('X',(),{'foo':lambda self:'foo'})
Allocating memory for class X
Init'ing (configuring) class X
>>> X, X().foo()
(<class '__main__.X'>, 'foo')

The magic methods .__new__() and .__init__() are special, but in conceptually the same way that they are for any other class. The .__init__() method lets you configure the created object and the .__new__() method lets you customize its allocation. The latter is not widely overridden, but does exist for every Python 2.2 new-style class.

There is one feature of type descendents to be careful about; it catches everyone who first plays with metaclasses. The first argument to methods is conventionally called cls rather than self, because the methods operate on the produced class, not the metaclass. Actually, there is nothing special about this. All methods attach to their instances, and the instance of a metaclass is a class. A better name makes this more obvious:

Example 5. Attaching class methods to produced classes

>>> class Printable(type):
...     def whoami(cls): print "I am a", cls.__name__
...
>>> Foo = Printable('Foo',(),{})
>>> Foo.whoami()
I am a Foo
>>> Printable.whoami()
Traceback (most recent call last):
TypeError:  unbound method whoami() [...]

All of this surprisingly non-remarkable machinery comes with some syntax sugar that both makes working with metaclasses easier and confuses new users. There are several elements to the extra syntax. The resolution order of these new variations is tricky though. Classes can inherit metaclasses from their ancestors--notice that this is not the same thing as having metaclasses as ancestors (another common confusion). For old-style classes, defining a global __metaclass__ variable can force a custom metaclass to be used. Most of the time, the safest approach is to set a __metaclass__ class attribute for a class that wants to be created via a custom metaclass. You must set the variable in the class definition itself, since the metaclass is not used if the attribute is set later (after the class object has already been created). For example:

Example 6. Setting a metaclass with a class attribute

>>> class Bar:
...     __metaclass__ = Printable
...     def foomethod(self): print 'foo'
...
>>> Bar.whoami()
I am a Bar
>>> Bar().foomethod()
foo

Solving Problems with Magic

So far, we have seen the basics of metaclasses. Putting them to work is more subtle. The challenge of using metaclasses is that in typical OOP design, classes do not really do much. Class inheritance structures encapsulate and package data and methods, but one typically works with instances in the concrete.

There are two general categories of programming tasks where I think metaclasses are genuinely valuable.

The first, and probably more common, category is where you do not know at design time exactly what a class needs to do. Obviously, you will have some idea about it, but some particular detail might depend on information that will not be available until later. "Later" itself can be of two sorts: a), when a library module is used by an application, and b), at runtime when some situation exists. This category is close to what is often called "Aspect Oriented Programming" (AOP). Let me show an elegant example:

Example 7. Metaclass configuration at runtime

% cat dump.py
#!/usr/bin/python
import sys
if len(sys.argv) > 2:
    module, metaklass  = sys.argv[1:3]
    m = __import__(module, globals(), locals(), [metaklass])
    __metaclass__ = getattr(m, metaklass)

class Data:
    def __init__(self):
        self.num = 38
        self.lst = ['a','b','c']
        self.str = 'spam'
    dumps   = lambda self: `self`
    __str__ = lambda self: self.dumps()

data = Data()
print data

% dump.py
<__main__.Data instance at 1686a0> 

As you would expect, this application prints out a rather generic description of the data object (a conventional instance). We get a rather different result by passing runtime arguments to the application:

Example 8. Adding an external serialization metaclass

% dump.py gnosis.magic MetaXMLPickler
<?xml version="1.0"?>
<!DOCTYPE PyObject SYSTEM "PyObjects.dtd">
<PyObject module="__main__" class="Data" id="720748">
<attr name="lst" type="list" id="980012" >
  <item type="string" value="a" />
  <item type="string" value="b" />
  <item type="string" value="c" />
</attr>
<attr name="num" type="numeric" value="38" />
<attr name="str" type="string" value="spam" />
</PyObject>

The particular example uses the serialization style of gnosis.xml.pickle, but the most current gnosis.magic package also contains the metaclass serializers MetaYamlDump, MetaPyPickler, and MetaPrettyPrint. Moreover, a user of the dump.py "application" can impose the use of any "MetaPickler" she wishes, from any Python package that defines one. Writing an appropriate metaclass for this purpose will look something like this:

Example 9. Adding an attribute with a metaclass

class MetaPickler(type):
    "Metaclass for gnosis.xml.pickle serialization"
    def __init__(cls, name, bases, dict):
        from gnosis.xml.pickle import dumps
        super(MetaPickler, cls).__init__(name, bases, dict)
        setattr(cls, 'dumps', dumps)

The remarkable achievement of this arrangement is that the application programmer need have no knowledge about what serialization will be used--nor even whether serialization or some other cross-sectional capability will be added at the command line.

Perhaps the most common use of metaclasses is similar to that of MetaPicklers: adding, deleting, renaming, or substituting methods for those defined in the produced class. In our example, a "native" Data.dump() method is replaced by a different one from outside of the application, at the time the class Data is created (and therefore, in every subsequent instance).

Resources

A useful book on metaclasses is Putting Metaclasses to Work, by Ira R. Forman, Scott Danforth, Addison-Wesley 1999 (ISBN 0201433052).

For metaclasses in Python specifically, Guido van Rossum's essay, " Unifying types and classes in Python 2.2," is useful.

My Gnosis Utilities package contains functions to make working with metaclasses easier and more powerful.

David Mertz , being a sort of Foucauldian Berkeley, believes, esse est denunte.

Return to Python DevCenter.

Copyright © 2007 O'Reilly Media, Inc.