Search This Blog

Labels

Friday, December 31, 2010

Metaclass programming in Python——Understanding the arcana of inheritance and instance creation

Michele Simionato (mis6+@pitt.edu), Physicist, University of Pittsburgh

David Mertz (mertz@gnosis.cx), Developer, Gnosis Software, Inc.

Summary:  Michele and David's initial developerWorks article on metaclass programming prompted quite a bit of feedback, some of it from perplexed readers trying to grasp the subtleties of Python metaclasses. This article revisits the working of metaclasses and their relation to other OOP concepts. It contrasts class instantiation with inheritance, distinguishes classmethods and metamethods, and explains and solves metaclass conflicts.

Metaclasses and their discontents

In our initial article on metaclass programming in Python, we introduced the concept of metaclasses, showed some of their power, and demonstrated their use in solving problems such as dynamic customization of classes and libraries at run-time.

That article proved quite popular, but there were elisions in our condensed summary. Certain details in the use of metaclasses merit further explanation. Based on the feedback of our readers and on discussions in comp.lang.python, we will address some of those trickier point in this second article. In particular, we think the following points are important for any programmer wanting to master metaclasses:

  • Users must understand the differences and interactions between metaclass programming and traditional object-oriented programming (under both single and multiple inheritance).
  • Python 2.2 added the built-in functions staticmethod() andclassmethod() to create methods that do not require an instance object during invocation. To an extent, classmethodsoverlap in purpose with (meta)methods defined in metaclasses. But the precise similarities and differences have also generated confusion in the minds of many programmers.
  • Users should understand the cause and the resolution of metatype conflicts. This becomes essential when you want to use more than one custom metaclass. We explain the concept of composition of metaclasses.

Instantiation versus inheritance

Many programmers are confused about the difference between a metaclass and a base class. At the superficial level of "determining" a class, both look similar. But once you look deeper, the concepts drift apart.

Before presenting some examples, it is worth being precise about some nomenclature. An instance is a Python object that was "manufactured" by a class; the class acts as a sort of template for the instance. Every instance is an instance of exactly oneclass (but a class might have multiple instances). What we often call an instance object -- or perhaps a "simple instance" -- is "final" in the sense that it cannot act as a template for other objects (but it might still be a factory or a delegate, which serve overlapping purposes).

Some instance objects are themselves classes; and all classes are instances of a corresponding metaclass. Even classes only come into existence through the instantiation mechanism. Usually classes are instances of the built-in, standard metaclass type; it is only when we specify metaclasses other than type that we need to think about metaclass programming. We also call the class used to instantiate an object the type of that object.

Running orthogonal to the idea of instantiation is the notion of inheritance. Here, a class can have one or multiple parents, not just one unique type. And parents can have parents, creating a transitive subclass relation, conveniently accessible with the built-in function issubclass(). For example, if we define a few classes and an instance:

Listing 1. Typical inheritance hierarchy 

>>> class A(object): a1 = "A"
...
>>> class B(object): a2 = "B"
...
>>> class C(A,B):    a3 = "C(A,B)"
...
>>> class D(C):      a4 = "D(C)"
...
>>> d = D()
>>> d.a5 = "instance d of D"

Then we can test the relations:

Listing 2. Testing ancestry

>>> issubclass(D,C)
True
>>> issubclass(D,A)
True
>>> issubclass(A,B)
False
>>> issubclass(d,D)
[...]
TypeError: issubclass() arg 1 must be a class

The interesting question now -- the one necessary for understanding the contrast between superclasses and metaclasses -- is how an attribute like d.attr is resolved. For simplicity, we discuss only the standard look-rule, not the fallback to.__getattr__(). The first step in such resolution is to look in d.__dict__ for the name attr. If found, that's that; but if not, something fancy needs to happen, such as:

>>> d.__dict__, d.a5, d.a1
({'a5': 'instance d'}, 'instance d', 'A')


The trick to finding an attribute that isn't attached to an instance is to look for it in the class of the instance, then after that in all the superclasses. The order in which superclasses are checked is called the method resolution order for the class. You can look at it with the (meta)method .mro() (but only from class objects):

>>> [k.__name__ for k in d.__class__.mro()]
['D', 'C', 'A', 'B', 'object']


In other words, the access to d.attr first looks in d.__dict__, then in D.__dict__, C.__dict__, A.__dict__,B.__dict__, and finally in object.__dict__. If the name is not found in any of those places, an AttributeError is raised.

Notice that metaclasses were never mentioned in the lookup procedure.

Metaclasses versus ancestors


Here is a simple example of normal inheritance. We define a Noble base class, with subclasses such as Prince, Duke, Baron, etc.

Listing 3. Attribute inheritance

>>> for s in "Power Wealth Beauty".split(): exec '%s="%s"'%(s,s)
...
>>> class Noble(object):      # ...in fairy tale world
...     attributes = Power, Wealth, Beauty
...
>>> class Prince(Noble):
...     pass
...
>>> Prince.attributes
('Power', 'Wealth', 'Beauty')

The class Prince inherits the attributes of the class Noble. An instance of Prince still follows the lookup chain discussed above:

Listing 4. Attributes in instances

>>> charles=Prince()
>>> charles.attributes        # ...remember, not the real world
('Power', 'Wealth', 'Beauty')

If the Duke class should happen to have a custom metaclasses, it can obtain some attributes that way:

>>> class Nobility(type): attributes = Power, Wealth, Beauty
...
>>> class Duke(object): __metaclass__ = Nobility
...


As well as being a class, Duke is an instance of the metaclass Nobility--attribute lookup proceeds as with any object:

>>> Duke.attributes
('Power', 'Wealth', 'Beauty')


But Nobility is not a superclass of Duke, so there is no reason why an instance of Duke would findNobility.attributes:

Listing 5. Attributes and metaclasses

>>> Duke.mro()
[<class '__main__.Duke'>, <type 'object'>]
>>> earl = Duke()
>>> earl.attributes
[...]
AttributeError: 'Duke' object has no attribute 'attributes'

The availability of metaclass attributes is not transitive; in other words, the attributes of a metaclass are available to its instances, but not to the instances of the instances. Just this is the main difference between metaclasses and superclasses. A diagram emphasizes the orthogonality of inheritance and instantiation:

Figure 1. Instantiation versus inheritance
Figure 1. Instantiation versus inheritance


Since earl still has a class, you can indirectly retrieve the attributes, however:

>>> earl.__class__.attributes

Figure 1 contrasts simple cases where either inheritance or metaclasses are involved, but not both. Sometimes, however, a class C has both a custom metaclass M and a base class B:

Listing 6. Combining metaclasses and superclasses

                >>> class M(type):
...     a = 'M.a'
...     x = 'M.x'
...
>>> class B(object): a = 'B.a'
...
>>> class C(B): __metaclass__=M
...
>>> c=C()

Graphically:

Figure 2. Combined superclass and metaclass

Figure 2. Combined superclass and metaclass


From the prior explanation, we could imagine that C.a would resolve to either M.a or B.a. As it turns out, lookup on a class follows its MRO before it looks in its instantiating metaclass:

Listing 7. Resolution with metaclasses and superclasses

>>> C.a, C.x
('B.a', 'M.x')
>>> c.a
'B.a'
>>> c.x
[...]
AttributeError: 'C' object has no attribute 'x'

You can still enforce a attribute value using a metaclass, you just need to set it on the class object being instantiated rather than as an attribute of the metaclass:

Listing 8. Setting attribute in metaclass

>>> class M(type):
...     def __init__(cls, *args):
...         cls.a = 'M.a'
...
>>> class C(B): __metaclass__=M
...
>>> C.a, C().a
('M.a', 'M.a')

More on class magic


The fact that the instantiation constraint is weaker than the inheritance constraint is essential for implementing the special methods like .__new__(), .__init__(), .__str__(), etc. We will discuss the .__str__() method; an analysis is similar for the other special methods.

Readers probably know that the printed representation of a class object can be modified by overriding its .__str__() method. In the same sense, the printed representation of a class can be modified by overriding the .__str__() methods of its metaclass. For instance:

Listing 9. Customizing printout of a class

>>> class Printable(type):
...    def __str__(cls):
...        return "This is class %s" % cls.__name__
...
>>> class C(object): __metaclass__ = Printable
...
>>> print C       # equivalent to print Printable.__str__(C)
This is class C
>>> c = C()
>>> print c       # equivalent to print C.__str__(c)
<C object at 0x40380a6c>

The situation can be represented with the following diagram:

Figure 3. Metaclasses and magic methods

Figure 3. Metaclasses and magic methods


From the previous discussion, it is clear that the .__str__() method in Printable cannot override the .__str__() method in C, which is inherited from object and therefore has precedence; printing c still gives the standard result.

If C inherited its .__str__() method from Printable rather than from object, it would cause a problem: C instances do not have a .__name__ attribute and printing c would generate an error. Of course, you could still define a .__str__() method in Cthat would change the way c prints.

Classmethods versus metamethods


Another common confusion arises between Python classmethods and methods defined in a metaclass, best called metamethods.

Consider this example:

Listing 10. Metamethods and classmethods

>>> class M(Printable):
...     def mm(cls):
...         return "I am a metamethod of %s" % cls.__name__
...
>>> class C(object):
...     __metaclass__=M
...     def cm(cls):
...         return "I am a classmethod of %s" % cls.__name__
...     cm=classmethod(cm)
...
>>> c=C()

Part of the confusion is due to the fact that in Smalltalk terminology, C.mm would be called a "class method of C." Python classmethods are a different beast, however.

The metamethod "mm" can be invoked from either the metaclass or from the class, but not from the instance. The classmethod can be called both from the class and from its instances (but does not exist in the metaclass).

Listing 11. Invoking a metamethod

>>> print M.mm(C)
I am a metamethod of C
>>> print C.mm()
I am a metamethod of C
>>> print c.mm()
[...]
AttributeError: 'C' object has no attribute 'mm'
>>> print C.cm()
I am a classmethod of C
>>> print c.cm()
I am a classmethod of C

Also, the metamethod is retrieved by dir(M) but not by dir(C) whereas the classmethod is retrieved by dir(C) and dir(c).

You can only call the metaclass methods that are defined in the class MRO by dispatching on the metaclass (built-ins like printdo this behind the scenes):

Listing 12. Magic metaclass method

>>> print C.__str__()
[...]
TypeError: descriptor '__str__' of 'object' object needs an argument
>>> print M.__str__(C)
This is class C

It is important to notice that this dispatch conflict is not limited to magic methods. If we change C by adding an attribute C.mm, the same issue exists (it does not matter if the name is a regular method, classmethod, staticmethod, or simple attribute):

Listing 13. Non-magic metaclass method

>>> C.mm=lambda self: "I am a regular method of %s" % self.__class__
>>> print C.mm()
[...]
TypeError: unbound method <lambda>() must be called with
    C instance as first argument (got nothing instead)

Conflicting metaclasses


Once you work seriously with metaclasses, you will be bitten at least once by metaclass/metatype conflicts. Consider a class Awith metaclass M_A and a class B with metaclass M_B; suppose we derive C from A and B. The question is: what is the metaclass of C? Is it M_A or M_B?

The correct answer is M_C, where M_C is a metaclass that inherits from M_A and M_B, as in the following graph (in the Resourceslater in this article, see the link to Putting metaclasses to work for a discussion):

Figure 4. Avoiding the metaclass conflict

Figure 4. Avoiding the metaclass conflict


However, Python does not (yet) automatically create M_C. Instead, it raises a TypeError, warning the programmer of the conflict:

Listing 14. Metaclass conflicts

>>> class M_A(type): pass
...
>>> class M_B(type): pass
...
>>> class A(object): __metaclass__ = M_A
...
>>> class B(object): __metaclass__ = M_B
...
>>> class C(A,B): pass    # Error message less specific under 2.2
[...]
TypeError: metaclass conflict: the metaclass of a derived class must
    be a (non-strict) subclass of the metaclasses of all its bases

The metatype conflict can be avoided by manually creating the needed metaclass for C:

Listing 15. Manually resolving metaclass conflict

>>> M_AM_B = type("M_AM_B", (M_A,M_B), {})
>>> class C(A,B): __metaclass__ = M_AM_B
...
>>> type(C)
<class 'M_AM_B'>

The resolution of metatype conflicts becomes more complicated when you wish to "inject" additional metaclasses into a class, beyond those in its ancestors. Also, depending on the metaclasses of parent classes, redundant metaclasses can occur -- both identical metaclasses in different ancestors and superclass/subclass relationships among metaclasses. The module noconflictis available to help users resolve these issues in a robust and automatic way (see Resources).

Conclusion


A number of warnings and corner cases are discussed in this article. Working with metaclasses requires a certain degree of trial-and-error before the behavior becomes intuitive. However, the issues are by no means intractable -- this fairly short article touches on most of the pitfalls. Play with the cases yourself. You will find, at the end of the day, that whole new realms of program generalization are available with metaclasses; the gains are well worth the few dangers.

Resources


Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)