Calling super() inside a class' classmethod to get at the metaclass method

Question

Just when I though I understood metaclasses...

Disclaimer: I have looked around for an answer before posting, but most of the answers I have found are about calling super() to get at another @classmethod in the MRO (no metaclass involved) or, surprisingly, a lot of them were about trying to do something in metaclass.__new__ or metaclass.__call__ which meant the class wasn't fully created yet. I'm pretty sure (let's say 97%) that this is not one of those problems.

Environment: Python 3.7.2

The problem: I have a metaclass FooMeta that defines a method get_foo(cls), a class Foo that is built from that metaclass (so an instance of FooMeta) and has a @classmethod get_bar(cls). Then another class Foo2 that inherits from Foo. In Foo2, I subclass get_foo by declaring it a @classmethod and calling super(). This fails miserably...

i.e. with this code

class FooMeta(type):
    def get_foo(cls):
        return 5


class Foo(metaclass=FooMeta):
    @classmethod
    def get_bar(cls):
        return 3


print(Foo.get_foo)
# >>> <bound method FooMeta.get_foo of <class '__main__.Foo'>>
print(Foo.get_bar)
# >>> <bound method Foo.get_bar of <class '__main__.Foo'>>


class Foo2(Foo):
    @classmethod
    def get_foo(cls):
        print(cls.__mro__)
        # >>> (<class '__main__.Foo2'>, <class '__main__.Foo'>, <class 'object'>)
        return super().get_foo()

    @classmethod
    def get_bar(cls):
        return super().get_bar()


print(Foo2().get_bar())
# >>> 3
print(Foo2().get_foo())
# >>> AttributeError: 'super' object has no attribute 'get_foo'

The question: So with my class being an instance of the metaclass, and having verified that both class methods exist on the class Foo, why aren't both calls to the super().get_***() working inside Foo2? What am I not understanding about metaclasses or super() that's preventing me from finding these results logical?

EDIT: Further testing shows that the methods on Foo2 being class methods or instance methods doesn't change the result.

EDIT 2: Thanks to @chepner's answer, I think the problem was that super() was returning a super object representing Foo (this is verified with super().__thisclass__) and I was expecting super().get_foo() to behave (maybe even to call) get_attr(Foo, 'get_foo') behind the scene. It seems that it isn't... I'm still wondering why, but it is getting clearer :)

Answer 1

Foo may have a get_foo method, but super isn't designed to check what attributes a superclass has. super cares about what attributes originate in a superclass.

---

To understand super's design, consider the following multiple inheritance hierarchy:

class A:
    @classmethod
    def f(cls):
        return 1
class B(A):
    pass
class C(A):
    @classmethod
    def f(cls):
        return 2
class D(B, C):
    @classmethod
    def f(cls):
        return super().f() + 1

A, B, C, and D all have an f classmethod, but B's f is inherited from A. D's method resolution order, the sequence of classes checked for attribute lookup, goes (D, B, C, A, object).

super().f() + 1 searches the MRO of cls for an f implementation. The one it should find is C.f, but B has an inherited f implementation, and B is before C in the MRO. If super were to pick up B.f, this would break C's attempt to override f, in a situation commonly referred to as the "diamond problem".

Instead of looking at what attributes B has, super looks directly in B's __dict__, so it only considers attributes actually provided by B instead of by B's superclasses or metaclasses.

---

Now, back to your get_foo/get_bar situation. get_bar comes from Foo itself, so super().get_bar() finds Foo.get_bar. However, get_foo is provided not by Foo, but by the FooMeta metaclass, and there is no entry for get_foo in Foo.__dict__. Thus, super().get_foo() finds nothing.

Answer 2

get_foo is not an attribute of Foo, but rather an attribute of type(Foo):

>>> 'get_foo' in Foo.__dict__
False
>>> 'get_foo' in type(Foo).__dict__
True

So while Foo.get_foo will resolve to type(Foo).get_foo, super().get_foo does not because the proxy returned by super() is similar to Foo, but is not Foo itself.

Answer 3

Note 1

Just to be clear:

• Foo does not inherit from FooMeta.
• FooMeta is not a super-class of Foo

super will not work.

Note 2

Now that note (1) is out of the way, if you want to access a metaclass method from inside of a method of an instance of the metaclass, you can do it like this:

class FooMeta(type):
    _foo = 5

    def get_foo(cls):
        print("`get_foo` from `FooMeta` was called!")

    class Foo(metaclass=FooMeta):

        @classmethod
        def bar(Foo):
            FooMeta = type(Foo)
            FooMeta_dot_getfoo = FooMeta.get_foo
            FooMeta_dot_getfoo(Foo)

        def baz(self):
            Foo = type(self)
            FooMeta = type(Foo)
            FooMeta_dot_getfoo = FooMeta.get_foo
            FooMeta_dot_getfoo(Foo)

    Foo.bar()
    foo = Foo()
    foo.baz()

The output is:

`get_foo` from `FooMeta` was called!
`get_foo` from `FooMeta` was called!

Note 3

If you have a classmethod with the same name as a method in the metaclass, why does the metaclass method NOT get called? Consider the following code:

class FooMeta(type):
    def get_foo(cls):
        print("META!")

class Foo(metaclass=FooMeta):
    @classmethod
    def get_foo(cls):
        print("NOT META!")

Foo.get_foo()

The output is NOT META! In the following discussion, assume that:

• foo is instance of Foo
• Foo is instance of FooMeta

For the first time in this post, I will have pseudo code, not python. Don't try to run the following. __getattribute__ sorta looks like the following:

class FooMeta(type):
    def get_foo(Foo):
        print("META!")

class Foo(metaclass=FooMeta):
    @classmethod
    def get_foo(Foo):
        print("NOT META!")

    def __getattribute__(foo, the string "get_foo"):
        try:
            attribute = "get_foo" from instance foo
        except AttributeError:
            attribute = "get_foo" from class Foo

        # Begin code for handling "descriptors"
        if hasattr(attribute, '__get__'):
            attr = attribute.__get__(None, Foo)
        # End code for handling "descriptors"

        return attribute

foo = Foo()
foo.get_foo() # prints "NOT META!"

get_foo = Foo.__getattribute__(foo, "get_foo")
get_foo.__call__()

You can actually ignore the stuff which says, "code for handling "descriptors"." I only included that for completeness.

Note that nowhere does __getattribute__ say, "get get_foo from the meta class."

1. First, We try to get get_foo from the instance. Maybe get_foo is a member variable. Maybe one instance has get_foo = 1 and another instance has get_foo = 5 The computer does not know. The computer is stupid.
2. The computer realizes that the instance doesn't have a member variable named get_foo. It then says, "ah ha! I bet that get_foo belongs to the CLASS." So, it looks there, and lo-and-behold, there it is: Foo has an attribute named get_foo. FooMeta also has an attribute called get_foo, but who cares about that.

Something to focus on is that:

• Foo has an attribute named get_foo
• MetaFoo has an attribute named get_foo

They both have attributes named get_foo, but Foo and MetaFoo are different objects. It's not as if the two get_foos are shared. I can have obj1.x = 1 and obj2.x = 99. No problem.

FooMeta has its own __getattribute__ method. Before I talked about Foo.__getattribute__, but now let's talk about the MeTa __getattribute__

class FooMeta(type):
    def get_foo(Foo):
        print("META!")

    def __getattribute__(Foo, the string "get_foo"):
        try:                                            # LINE 1
            attribute = "get_foo" from class Foo        # LINE 2
        except AttributeError:                          # LINE 3
            attribute = "get_foo" from class FooMeta    # LINE 4
                                                        # LINE 5
        # Begin code for handling "descriptors"
        if hasattr(attribute, '__get__'):
            attr = attribute.__get__(None, Foo)
        # End code for handling "descriptors"

        return attribute

class Foo(metaclass=FooMeta):
    @classmethod
    def get_foo(Foo):
        print("NOT META!")

Foo.get_foo()

get_foo = FooMeta.__getattribute__(Foo, "get_foo")
get_foo.__call__() 

The order of events:

• Lines 1 and 2 happen
• Lines 3, 4, & 5 do not happen
• You can ignore the stuff about descriptors, because none of the different get_foos in this problem have a __get__ method

Okay now! Why only lines 1 & 2? Because you made a @classmethod silly! We check Foo to see if it has a get_foo and it does! Why check for class attributes if we find an instance attribute first? We always check to see if an attribute belong to the instance (Foo) first-and-foremost before checking if maybe there happens to be only one copy of a static member variable belonging to the class (FooMeta) and shared by all of the instances.

Note that if Foo does not have a get_foo then FooMeta.__getattribute__(Foo, "get_foo") will return get_foo from the metaclass because the first attempt (getting it from the instance) failed. You kinda blocked that option out by giving the instance something of the same name as the class's static member variable.

class K:

    im_supposed_to_be_shared = 1

    def __init__(self, x):
        # NOPE!
        self.im_supposed_to_be_shared = x
        # maybe try type(self)

obj1 = K(14)
obj2 = K(29)
print(obj1.im_supposed_to_be_shared)
print(obj2.im_supposed_to_be_shared)
print(K.im_supposed_to_be_shared)

Prints:

14
29
1

does NOT print:

29
29
29

Note that if you want to set a static class member variable, instance.mem_var = 5 is a very ⱽᵉᴿʸ bad idea. You will give the instance a new member variable, and the class static (shared) member variable will be shadowed. You can fix that with something like this:

def __setattr__(self, attr_name, attr_val):
    if hasattr(type(self), attr_name):
        setattr(type(self), attr_name, attr_val)
    else:
        super_class = inspect.getmro(type(self))[1]
        super_class.__setattr__(self, attr_name, attr_val)

Then your lil' compy will print:

29
29
29

Note 4

class Foo:
    @classmethod
    def funky(cls):
        pass

is NOT MetaClass.funky = funky. Instead, it's:

def funky(cls)
   pass
Funky = classmethod (funky)

... which is almost the same as:

def funky(cls):
     pass
funky = lambda self, *args, **kwargs: funky(type(self), *args, **kwargs)

The moral of note 4s story is that classmethod funky is an attribute of Foo and not an attribute of FooMeta

Answer 4

obj.attr calls type(obj).__getattribute__(obj, 'attr'), that is

• a) object.__getattribute__(obj, 'attr'), which looks up 'attr' in obj itself, as well as in the class of obj and its parents; or
• b) type.__getattribute__(obj, 'attr') if obj is a type instance, which looks up 'attr' in obj itself and its parents, as well as in the class of obj and its parents; or
• c) super.__getattribute__(obj, 'attr') if obj is a super instance,
  • c.1) which looks up 'attr' in the class of instance and its parents past cls, if obj is super(cls, instance), or
  • c.2) which looks up 'attr' in subclass itself and its parents past cls, if obj is super(cls, subclass).

When you call super().get_foo() in the class method Foo2.get_foo, which is equivalent to calling super(Foo2, cls).get_foo(), you are in case c.2), i.e. you are looking up 'get_foo' in cls itself and its parents past Foo2, i.e. you are looking up 'get_foo' in Foo. That is why the call fails.

You expect the call super().get_foo() in the class method Foo2.get_foo to succeed because you think it is equivalent to the call Foo.get_foo() which is in case b), i.e. you think you are looking up 'get_foo' in cls itself and its parents, as well as in the class of Foo (FooMeta) and its parents.