I am trying to figure out a way to gracefully deal with uncallable
classmethod objects. The class hierarchy below illustrates the issue. (Unfortunately, I haven't been able to come up with a shorter example.)


import datetime


class DUID(object):

_subclasses = {}

def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls._subclasses[cls.duid_type] = cls

def __init__(self, d):
for attr, factory in self._attrs.items():
setattr(self, attr, factory(d[attr]))

@classmethod
def from_dict(cls, d):
subcls = cls._subclasses[d['duid_type']]
return subcls(d)


class DuidLL(DUID):

@staticmethod
def _parse_l2addr(addr):
return bytes.fromhex(addr.replace(':', ''))

duid_type = 'DUID-LL'
_attrs = { 'layer2_addr': _parse_l2addr }


class DuidLLT(DuidLL):

@classmethod
def _parse_l2addr(cls, addr):
return super()._parse_l2addr(addr)

duid_type = 'DUID-LLT'
_attrs = {
'layer2_addr': _parse_l2addr,
'time': datetime.datetime.fromisoformat
}


A bit of context on why I want to do this ...

This is a simplified subset of a larger body of code that parses a
somewhat complex configuration. The configuration is a YAML document,
that pyyaml parses into a dictionary (which contains other dictionaries,
lists, etc., etc.). My code then parses that dictionary into an object
graph that represents the configuration.

Rather than embedding parsing logic into each of my object classes, I
have "lifted" it into the parent class (DUID in the example). A
subclasses need only provide a few attributes that identifies its
required and optional attributes, default values, etc. (simplified to DuidLL._attrs and DuidLLT._attrs in the example).

The parent class factory function (DUID.from_dict) uses the information
in the subclass's _attrs attribute to control how it parses the
configuration dictionary. Importantly, a subclass's _attrs attribute
maps attribute names to "factories" that are used to parse the values
into various types of objects.

Thus, DuidLL's 'layer2_addr' attribute is parsed with its
_parse_l2addr() static method, and DuidLLT's 'time' attribute is parsed
with datetime.datetime.fromisoformat(). A factory can be any callable
object that takes a dictionary as its only argument.

This works with static methods (as well as normal functions and object
types that have an appropriate constructor):

duid_ll = DUID.from_dict({ 'duid_type': 'DUID-LL', 'layer2_addr': 'de:ad:be:ef:00:00' })
type(duid_ll)

<class '__main__.DuidLL'>

duid_ll.duid_type

'DUID-LL'

duid_ll.layer2_addr

b'\xde\xad\xbe\xef\x00\x00'

It doesn't work with a class method, such as DuidLLT._parse_l2addr():

duid_llt = DUID.from_dict({ 'duid_type': 'DUID-LLT', 'layer2_addr': 'de:ad:be:ef:00:00', 'time': '2015-09-04T07:53:04-05:00' })

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pilcher/subservient/wtf/wtf.py", line 19, in from_dict
return subcls(d)
File "/home/pilcher/subservient/wtf/wtf.py", line 14, in __init__
setattr(self, attr, factory(d[attr]))
TypeError: 'classmethod' object is not callable

In searching, I've found a few articles that discuss the fact that
classmethod objects aren't callable, but the situation actually seems to
be more complicated.

type(DuidLLT._parse_l2addr)

<class 'method'>

callable(DuidLLT._parse_l2addr)

True

The method itself is callable, which makes sense. The factory function
doesn't access it directly, however, it gets it out of the _attrs
dictionary.

type(DuidLLT._attrs['layer2_addr'])

<class 'classmethod'>

callable(DuidLLT._attrs['layer2_addr'])

False

I'm not 100% sure, but I believe that this is happening because the
class (DuidLLT) doesn't exist at the time that its _attrs dictionary is defined. Thus, there is no class to which the method can be bound at
that time and the dictionary ends up containing the "unbound version."

Fortunately, I do know the class in the context from which I actually
need to call the method, so I am able to call it with its __func__
attribute. A modified version of DUID.__init__() appears to work:

def __init__(self, d):
for attr, factory in self._attrs.items():
if callable(factory): # <============= ???!
value = factory(d[attr])
else:
value = factory.__func__(type(self), d[attr])
setattr(self, attr, value)

A couple of questions (finally!):

* Is my analysis of why this is happening correct?

* Can I improve the 'if callable(factory):' test above? This treats
all non-callable objects as classmethods, which is obviously not
correct. Ideally, I would check specifically for a classmethod, but
there doesn't seem to be any literal against which I could check the
factory's type.

Note: I am aware that there are any number of workarounds for this
issue. I just want to make sure that I understand what is going on, and determine if there's a better way to test for a classmethod object.

Thanks!

--
========================================================================
Google Where SkyNet meets Idiocracy ========================================================================

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11Nov2022 15:29, Ian Pilcher <arequipeno@gmail.com> wrote:

I am trying to figure out a way to gracefully deal with uncallable >classmethod objects.

I'm just going to trim your example below a bit for reference purposes:

class DUID(object):
def __init__(self, d):
for attr, factory in self._attrs.items():
setattr(self, attr, factory(d[attr]))
@classmethod
def from_dict(cls, d):
subcls = cls._subclasses[d['duid_type']]
return subcls(d)

class DuidLL(DUID):
@staticmethod
def _parse_l2addr(addr):
return bytes.fromhex(addr.replace(':', ''))
_attrs = { 'layer2_addr': _parse_l2addr }

class DuidLLT(DuidLL):
@classmethod
def _parse_l2addr(cls, addr):
return super()._parse_l2addr(addr)
_attrs = {
'layer2_addr': _parse_l2addr,
}

So what you've got is that `for attr, factory in self._attrs.items():`
loop, where the factory comes from the subclass `_attrs` mapping. For
`DuidLL` you get the static method `_parse_l2addr` object and for
`DuidLLT` you get the class method object.

[...]

This works with static methods (as well as normal functions and object
types that have an appropriate constructor): [...]

[...]

It doesn't work with a class method, such as DuidLLT._parse_l2addr():

duid_llt = DUID.from_dict({ 'duid_type': 'DUID-LLT', 'layer2_addr': 'de:ad:be:ef:00:00', 'time': '2015-09-04T07:53:04-05:00' })

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pilcher/subservient/wtf/wtf.py", line 19, in from_dict
return subcls(d)
File "/home/pilcher/subservient/wtf/wtf.py", line 14, in __init__
setattr(self, attr, factory(d[attr]))
TypeError: 'classmethod' object is not callable

In searching, I've found a few articles that discuss the fact that >classmethod objects aren't callable, but the situation actually seems to
be more complicated.

type(DuidLLT._parse_l2addr)

<class 'method'>

callable(DuidLLT._parse_l2addr)

True

The method itself is callable, which makes sense. The factory function >doesn't access it directly, however, it gets it out of the _attrs
dictionary.

type(DuidLLT._attrs['layer2_addr'])

<class 'classmethod'>

callable(DuidLLT._attrs['layer2_addr'])

False

I'm not 100% sure, but I believe that this is happening because the
class (DuidLLT) doesn't exist at the time that its _attrs dictionary is >defined. Thus, there is no class to which the method can be bound at
that time and the dictionary ends up containing the "unbound version."

Yes. When you define the dictionary `_parse_l2addr` is an unbound class
method object. That doesn't change.

Fortunately, I do know the class in the context from which I actually
need to call the method, so I am able to call it with its __func__
attribute. A modified version of DUID.__init__() appears to work:

def __init__(self, d):
for attr, factory in self._attrs.items():
if callable(factory): # <============= ???!
value = factory(d[attr])
else:
value = factory.__func__(type(self), d[attr])
setattr(self, attr, value)

Neat!

A couple of questions (finally!):
* Is my analysis of why this is happening correct?

It seems so to me. Although I only learned some of these nuances
recently.

* Can I improve the 'if callable(factory):' test above? This treats
all non-callable objects as classmethods, which is obviously not
correct. Ideally, I would check specifically for a classmethod, but
there doesn't seem to be any literal against which I could check the
factory's type.

Yeah, it does feel a bit touchy feely.

You could see if the `inspect` module tells you more precise things
about the `factory`.

The other suggestion I have is to put the method name in `_attrs`; if
that's a `str` you could special case it as a well known type for the
factory and look it up with `getattr(cls,factory)`.

Cheers,
Cameron Simpson <cs@cskk.id.au>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Ian Pilcher wrote at 2022-11-11 15:29 -0600:

...
In searching, I've found a few articles that discuss the fact that >classmethod objects aren't callable, but the situation actually seems to
be more complicated.

type(DuidLLT._parse_l2addr)

<class 'method'>

callable(DuidLLT._parse_l2addr)

True

The method itself is callable, which makes sense. The factory function >doesn't access it directly, however, it gets it out of the _attrs
dictionary.

type(DuidLLT._attrs['layer2_addr'])

<class 'classmethod'>

callable(DuidLLT._attrs['layer2_addr'])

False

Accessing an object via a `dict` does not change its type,
nor does putting it into a `dict`.
Thus, you did not put `DuidLLT._parse_l2addr` (of type `method`)
into your `_attrs` `dict` but something else (of type `classmethod`).

This narrows down the space for your investigation: why was
the object you have put into `_attr` was not what you have expected.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/11/22 16:47, Cameron Simpson wrote:

On 11Nov2022 15:29, Ian Pilcher <arequipeno@gmail.com> wrote:

* Can I improve the 'if callable(factory):' test above?  This treats
 all non-callable objects as classmethods, which is obviously not
 correct.  Ideally, I would check specifically for a classmethod, but
 there doesn't seem to be any literal against which I could check the
 factory's type.

Yeah, it does feel a bit touchy feely.

You could see if the `inspect` module tells you more precise things
about the `factory`.

The other suggestion I have is to put the method name in `_attrs`; if
that's a `str` you could special case it as a well known type for the
factory and look it up with `getattr(cls,factory)`.

So I've done this.

class _HasUnboundClassMethod(object):
@classmethod
def _classmethod(cls):
pass # pragma: no cover
_methods = [ _classmethod ]

_ClassMethodType = type(_HasUnboundClassMethod._methods[0])

Which allows me to do this:

def __init__(self, d):
for attr, factory in self._attrs.items():
if callable(factory):
value = factory(d[attr])
else:
assert type(factory) is self._ClassMethodType
value = factory.__func__(type(self), d[attr])
setattr(self, attr, value)

It's a bit cleaner, although I'm not thrilled about having a throwaway
class, just to define a literal that ought to be provided by the
runtime.

--
========================================================================
Google Where SkyNet meets Idiocracy ========================================================================

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Use the inspect module as Cameron suggested.


import inspect


def analyze_class(clzz):
"""Analyze a class proof of concept"""
assert inspect.isclass(clzz)
for k, v in inspect.getmembers(clzz, lambda v: inspect.isfunction(v) or inspect.ismethod(v)):
if inspect.ismethod(v):
print(f"{v.__qualname__} -> class method ")
if inspect.isfunction(v):
sig = inspect.signature(v)
names = [n for n, _ in sig.parameters.items()]
if len(names) > 0 and str(names[0]) == 'self':
print(f"{v.__qualname__}-> probably a bound method ")
else:
print(f"{v.__qualname__}-> probably a static method ")


class Demo:

@classmethod
def us(cls):
print(cls.__name__)

@staticmethod
def double(x):
return x + x

def triple(self, y):
return 3 * y


analyze_class(Demo)

output:

Demo.double-> probably a static method
Demo.triple-> probably a bound method
Demo.us -> class method



From: Python-list <python-list-bounces+gweatherby=uchc.edu@python.org> on behalf of Ian Pilcher <arequipeno@gmail.com>
Date: Saturday, November 12, 2022 at 11:36 AM
To: python-list@python.org <python-list@python.org>
Subject: Re: Dealing with non-callable classmethod objects
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***

On 11/11/22 16:47, Cameron Simpson wrote:

On 11Nov2022 15:29, Ian Pilcher <arequipeno@gmail.com> wrote:

* Can I improve the 'if callable(factory):' test above? This treats
all non-callable objects as classmethods, which is obviously not
correct. Ideally, I would check specifically for a classmethod, but
there doesn't seem to be any literal against which I could check the
factory's type.

Yeah, it does feel a bit touchy feely.

You could see if the `inspect` module tells you more precise things
about the `factory`.

The other suggestion I have is to put the method name in `_attrs`; if
that's a `str` you could special case it as a well known type for the
factory and look it up with `getattr(cls,factory)`.

So I've done this.

class _HasUnboundClassMethod(object):
@classmethod
def _classmethod(cls):
pass # pragma: no cover
_methods = [ _classmethod ]

_ClassMethodType = type(_HasUnboundClassMethod._methods[0])

Which allows me to do this:

def __init__(self, d):
for attr, factory in self._attrs.items():
if callable(factory):
value = factory(d[attr])
else:
assert type(factory) is self._ClassMethodType
value = factory.__func__(type(self), d[attr])
setattr(self, attr, value)

It's a bit cleaner, although I'm not thrilled about having a throwaway
class, just to define a literal that ought to be provided by the
runtime.

--
========================================================================
Google Where SkyNet meets Idiocracy ========================================================================

-- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!nx6jxVGHt4Gj1WplLAV4uuhaMyS7Ry0qTCGvZm7jLCj9GbK4vto49sfmP12TTgcAT6Akjz5hJWw9JoylO_FrgQ$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/
python-list__;!!Cn_UX_p3!nx6jxVGHt4Gj1WplLAV4uuhaMyS7Ry0qTCGvZm7jLCj9GbK4vto49sfmP12TTgcAT6Akjz5hJWw9JoylO_FrgQ$>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 12Nov2022 10:34, Ian Pilcher <arequipeno@gmail.com> wrote:

So I've done this.

class _HasUnboundClassMethod(object):
@classmethod
def _classmethod(cls):
pass # pragma: no cover
_methods = [ _classmethod ]

_ClassMethodType = type(_HasUnboundClassMethod._methods[0])

Which allows me to do this:

def __init__(self, d):
for attr, factory in self._attrs.items():
if callable(factory):
value = factory(d[attr])
else:
assert type(factory) is self._ClassMethodType
value = factory.__func__(type(self), d[attr])
setattr(self, attr, value)

It's a bit cleaner, although I'm not thrilled about having a throwaway
class, just to define a literal that ought to be provided by the
runtime.

Ah, nice again.

You shouldn't need a throwaway class, just use the name "classmethod"
directly - it's the type!

if not callable(factory):
if type(factory) is classmethod:
# replace fctory with a function calling factory.__func__
factory = lambda arg: factory.__func__(classmethod, arg)
else:
raise TypeError("unhandled factory type %s:%r" % (type(factory), factory)
value = factory(d[attr])

Cheers,
Cameron Simpson <cs@cskk.id.au>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 12Nov2022 18:44, Weatherby,Gerard <gweatherby@uchc.edu> wrote:

Use the inspect module as Cameron suggested.

My suggestions was not for the post-construction class attributes but
for the comtents of the "_attrs" mapping. Post construction the class
method is callable. But the classmethod object stashed in "_attrs" is
not.

However, it _is_ of type "classmethod", which means it can be identified without using "inspect". Here's an alternative demo programme:

class Demo:

@classmethod
def us(cls):
print(cls.__name__)

@staticmethod
def double(x):
return x + x

def triple(self, y):
return 3 * y

_attrs = {
"double": double,
"triple": triple,
"us": us,
}

for name, attr in Demo._attrs.items():
print(name, "->", attr, type(attr))
print(" is class method =", type(attr) is classmethod)
print(" is callable =", callable(attr))
if inspect.ismethod(attr):
print(" ismethod")
if inspect.isfunction(attr):
print(" isfunction")

breakpoint()

I stuck a breakpoint in so that I could inspect things after the run.
The run looks like this:

py3 demo1.py
double -> <staticmethod object at 0x10e9c1340> <class 'staticmethod'>
is class method = False
is callable = False
triple -> <function Demo.triple at 0x10eafcd30> <class 'function'>
is class method = False
is callable = True
isfunction
us -> <classmethod object at 0x10e9c1250> <class 'classmethod'>
is class method = True
is callable = False

so just testing "type(attr) is classmethod" identifies the unpromoted
class method.

I need to real Ian's other post to see what he did to turn that into a
callable factory function.

Cheers,
Cameron Simpson <cs@cskk.id.au>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 13Nov2022 07:57, Cameron Simpson <cs@cskk.id.au> wrote:

# replace fctory with a function calling factory.__func__
factory = lambda arg: factory.__func__(classmethod, arg)

It just occurred to me that you might need to grab the value of factory.__func__ first:

factory0 = factory
factory = lambda arg: factory0.__func__(classmethod, arg)

Otherwise the closure might introduce a recursion.

Cheers,
Cameron Simpson <cs@cskk.id.au>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/12/22 14:57, Cameron Simpson wrote:

You shouldn't need a throwaway class, just use the name "classmethod" directly - it's the type!

    if not callable(factory):
        if type(factory) is classmethod:
            # replace fctory with a function calling factory.__func__
            factory = lambda arg: factory.__func__(classmethod, arg)
        else:
            raise TypeError("unhandled factory type %s:%r" % (type(factory), factory)
    value = factory(d[attr])

Or I could use the C++ version:

face << palm;

Thanks!

--
========================================================================
Google Where SkyNet meets Idiocracy ========================================================================

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 13Nov2022 10:08, Cameron Simpson <cs@cskk.id.au> wrote:

On 13Nov2022 07:57, Cameron Simpson <cs@cskk.id.au> wrote:

# replace fctory with a function calling factory.__func__
factory = lambda arg: factory.__func__(classmethod, arg)

It just occurred to me that you might need to grab the value of >factory.__func__ first:

factory0 = factory
factory = lambda arg: factory0.__func__(classmethod, arg)

Otherwise the closure might introduce a recursion.

Or avoid the closure:

from functools import partial
......
factory = partial(factory.__func__, classmethod)

to produces a partially filled in function.

Cheers,
Cameron Simpson <cs@cskk.id.au>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)