When inheriting a type variable defined as Generic, how can I write documentation with the concretized type variable?

[jonghwanhyeon]

Based on what I’ve found, unless the child class explicitly defines the type annotation, the type will be represented by the type variable defined as Generic.

To address this, I wrote an Extension, but I found that it applies to the Signature section and not to the Parameters section.

Is there a part I missed in the Extension I wrote, or is there an easier way to solve this problem?

[pawamoy]

Could you elaborate on what you're trying to achieve with code examples? I'm not sure to understand. I also wasn't able to understand what your code is doing by quickly looking at it.

If you modify the signature in-place, it should also impact how the docstring is parsed, and therefore what the parameters section displays. Maybe it doesn't because you're copying objects instead of modifying them in-place?

[pawamoy]

@jonghwanhyeon I see you closed the discussion. Have you resolved your issue? Could you tell us how?

[jonghwanhyeon]

The following is a proof-of-concept code for what I want to do. As you can see, deepcopy() takes a very long time when iterating through members.

class GenericsExtension(Extension):
    def on_package_loaded(self, *, pkg: Module) -> None:
        cls = pkg.get_member("eventemitter.EventEmitter")
        console.log("- cls:", cls)
        for base_cls in reversed(cls.mro()):
            console.log("- base_cls:", base_cls)
            for key in base_cls.members.keys():
                console.log("- key:", key)
                if key in cls.members:
                    continue

                member = base_cls.get_member(key)
                copied = deepcopy(member)
                cls.set_member(key, copied)

[17:40:58] - cls: Class('EventEmitter', 349, 391)                                                                                                                           generics.py:365
           - base_cls: Class('AbstractEventEmitter', 116, 346)                                                                                                              generics.py:367
           - key: __slots__                                                                                                                                                 generics.py:369
           - key: _handler_cls                                                                                                                                              generics.py:369
           - key: __init__                                                                                                                                                  generics.py:369
           - key: _events                                                                                                                                                   generics.py:369
           - key: add_listener                                                                                                                                              generics.py:369
           - key: prepend_listener                                                                                                                                          generics.py:369
[17:40:59] - key: prepend_once_listener                                                                                                                                     generics.py:369
           - key: emit                                                                                                                                                      generics.py:369
           - key: events                                                                                                                                                    generics.py:369
           - key: listeners                                                                                                                                                 generics.py:369
[17:41:00] - key: off                                                                                                                                                       generics.py:369
[17:41:02] - key: on                                                                                                                                                        generics.py:369
[17:41:06] - key: once                                                                                                                                                      generics.py:369
[17:41:14] - key: remove_all_listeners   

[pawamoy]

Griffe trees are very interlinked, so I suppose deepcopying an object results in copying the whole tree (or almost the whole tree). I'm still not sure why you want to copy objects or (re)create them, as you haven't elaborated on your original use-case 😅

[jonghwanhyeon]

I think the following serialization can explain my use case. Since I defined my base classes using typing.Generic, all subclasses have defined their concrete types for the generic type parameters as follows:

class AbstractEventEmitter(ABC, Generic[L, H]):
    pass 

class EventEmitter(AbstractEventEmitter[Listenable, Handler]):
    pass

(Full code: https://gist.github.com/jonghwanhyeon/3c9ce95e665e6e036b193ec8c7d944e9)

The problem is that since all the common code is written in the AbstractEventEmitter class, only minimal code is written in the EventEmitter class. It would be nice to simply use the inherited_members option of mkdocstrings, but this option just uses a reference from the Base class and ignores the Concrete Types defined by the Base class (e.g., L -> Listenable).

The following serialization example would help you understand my use-case hopefully.

# Before - parameters[2].annotation is a type parameter defined in Generic[]
# class AbstractEventEmitter(ABC, Generic[L, H]):
#   pass 
{                                                                                                                                                                generics.py:380
    'kind': <Kind.FUNCTION: 'function'>,                                                                                                                                        
    'name': 'add_listener',                                                                                                                                                     
    'lineno': 139,                                                                                                                                                              
    'endlineno': 149,                                                                                                                                                           
    'docstring': <_griffe.models.Docstring object at 0x7fd38e8058b0>,                                                                                                           
    'labels': set(),                                                                                                                                                            
    'members': [],                                                                                                                                                              
    'decorators': [],                                                                                                                                                           
    'parameters': [                                                                                                                                                             
        {'name': 'self', 'annotation': None, 'kind': <ParameterKind.positional_or_keyword: 'positional or keyword'>, 'default': None},                                          
        {                                                                                                                                                                       
            'name': 'event',                                                                                                                                                    
            'annotation': ExprName(name='Hashable', parent=Class('AbstractEventEmitter', 116, 346)),                                                                            
            'kind': <ParameterKind.positional_or_keyword: 'positional or keyword'>,                                                                                             
            'default': None                                                                                                                                                     
        },                                                                                                                                                                      
        {                                                                                                                                                                       
            'name': 'listener',                                                                                                                                                 
            'annotation': ExprName(name='L', parent=Class('AbstractEventEmitter', 116, 346)),                                                                                   
            'kind': <ParameterKind.positional_or_keyword: 'positional or keyword'>,                                                                                             
            'default': None                                                                                                                                                     
        }                                                                                                                                                                       
    ],                                                                                                                                                                          
    'returns': ExprName(name='Self', parent=Class('AbstractEventEmitter', 116, 346))                                                                                            
}                          

# After - parameters[2].annotation is an actual type, `Listener`
# class EventEmitter(AbstractEventEmitter[Listenable, Handler]):
#     pass
{                                                                                                                                                                generics.py:385
    'kind': <Kind.FUNCTION: 'function'>,                                                                                                                                        
    'name': 'add_listener',                                                                                                                                                     
    'lineno': 139,                                                                                                                                                              
    'endlineno': 149,                                                                                                                                                           
    'docstring': <_griffe.models.Docstring object at 0x7fd38e8058b0>,                                                                                                           
    'labels': set(),                                                                                                                                                            
    'members': [],                                                                                                                                                              
    'decorators': [],                                                                                                                                                           
    'parameters': [                                                                                                                                                             
        {'name': 'self', 'annotation': None, 'kind': <ParameterKind.positional_or_keyword: 'positional or keyword'>, 'default': None},                                          
        {                                                                                                                                                                       
            'name': 'event',                                                                                                                                                    
            'annotation': ExprName(name='Hashable', parent=Class('AbstractEventEmitter', 116, 346)),                                                                            
            'kind': <ParameterKind.positional_or_keyword: 'positional or keyword'>,                                                                                             
            'default': None                                                                                                                                                     
        },                                                                                                                                                                      
        {                                                                                                                                                                       
            'name': 'listener',                                                                                                                                                 
            'annotation': ExprName(name='Listenable', parent=Module(PosixPath('/workspace/python-eventemitter/eventemitter/eventemitter.py'))),                                 
            'kind': <ParameterKind.positional_or_keyword: 'positional or keyword'>,                                                                                             
            'default': None                                                                                                                                                     
        }                                                                                                                                                                       
    ],                                                                                                                                                                          
    'returns': ExprName(name='Self', parent=Class('AbstractEventEmitter', 116, 346))                                                                                            
}      

[jonghwanhyeon]

If deepcopy() is not a suitable method, what are the alternative ways to create Function objects? I only want to dynamically add annotations to Functions. I have also tried using aliases to create objects, but I found that they still share the same reference value. Consequently, any modifications made in AbstractEventEmitter.add_listener() are also reflected in EventEmitter.add_listener().

[pawamoy]

Thanks for the explanation, that's much more clear now! So indeed your approach seems right: re-creating functions of subclasses to update their type annotation to use the concrete types. As you found out, it's possible to deep-copy objects while preventing copying too much of the tree. I suppose it's also possible to recreate the objects manually (and I think you tried that, but something didn't work?).

old_parameters = old_function.parameters
new_parameters = [
    Parameter(name=p.name, annotation=new_annotation, kind=p.kind, default=p.default)
    for p in old_parameters
]
new_function = Function(name=old_function.name, ..., parameters=Parameters(*new_parameters))
parent.set_member(new_function.name, new_function)

...something like this.

[jonghwanhyeon]

I solved the problem by using memo parameter of deepcopy() as follows:

function =deepcopy(member, memo={id(member.parent): member.parent})

[jonghwanhyeon]

Also I made an Extension package for my use-case. https://github.com/jonghwanhyeon/griffe-generics

Anyway, thanks for your support! 🙏

[pawamoy]

Nice, congrats on fixing your issue!