gh-89263: Add typing.get_overloads

[JelleZijlstra]

Credit to Spencer Brown for the idea to use overloads' firstlineno to decide whether to clear the registry.

#89263

https://bugs.python.org/issue45100

[AlexWaygood]

Thanks! I have a few thoughts.

Overall, I guess I'm a little concerned about "dumping" so many new functions in the global functools namespace. functools is already kind of a hodgepodge of unrelated things, and I worry that this will make the problem worse.

Normally I'd say that having a class with 0 instance methods would be a code smell, but what about something like this?

class VariantRegistry:
    _registry = defaultdict(list)

    @staticmethod
    def _get_key_for_callable(func):
        func = getattr(func, "__func__", func)
        try:
            return f"{func.__module__}.{func.__qualname__}"
        except AttributeError:
            return None

    @classmethod
    def register_variant(func, variant):
        key = cls._get_key_for_callable(func)
        if key is not None:
            cls._variant_registry[key].append(variant)

    @classmethod
    def get_variants(cls, func):
        return cls._registry[cls._get_key_for_callable(func)]

    @classmethod
    def clear_variants(cls, func=None):
        if func is None:
           cls._variant_registry.clear()
        else:
            cls._variant_registry.pop(cls._get_key_for_callable(func), None)

[JelleZijlstra]

Thanks for the review!

I see the point about dumping, but I don't think using a class just as a bag of methods is a good solution. This group of functions serves a similar purpose to update_wrapper and wraps (enabling introspection for advanced usage of functions) and functools now feels like a good home for them.

[AlexWaygood]

What happens if a user does something weird (and probably misguided?) like this?

@overload
def complex_func(arg: str) -> int: ...

@overload
def complex_func(arg: int) -> str: ...

@singledispatch
def complex_func(arg: object):
    raise NotImplementedError

@complex_func.register
def _(arg: str) -> int: 
    return int(arg)

@complex_func.register
def _(arg: int) -> str:
    return str(arg)

What's the runtime behaviour in this case? Do we care? Do we warn users not to do this in the docs? Do we add a test case to make sure the behaviour is "correct"?

I think we need to account for this possible (mis)usage somehow, even if type checkers would probably balk at it.

[JelleZijlstra]

What's the runtime behaviour in this case? Do we care? Do we warn users not to do this in the docs? Do we add a test case to make sure the behaviour is "correct"?

I think we need to account for this possible (mis)usage somehow, even if type checkers would probably balk at it.

We simply get variants both from overload and singledispatch. I added a test case based on this example.

[JelleZijlstra]

Implemented this approach. I now see a 5x performance ratio:

In [9]: %timeit overload(f)
223 ns ± 1.85 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [10]: %timeit old_overload(f)
44.7 ns ± 0.74 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [11]: 223 / 44.7
Out[11]: 4.988814317673378

[gvanrossum]

Can you add a comment explaining why this sequence is repeated?

[JelleZijlstra]

Done. Also added a test case to make sure that clearing one function's overloads doesn't clear another.

Thinking out loud, clear_overloads() is becoming the most complicated part of this. Perhaps we don't need to support clearing overloads per function, which would simplify it greatly. Use cases I can think of are:

• You want to save some memory after importing all your code -> you call clear_overloads() and get rid of them all.
• You are constantly reloading a module with overloads in it and changing the line numbers. Then maybe you want to clear_overloads() for just that function. But that seems really exotic.

So maybe we should only support clearing the whole overload registry at once. Can you think of other realistic reasons people may want to use clear_overloads()?

[gvanrossum]

Hm. Given the reload(module) use case, maybe the only options should be clear all or clear a specific module? The new data structure makes that easy. The question would then be whether the argument should be the (full) module name or the module object? (Or either?) The sequence of events would be something like

typing.clear_overload(mod)
importlib.reload(mod)

I can't really think of a use case for clearing a specific function. Did you have one in mind originally? Or was it just convenient for testing?

[JelleZijlstra]

I think for now we should allow only clearing the whole registry. I see the use case for wanting to repeatedly reload modules while wanting to introspect overloads, but it seems unlikely to be common. We can always add an argument later to clear overloads by module if someone asks for it, but once we add it we're stuck with it.

[JelleZijlstra]

@gvanrossum @AlexWaygood any further feedback? The last thing to resolve is whether clear_overloads should take arguments (#31716 (comment)).

[AlexWaygood]

This looks pretty good now! Just a few thoughts:

[AlexWaygood]

This sentence sounds quite strange to me:

*func* is the function object for the implementation of the overloaded function.

But I'm not sure I have a better suggestion off the top of my head ://

[JelleZijlstra]

Yes, unfortunately the overload docs don't have a clear term for the implementation function either.

[gvanrossum]

I'll wait until @AlexWaygood approves, then I'll do one more pass if you want me to.

[AlexWaygood]

Suggested change

	``get_overloads`` returns an empty sequence.
	return an empty sequence.

Tiny nit -- this paragraph starts off using the imperative mood

[JelleZijlstra]

But the previous sentence is in the indicative, so I think this is fine?

[AlexWaygood]

Well, it's not really a big deal either way :)

[AlexWaygood]

Hooray!

[JelleZijlstra]

@gvanrossum would you like to take another look or should I merge this?

[gvanrossum]

Go for it!