How to convert annotations to Format.SOURCE in __annotate__?

[sobolevn]

Feature or enhancement

Let's say you have a dict of some custom annotations like I have in #122262

How users are expected to convert say an annotation dict of {'user': CustomUser[AuthToken], 'auth_callback': Callable[[CustomUser[T]], T]} to string?

There are several ways right now:

1. repr for simple types, which might not work for complex ones

2. cpython/Lib/annotationlib.py

   Lines 466 to 501 in 536bc8a

   	if format == Format.SOURCE:
   	# SOURCE is implemented by calling the annotate function in a special
   	# environment where every name lookup results in an instance of _Stringifier.
   	# _Stringifier supports every dunder operation and returns a new _Stringifier.
   	# At the end, we get a dictionary that mostly contains _Stringifier objects (or
   	# possibly constants if the annotate function uses them directly). We then
   	# convert each of those into a string to get an approximation of the
   	# original source.
   	globals = _StringifierDict({})
   	if annotate.__closure__:
   	freevars = annotate.__code__.co_freevars
   	new_closure = []
   	for i, cell in enumerate(annotate.__closure__):
   	if i < len(freevars):
   	name = freevars[i]
   	else:
   	name = "__cell__"
   	fwdref = _Stringifier(ast.Name(id=name))
   	new_closure.append(types.CellType(fwdref))
   	closure = tuple(new_closure)
   	else:
   	closure = None
   	func = types.FunctionType(
   	annotate.__code__,
   	globals,
   	closure=closure,
   	argdefs=annotate.__defaults__,
   	kwdefaults=annotate.__kwdefaults__,
   	)
   	annos = func(Format.VALUE)
   	if _is_evaluate:
   	return annos if isinstance(annos, str) else repr(annos)
   	return {
   	key: val if isinstance(val, str) else repr(val)
   	for key, val in annos.items()
   	}

   but, it requires complex annotate object

3. cpython/Lib/typing.py

   Lines 2955 to 2956 in 536bc8a

   	def _convert_to_source(types):
   	return {n: t if isinstance(t, str) else _type_repr(t) for n, t in types.items()}

   but it is a private API

I propose adding a public and documented API for that.

Linked PRs

• gh-124412: Add helpers for converting annotations to source format #124551

[JelleZijlstra]

The use cases for this would be cases where types get provided outside annotations (e.g., with the functional syntax for NamedTuple and TypedDict, and with make_dataclass), and you want the __annotate__ method to support the SOURCE format.

The implementation is repr() for most cases, but for types we want the fully qualified name instead of <type 'int'>, and there are a few other special cases. Currently we have typing._convert_to_source, which takes an annotations dict and uses typing._type_repr to repr each annotation.

Since we already have use cases from two standard library modules (typing and dataclasses), I think it makes sense to add something to annotationlib. I would suggest:

• annotationlib.repr_annotations(dict), similar to the current typing._convert_source
• annotationlib.repr_type(type), which works like typing._type_repr but without the special case for tuples

[sobolevn]

I want to investigate on this feature, because I still have a limited understanding of __annotate__ and all of its corner-cases.

Will probably work on this tomorrow 👍

[JelleZijlstra]

Thanks! We'll also want to add the new functions to PEP 749.

[JelleZijlstra]

We also need this in annotationlib itself, for the edge case where a user-created object has __annotations__ but not __annotate__. Currently, in that case SOURCE returns non-strings:

>>> class X:
...     @property
...     def __annotations__(self):
...         return {"x": int}
...         
>>> x = X()
>>> import annotationlib
>>> annotationlib.get_annotations(x, format=annotationlib.Format.SOURCE)
{'x': <class 'int'>}

I think the most reasonable behavior for this case is to return {'x': 'int'}.

[larryhastings]

My initial reaction is, this seems like a least-worst option, and it may even be the least-worst option. I'd like to marinate on it further. I won't mind too much if you go ahead and merge without my blessing--as long as you don't mind me retroactively pushing back if I (eventually) arrive at some other conclusion.

[larryhastings]

I will say, shenanigans like this are an argument against calling this format SOURCE.

[sobolevn]

@JelleZijlstra can this be closed now? Sorry, I was not quick enough to send a PR :( It was only partially ready yesterday, due to some other work.

[JelleZijlstra]

Let's leave it open for a while to allow Larry to ruminate.

[larryhastings]

Jelle has convinced himself that we should change the name away from SOURCE, I think he preferred STRINGS which is fine.

It's a day later and this approach seems like a good idea. Ship it! We can always regret it later.

[JelleZijlstra]

To be precise I propose STRING, since the other formats are also in the singular.

[larryhastings]

STRING is fine by me. My only counter-proposal is TEXT, which is also sufficiently singular. I don't have a strong opinion about it either way; I don't think TEXT does any better job of conveying the intended semantics of the format.