A couple of _ast related issues

[isidentical]

I was checking out the type stub, and found these points;

• There are a bunch of docstring: str nodes guarded with sys.version >= (3, 7). This is not actually true since bpo-32911: Remove the docstring attribute of AST types cpython#7121 reverts the change that introduced docstring (that PR also targets 3.7). They can be safely eliminated
• There is no type_comment on FunctionDef/AsyncFunctionDef (and maybe on other nodes)
• string fields in the ASDL are encoded as str. The problem is string corresponds to typing.AnyStr, not str. (https://github.com/python/cpython/blob/cda99b4022daa08ac74b0420e9903cce883d91c6/Python/Python-ast.c#L1195-L1202)
• As mentioned in this TODO: , every node has current position information (lineno/col_offset) however this is not the case.

While examining this, I got an idea of autogenerating stub files by inputting a program with ASDLs (abstract syntax description language). So I wrote generators/typing_stub.py to my ASDL implementation, also generated test stub file.

I didn't test using that stub file, so don't know if there is a problem with it, but from a high level view they look similar with the _ast.pyi here, plus it covers the points I just mentioned (be aware, it is completely auto generated). There is only a single problem with that stub file, it ignores one of the broken parts of our AST. Dict(expr* keys, expr* values) actually doesn't describe the Dict well since when one of the items is **star_unpack the keys will contain None (so it should be List[Optional[expr]] not List[expr]). The currently type stub handles it, so if anyone wants to make a PR by copying things over the auto generated type stub, this point should not be actually copied.

[isidentical]

Another thing I noticed is;

import ast
node = ast.AST(body=[])
print(node.body)

 $ mypy t.py                    
t.py:4: error: "AST" has no attribute "body"
Found 1 error in 1 file (checked 1 source file)

I believe (did not test) can be fixed using something similiar to types.SimpleNamespace.
https://github.com/isidentical/pyasdl/blob/c44b9a87ae9b6cf5ab98ef0f32b1757964d5a0de/generators/PythonAST.pyi#L10-L12

[srittau]

Thank you and sorry for the late reply. Autogenerating the stub file sounds interesting indeed. I will take a closer look when I find the time.

[srittau]

The only problem I see with autogenerating is supporting multiple Python version, which we need to do in typeshed.

[isidentical]

The only problem I see with autogenerating is supporting multiple Python version, which we need to do in typeshed.

I don't quite get what you meant here, pyasdl's type stub generator generates for multiple versions. And also tries to do it in a gentle way. Check out the output stub.

[srittau]

I totally missed that, that looks great!

[Avasam]

1. Python 3.7: Addressed in ast: remove docstring attribute #4868 But also support for 3.7 and 3.8 are now dropped.
2. type_comment: Added in improvement: Add AST subclass constructors #11880
3. Is there specific string fields identified? I'm not sure whether that's still something that needs to be updated or not.
4. The TODO was also addressed in improvement: Add AST subclass constructors #11880
5. Is this something we want to support? (with a typevar with can at least encode whether there are any custom attributes)

   from ast import AST
   AST(custom_attribute=[]).custom_attribute

It's not the best but it kinda works better than just a bare Any:

_CustomASTAttributesT = typing_extensions.TypeVar("_CustomASTAttributesT", default=NoReturn)
class AST(Generic[_CustomASTAttributesT]):
    @overload
    def __init__(self) -> None: ...
    @overload
    def __init__(self, **kwargs: _CustomASTAttributesT) -> None: ...
    def __getattribute__(self, name: str) -> _CustomASTAttributesT: ...

(the wonders AnyOf from a union would do here 😉 )

Edit: Oh! Even better: <python-input-1>:1: DeprecationWarning: ast.AST.__init__ got an unexpected keyword argument 'test'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.