Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autogenerate Query classes #1890

Merged
merged 27 commits into from
Sep 13, 2024

Conversation

miguelgrinberg
Copy link
Collaborator

@miguelgrinberg miguelgrinberg commented Aug 28, 2024

This change adds a code-generation phase to this client. In this first phase, code generation is applied to two files:

  • query.py: All the sub-classes of Query are auto-generated and include docstrings, type hints and a constructor with explicit arguments. All classes have kwargs that are backwards compatible.
  • interfaces.py: this is a new file in which all the types that are referenced by attributes of the Query sub-classes are defined.

Things that are not currently implemented but will likely be added later:

  • Shortcut properties.
  • Properties defined with generics in the specification are given an Any type for now.
  • All analyzers are given a Union[str, Dict[str, Any]] type for now.
  • Aggregations, Responses, possibly other things will be added in follow-up PRs.

@miguelgrinberg miguelgrinberg force-pushed the python-dsl-generator branch 3 times, most recently from ef8d929 to a0244fe Compare September 2, 2024 17:25
@miguelgrinberg miguelgrinberg marked this pull request as ready for review September 3, 2024 18:15
Copy link
Member

@pquentin pquentin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a first review after spending one hour scratching my head on this pull request.

While the inline comments help a lot, there are many things I don't understand yet. We're going to need the review of someone that maintains a typed client as well at some point. @JoshMock and @Anaethelion come to mind, as they have types in their client and are familiar with Python.

elasticsearch_dsl/utils.py Outdated Show resolved Hide resolved
elasticsearch_dsl/function.py Outdated Show resolved Hide resolved
elasticsearch_dsl/utils.py Show resolved Hide resolved
utils/generator.py Show resolved Hide resolved
utils/generator.py Outdated Show resolved Hide resolved
utils/generator.py Outdated Show resolved Hide resolved
@miguelgrinberg
Copy link
Collaborator Author

miguelgrinberg commented Sep 5, 2024

@pquentin thanks for all the feedback. I have addressed all your points I think, and made changes for each in a separate commit so that you can inspect them separately.

In addition to your feedback, I also added inherited attributes to all classes, which as I mentioned yesterday I forgot to include before. This actually made evident another thing I needed to handle, the AdditionalProperty behavior, which is used in a few of the query classes. This ends up being somewhat similar to a single-key dict, so I had to add this as well. I have to say that all the pain of adding types everywhere is proving very useful with this, because any time I make a mistake or forget something mypy reminds me I have more work to do!

Finally, I've been thinking that the name interfaces.py is odd for Python. Would renaming it to types.py be better? And also, do we want to dump all the queries and types in the root level of the package so that they can be imported directly via from elasticsearch_dsl import X? I recall you indicated you would prefer this way of working with these classes so let me know what you think.

Copy link
Member

@pquentin pquentin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's another round of review. I'm slowly getting to understand this better. Apologies for the reviewing delays, but this is quite daunting!

Finally, I've been thinking that the name interfaces.py is odd for Python. Would renaming it to types.py be better?

Yes, types is better than interfaces.

And also, do we want to dump all the queries and types in the root level of the package so that they can be imported directly via from elasticsearch_dsl import X? I recall you indicated you would prefer this way of working with these classes so let me know what you think.

Yes, I think that would fit the existing design better to dump all classes at the top-level. This will address one issue I have with the split between interfaces/types and queries: I still believe it should be an implementation detail that users don't have to care about.

elasticsearch_dsl/function.py Show resolved Hide resolved
elasticsearch_dsl/function.py Show resolved Hide resolved
elasticsearch_dsl/utils.py Show resolved Hide resolved
utils/generator.py Outdated Show resolved Hide resolved
elasticsearch_dsl/interfaces.py Outdated Show resolved Hide resolved
elasticsearch_dsl/interfaces.py Outdated Show resolved Hide resolved
elasticsearch_dsl/interfaces.py Outdated Show resolved Hide resolved
elasticsearch_dsl/interfaces.py Outdated Show resolved Hide resolved
utils/generator.py Outdated Show resolved Hide resolved
Copy link
Member

@pquentin pquentin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM.

utils/templates/query.py.tpl Outdated Show resolved Hide resolved
utils/templates/query.py.tpl Show resolved Hide resolved
utils/templates/query.py.tpl Show resolved Hide resolved
utils/templates/types.py.tpl Outdated Show resolved Hide resolved
utils/generator.py Show resolved Hide resolved
utils/generator.py Outdated Show resolved Hide resolved
@miguelgrinberg miguelgrinberg merged commit bacfc74 into elastic:main Sep 13, 2024
16 checks passed
@miguelgrinberg miguelgrinberg deleted the python-dsl-generator branch September 13, 2024 13:01
@miguelgrinberg miguelgrinberg added the backport 8.x Backport to 8.x label Sep 13, 2024
github-actions bot pushed a commit that referenced this pull request Sep 13, 2024
* Autogenerate Query classes

* handle Python reserved keywords such as from

* replace long deprecated 'filtered' query from tests

* minor code generation updates

* more code generation updates

* address typing issues

* clean up code generation templates

* more code generator cleanup

* add a unit test using some of the generated classes

* no need to "reset" the interface list

* use the transport's DEFAULT type

* include inherited properties in docstrings and constructors

* support legacy FieldValueFactor name for FieldValueFactorScore

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* use the identity operator to check for defaults

* rename interfaces.py to types.py

* leave undocumented classes and attributes with an empty docstring

* add unit test for AttrDict with from reserved keyword

* add a dependency on the transport library

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* list required arguments first in types.py

* add server defaults to argument docstrings

* remove unnecessary quotes from type hints in types.py

* Update utils/templates/types.py.tpl

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* final round of review improvements

---------

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>
(cherry picked from commit bacfc74)
miguelgrinberg added a commit that referenced this pull request Oct 7, 2024
* Autogenerate Query classes

* handle Python reserved keywords such as from

* replace long deprecated 'filtered' query from tests

* minor code generation updates

* more code generation updates

* address typing issues

* clean up code generation templates

* more code generator cleanup

* add a unit test using some of the generated classes

* no need to "reset" the interface list

* use the transport's DEFAULT type

* include inherited properties in docstrings and constructors

* support legacy FieldValueFactor name for FieldValueFactorScore

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* use the identity operator to check for defaults

* rename interfaces.py to types.py

* leave undocumented classes and attributes with an empty docstring

* add unit test for AttrDict with from reserved keyword

* add a dependency on the transport library

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* list required arguments first in types.py

* add server defaults to argument docstrings

* remove unnecessary quotes from type hints in types.py

* Update utils/templates/types.py.tpl

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* final round of review improvements

---------

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>
(cherry picked from commit bacfc74)

Co-authored-by: Miguel Grinberg <miguel.grinberg@gmail.com>
miguelgrinberg added a commit to miguelgrinberg/elasticsearch-dsl-py that referenced this pull request Dec 9, 2024
* Autogenerate Query classes

* handle Python reserved keywords such as from

* replace long deprecated 'filtered' query from tests

* minor code generation updates

* more code generation updates

* address typing issues

* clean up code generation templates

* more code generator cleanup

* add a unit test using some of the generated classes

* no need to "reset" the interface list

* use the transport's DEFAULT type

* include inherited properties in docstrings and constructors

* support legacy FieldValueFactor name for FieldValueFactorScore

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* use the identity operator to check for defaults

* rename interfaces.py to types.py

* leave undocumented classes and attributes with an empty docstring

* add unit test for AttrDict with from reserved keyword

* add a dependency on the transport library

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* list required arguments first in types.py

* add server defaults to argument docstrings

* remove unnecessary quotes from type hints in types.py

* Update utils/templates/types.py.tpl

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* Update utils/generator.py

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>

* final round of review improvements

---------

Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 8.x Backport to 8.x
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants