Skip to content

ES|QL query builder #2997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Jul 28, 2025
Merged

ES|QL query builder #2997

merged 14 commits into from
Jul 28, 2025
+4,283 −18

Conversation

miguelgrinberg
Copy link
Contributor

@miguelgrinberg miguelgrinberg commented Jul 1, 2025 •
edited
Loading

This PR adds a Python ES|QL query builder.

  • Implement all ES|QL commands
  • Add docstrings
  • Integration with DSL module --> Will be implemented later in a separate PR.
  • API documentation
  • Client documentation
  • Unit tests
  • Integration tests

@miguelgrinberg miguelgrinberg force-pushed the esql-query-builder branch 2 times, most recently from 168ddef to c6537d8 Compare July 21, 2025 19:20
@miguelgrinberg miguelgrinberg force-pushed the esql-query-builder branch 2 times, most recently from b4b6da4 to f87f3b5 Compare July 22, 2025 16:05
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 23, 2025 •
edited
Loading

🔍 Preview links for changed docs

@miguelgrinberg miguelgrinberg force-pushed the esql-query-builder branch 2 times, most recently from fce101d to 01374fb Compare July 24, 2025 13:22
@miguelgrinberg miguelgrinberg requested a review from pquentin July 24, 2025 14:18
@miguelgrinberg
Copy link
Contributor Author

miguelgrinberg commented Jul 24, 2025 •
edited
Loading

@pquentin this PR is ready for review. There are currently two remaining items:

  • The 3.8 integration tests continue to fail, unrelated to this change. There are 10 tests that fail, always the same ones, since the 15th. I haven't been able to figure out the cause, but all the failures seem to be related to indices that for some reason weren't wiped and consequently have more documents than expected or fail to be created again. In one of the 10 failures there is a permission failure when attempting to wipe an index, but the user is elastic, so I'm not quite follow why. For now I'm ignoring this problem.
  • I'm not going to write a full integration with the DSL module for the 9.1 release, as I don't want to rush this solution, which is not a trivial one. It is possible to do a lightweight integration by passing index or field names (as you can see in some of the tests), but what's missing is the ability to run an ES|QL query directly from a document class, which I'm going to design with more time.

@miguelgrinberg miguelgrinberg marked this pull request as ready for review July 24, 2025 14:27
miguelgrinberg
miguelgrinberg commented Jul 24, 2025
View reviewed changes
docs/reference/esql-query-builder.md
```python
>>> from elasticsearch import Elasticsearch
>>> client = Elasticsearch(hosts=[os.environ['ELASTICSEARCH_URL']])
>>> response = client.esql.query(query=str(query))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what your thoughts are about being able to pass the query object directly here and have the conversion to string in the generated method. Maybe something we can think about for the future as an improvement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea, and I don't think this can be a problem in the future

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have modified the Python client generator to allow this, so once we get that merged we can eliminate the requirement of converting the queries to strings. https://github.com/elastic/elastic-client-generator-python/pull/59.

@miguelgrinberg miguelgrinberg force-pushed the esql-query-builder branch 4 times, most recently from c7fc780 to 9c48982 Compare July 24, 2025 15:51
pquentin
pquentin reviewed Jul 25, 2025
View reviewed changes
Copy link
Member

@pquentin pquentin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! But I'd love if someone could properly review the code in elasticsearch/dsl.

docs/reference/esql-query-builder.md
```python
>>> from elasticsearch import Elasticsearch
>>> client = Elasticsearch(hosts=[os.environ['ELASTICSEARCH_URL']])
>>> response = client.esql.query(query=str(query))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea, and I don't think this can be a problem in the future

@miguelgrinberg miguelgrinberg merged commit 9eadabb into elastic:main Jul 28, 2025
15 checks passed
@miguelgrinberg miguelgrinberg deleted the esql-query-builder branch July 28, 2025 16:52
@github-actions GitHub Actions
Copy link

The backport to 8.19 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-8.19 8.19
# Navigate to the new working tree
cd .worktrees/backport-8.19
# Create a new branch
git switch --create backport-2997-to-8.19
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9eadabb0486f92bdc37312f56acfd40e62ec6fcf
# Push it to GitHub
git push --set-upstream origin backport-2997-to-8.19
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-8.19

Then, create a pull request where the base branch is 8.19 and the compare/head branch is backport-2997-to-8.19.

github-actions bot pushed a commit that referenced this pull request Jul 28, 2025
@miguelgrinberg @github-actions
ES|QL query builder (#2997)
a16d202 
* ES|QL query builder

* add missing esql api documentation

* add FORK command

* initial attempt at generating all functions

* unit tests

* more operators

* documentation

* integration tests

* add new COMPLETION command

* show ES|QL in all docs examples

* Docstring fixes

* add technical preview warning

(cherry picked from commit 9eadabb)
@github-actions github-actions bot mentioned this pull request Jul 28, 2025
Merged
miguelgrinberg added a commit to miguelgrinberg/elasticsearch-py that referenced this pull request Jul 28, 2025
@miguelgrinberg
ES|QL query builder (elastic#2997)
b503cfb 
* ES|QL query builder

* add missing esql api documentation

* add FORK command

* initial attempt at generating all functions

* unit tests

* more operators

* documentation

* integration tests

* add new COMPLETION command

* show ES|QL in all docs examples

* Docstring fixes

* add technical preview warning
@miguelgrinberg miguelgrinberg mentioned this pull request Jul 28, 2025
Draft
miguelgrinberg added a commit that referenced this pull request Jul 28, 2025
@github-actions @miguelgrinberg
ES|QL query builder (#2997) (#3009)
491adbe 
* ES|QL query builder

* add missing esql api documentation

* add FORK command

* initial attempt at generating all functions

* unit tests

* more operators

* documentation

* integration tests

* add new COMPLETION command

* show ES|QL in all docs examples

* Docstring fixes

* add technical preview warning

(cherry picked from commit 9eadabb)

Co-authored-by: Miguel Grinberg <miguel.grinberg@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pquentin pquentin pquentin left review comments

Assignees
No one assigned
Labels
backport 8.19 backport 9.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@miguelgrinberg @pquentin