Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional Arrow deserialization support #2632

Merged
merged 1 commit into from
Aug 7, 2024

Conversation

pquentin
Copy link
Member

@pquentin pquentin commented Aug 6, 2024

This is useful for the ES|QL Arrow output. This snippet:

query = "FROM kibana_sample_data_logs | DROP tags*,clientip | LIMIT 10"
df = client.esql.query(query=query, format="arrow").to_pandas()
print(df)

gives this:

               @timestamp                                              agent  \
0 2024-09-13 13:32:56.639  Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Geck...   
1 2024-09-13 12:06:25.326  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT ...   
2 2024-09-13 05:12:55.190  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT ...   
3 2024-09-13 11:23:28.525  Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Geck...   
4 2024-09-13 12:10:06.187  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT ...   
5 2024-09-13 09:49:22.691  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT ...   
6 2024-09-13 12:06:01.183  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT ...   
7 2024-09-13 09:46:57.860  Mozilla/5.0 (X11; Linux i686) AppleWebKit/534....   
8 2024-09-13 17:42:50.962  Mozilla/5.0 (X11; Linux i686) AppleWebKit/534....   
9 2024-09-13 16:19:34.755  Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.... 

And yes, IDEs can display data frames more nicely:

Screenshot 2024-08-06 at 17 51 50

Copy link

github-actions bot commented Aug 6, 2024

A documentation preview will be available soon.

Request a new doc build by commenting
  • Rebuild this PR: run docs-build
  • Rebuild this PR and all Elastic docs: run docs-build rebuild

run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.

If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.

@miguelgrinberg miguelgrinberg merged commit ef3da6e into elastic:main Aug 7, 2024
11 checks passed
github-actions bot pushed a commit that referenced this pull request Aug 7, 2024
miguelgrinberg pushed a commit that referenced this pull request Aug 8, 2024
(cherry picked from commit ef3da6e)

Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co>
@pquentin pquentin deleted the pyarrow-deserializer branch November 6, 2024 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants