-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
async ESQL and _search: add search_id, is_running and is_complete to response headers #109576
Comments
@ppisljar and I discussed and alternative, which is always sending |
I've tagged this as |
Pinging @elastic/es-analytical-engine (Team:Analytics) |
Pinging @elastic/es-search (Team:Search) |
Pinging @elastic/es-search-foundations (Team:Search Foundations) |
Technically I think all you need is |
All I see in the responses is
Is there an |
Adding some more context here around the relevance of this issue for Kibana. Kibana-server introduces overhead when querying Elasticsearch. This is due to how the implementation in the
(1) can be addressed by moving to http2. This would mainly be an architecture improvement, with some marginal performance benefits. (2) is blocked by this issue here (#109576). There are a few use-cases in polling for async-responses where some of metadata is needed from theES-response body. Can this be moved to a header (exact properties TBD). Running kibana by default in http2 and removing wrt cbor and streaming. Removing any decoding requirements from kibana-server, will allow longer-term improvements like using a more efficient data-format (cbor), or even faster delivery (stream results). However, these are longer-term improvements and not the most critical from a short-term perspective. |
Regarding "use cbor response format in kibana": starting with 8.15, ES|QL results can be output as Apache Arrow dataframes which are generally of the same size or smaller than cbor encoding and can be used with no decoding/deserialization. This was added in #109873 |
Add headers to async ESQL queries to show the status and query ID without having to parse the body. ESQL part of #109576
…c#111840) Add headers to async ESQL queries to show the status and query ID without having to parse the body. ESQL part of elastic#109576
Just merged the last PR adding headers.
|
…c#111840) Add headers to async ESQL queries to show the status and query ID without having to parse the body. ESQL part of elastic#109576
thx @swallez - wrt #109576 (comment) We have done some investigation in arrow and we noticed that the client-side support in Javascript is quite poor right now, and any gains in filesize, Kibana loses in parsing/reading of the format (elastic/kibana#183909). So there is no immediate plan to use arrow as transfer format for Kibana. |
Thanks for the feedback @thomasneirynck. I'll look at that PR to understand if/how we can avoid any form of parsing, which is an essential part of the benefits of Arrow beyond network payload size. |
I should clarify we mainly looked at it from a dashboard/charting perspective.
Much has to do with how Lens and elastic/charts require their data to be laid out a specific way before data is drawn as a chart. Different models (e.g. loading arrow-buffers straight on GPU) could avoid all this, but there are not actionable in the short term. More research is needed though, and especially when we're thinking about ES|QL-charts, we may be able to take shortcuts. |
I did some investigation an posted the results here elastic/kibana#175695 (comment) |
Description
In order to be able to use cbor response format in kibana and avoid decoding the body on kibana server, we would need search_id, is_running and is_complete to be returned as part of response headers (on top of returning them in body as today)
The text was updated successfully, but these errors were encountered: