[Discover/CSV Reporting] Use the new CSV export endpoint in Discover UI #151190
Labels
Feature:Reporting:CSV
Reporting issues pertaining to CSV file export
Team:DataDiscovery
Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL.
Team:SharedUX
Team label for AppEx-SharedUX (formerly Global Experience)
technical debt
Improvement of the software architecture and operational architecture
Comments
tsullivan
added
technical debt
|
Pinging @elastic/appex-sharedux (Team:SharedUX) |
tsullivan
added
the
impact:low
1 task
|
This requires enhancing the utilities in discover/server to handle "unsaved searches," or |
1 task
tsullivan
added a commit
that referenced
this issue
Feb 22, 2023
## Summary This new CSV endpoint will be usable for scripted reporting. As of this PR, the endpoint is not going to be used in the UI: that will be handled by [this issue](#151190). The new endpoint is not documented yet, but should be once the endpoint is used in the UI, according to [this issue](#151745). Depends on #150631 Closes #148775 - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ## Using the endpoint from a script A request to generate a report has a string of rison-encoded job parameters. The parameters can now be made into a template for customers to inject their own savedSearchID. The time range of the search, filters, and query can also be dynamically filled in at the time of requesting a report - much more easily than before. ```sh SAVED_SEARCH_ID="d779c5d0-a1c3-11ed-bd60-6957c24fc275" # Inject the search ID into the "jobParams" template JOB_PARAMS="(browserTimezone:UTC,locatorParams:!((id:DISCOVER_APP_LOCATOR,params:(savedSearchId:'${SAVED_SEARCH_ID}',timeRange:(from:now-1M,to:now)))),objectType:search)" # Send a request to generate a report curl -XPOST $ENDPOINT \ -H "${AUTH_HEADER}" \ -H 'Content-Type: application/json' \ -H 'kbn-xsrf: reporting' -d '{ "jobParams": "'$JOB_PARAMS'"}' ``` ## Limitations A locator without a `savedSearchId` is currently not supported. Getting that support will make it possible to create exports based on content in Discover tied only to a Data View (unsaved search support). ## (Demo) Developer Example app _The demo app is descoped from this PR_ ~~This PR updates the Reporting example app to feature a "CSV Explorer." In this app, you can select a saved search, build up a DiscoverAppLocatorParams object, and send a request to generate a report.~~ https://user-images.githubusercontent.com/908371/217356050-f556f897-33c6-4623-aa06-9af191378e48.mp4 ## Release Note We are skipping the release note for now, while waiting for documentation. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
9 tasks
tsullivan
added
the
(Deprecated) Feature:Reporting
Dosant
added a commit
that referenced
this issue
Jan 17, 2024
## Summary close #173390 This PR enables CSV report generation with ES|QL in Discover. Pre this PR, there are two report types for generating CSV reports with Discover: - https://github.com/elastic/kibana/blob/main/packages/kbn-reporting/export_types/csv/csv_searchsource.ts - old deprecated report type that relies on `SerializedSearchSourceFields`. This type is still currently used by Discover UI and we plan to migrate away from it #151190 - https://github.com/elastic/kibana/blob/main/packages/kbn-reporting/export_types/csv/csv_v2.ts - new report type that relies on Discover locator. Currently, it can only generate reports from searches backed by saved search and this report type was implemented as a public-facing API for simple report generation outside Kibana. Since we plan to migrate to the v2 report type and search source is not needed for es|ql, this PR implements es|ql csv reporting based on the v2 report type. Initially, I wanted to create new new report type similar to v2 just for es|ql, but it turned out to be a lot more boilerplate code without a significant benefit. you can see it in the PR #174448. So I changed my mind and the current PR adds es|ql capabilities inside the existing csv_v2 report. This is convenient as the input is the same (Discover locator), the output is the same csv file and meta information, and telemetry is also the same. As of this PR, the ES|QL report is capable of: - Using es|ql query from the locator - Adding time range filter if present in the locator. time field is picked from the data view (it is available in the locator, but otherwise is not used for es|ql). Other filters are also passed if available in the locator - Keeps into account "columns" from the locator. - Similar to current non-es|ql reports from discover UI, it doesn't use saved searches but only relies on the state from the URL. This probably will be improved in #151190 to support both. - Uses different CSV settings for functionality like checking formulas, escapes, bom, max size, etc... - Keeps regular CSV features like cancelation and giving event loop a break (even though those are not needed for now for es|ql since the number of results is limited) Some notable differences compared to regular discover search / csv reports: - A lot simpler, as it doesn't use search source and field formats - No pagination, less CPU heavy as esql responses are limited to 10000 results and a single request
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this issue
Feb 15, 2024
## Summary close elastic#173390 This PR enables CSV report generation with ES|QL in Discover. Pre this PR, there are two report types for generating CSV reports with Discover: - https://github.com/elastic/kibana/blob/main/packages/kbn-reporting/export_types/csv/csv_searchsource.ts - old deprecated report type that relies on `SerializedSearchSourceFields`. This type is still currently used by Discover UI and we plan to migrate away from it elastic#151190 - https://github.com/elastic/kibana/blob/main/packages/kbn-reporting/export_types/csv/csv_v2.ts - new report type that relies on Discover locator. Currently, it can only generate reports from searches backed by saved search and this report type was implemented as a public-facing API for simple report generation outside Kibana. Since we plan to migrate to the v2 report type and search source is not needed for es|ql, this PR implements es|ql csv reporting based on the v2 report type. Initially, I wanted to create new new report type similar to v2 just for es|ql, but it turned out to be a lot more boilerplate code without a significant benefit. you can see it in the PR elastic#174448. So I changed my mind and the current PR adds es|ql capabilities inside the existing csv_v2 report. This is convenient as the input is the same (Discover locator), the output is the same csv file and meta information, and telemetry is also the same. As of this PR, the ES|QL report is capable of: - Using es|ql query from the locator - Adding time range filter if present in the locator. time field is picked from the data view (it is available in the locator, but otherwise is not used for es|ql). Other filters are also passed if available in the locator - Keeps into account "columns" from the locator. - Similar to current non-es|ql reports from discover UI, it doesn't use saved searches but only relies on the state from the URL. This probably will be improved in elastic#151190 to support both. - Uses different CSV settings for functionality like checking formulas, escapes, bom, max size, etc... - Keeps regular CSV features like cancelation and giving event loop a break (even though those are not needed for now for es|ql since the number of results is limited) Some notable differences compared to regular discover search / csv reports: - A lot simpler, as it doesn't use search source and field formats - No pagination, less CPU heavy as esql responses are limited to 10000 results and a single request
petrklapka
added
the
Team:DataDiscovery
|
Pinging @elastic/kibana-data-discovery (Team:DataDiscovery) |
tsullivan
added
Feature:Reporting:CSV
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In #149172, a new CSV export type was added, which is powered by DiscoverAppLocatorParams and offers a much friendlier interface for custom reporting automation. The ease-of-use comes from the fact that DiscoverAppLocatorParams can simply have a
savedSearchIdinjected, which abstracts all of the details of the search configuration. That offers an advantage of powering CSV export using SearchSourceFields.This issue is to promote the new export type in the Discover UI. When users export CSV in Discover, the request will go to the new Reporting endpoint (naming TBD). Requests to the current
/api/reporting/generate/csv_searchsourcewill still work for BWC.📄 Deprecation Plan
The text was updated successfully, but these errors were encountered: