[Reporting] Restore the "csv by savedobject" endpoint for 7.17 #148030

tsullivan · 2022-12-22T19:45:36Z

Summary

This restores an endpoint that was added in 7.3 in this PR, and was removed in 7.9 in this PR. The changes are re-done on top of 7.17, but still has a mostly-compatible with the one that existed in 7.3-7.8. This serves 3rd parties that relied on the earlier experimental code.

Supports:

Saved searches with filters
Saved searches with custom sorting
Saved searches with or without selected columns
Exports based on Index Patterns with or without a "time field"
Requests can have an optional POST body with extra time range filters and/or specify a custom time zone.

LIMITATIONS:

This endpoint is currently not supported in 8.x at this time.
Saved Search objects created in older versions of Kibana may not work.
Searching across hundreds of shards in the query could cause Elasticsearch instability.
Some minor bugs in the output of the CSV may exist, such as fields not being formatted exactly as in the Discover table.
This code may be forward-ported to main in a way that uses a different API that is not compatible with this change.
Does not allow "raw state" to be merged with the Search object, as in the previous code. Otherwise, the API is compatible with the previous code.
This feature remains in "experimental" status, and is not ready to be documented at this time.

Testing

Since there is not a UI for this endpoint, there are a few options for testing:

Run the functional test:

node scripts/functional_tests.js \
  --config x-pack/test/reporting_api_integration/reporting_and_security.config.ts \
  --grep 'CSV Generation from Saved Search ID'

Create a saved search in Kibana, and use a script to send a request

POST_URL="${HOST}/api/reporting/v1/generate/csv/saved-object/search:"$SAVED_SEARCH_ID

## Run transaction to generate a report, wait for execution completion, download the report, and send the
# report as an email attachment

# 1. Send a request to generate a report
DOWNLOAD_PATH=$(curl --silent -XPOST "$POST_URL" -H "kbn-xsrf: kibana-reporting" -H "${AUTH_HEADER}" | jq -e -r ".payload.path | values")
if [ -z "$DOWNLOAD_PATH" ]; then
  echo "Something went wrong! Could not send the request to generate a report!" 1>&2
  # TEST
  curl --silent -XPOST "$POST_URL" -H "kbn-xsrf: kibana-reporting" -H "${AUTH_HEADER}"
  exit 1
fi

# 2. Log the path used to download the report
DOWNLOAD_PATH=${HOST}$DOWNLOAD_PATH
echo Download path: $DOWNLOAD_PATH

# 3. Wait for report execution to finish
echo While the report is executing in the Kibana server, the reporting service will return a 503 status code response.
STATUS=''
while [[ -z $STATUS || $STATUS =~ .*503.* ]]
do
  echo Waiting 5 seconds...
  sleep 5
  STATUS=$(curl --silent --head "$DOWNLOAD_PATH" -H "${AUTH_HEADER}" | head -1)
  if [[ -z "$STATUS" || $STATUS =~ .*500.* ]]; then
    echo "Something went wrong! Could not request the report execution status!" 1>&2
    curl "$DOWNLOAD_PATH" -H "${AUTH_HEADER}" 1>&2
    exit 1
  fi
  echo $STATUS
done

# 4. Download final report and show the contents in the console
curl -v "$DOWNLOAD_PATH" -H "$AUTH_HEADER"

Test that the above script from (2) works in 7.8, and continues to work after migrating to 7.17.

…owser (#148226) ## Summary This change allows developers on Mac to use the CSV generation APIs in 7.17. However, screenshot-type reports will continue to not be supported. In 7.17, there was no Kibana Reporting support for the Mac ARM processor. Now that many developers use that type of computer, they may be blocked from using any Reporting API at all, including CSV generation APIs. This change makes a correction to not block access to Reporting APIs when there is not registered headless browser. If a user attempts to generate a report in 7.17 on a server running an unsupported browser, they will receive an error message. Unblocks: #148030

Dosant

del

Dosant

I tested a bunch of different request scenarios and everything worked well. great job.

A scenario I was thinking of and didn't test is if Kibana was upgraded from <7.3 where we had this api, then to <7.17 where we removed it, and then to this branch where we have it again. Have you tested? Do you anticipate any issues with Kibana bootstrap?

jloleysens

LGTM, thanks for re-adding this functionality! Hopefully it can get some of our users out of any bind they may be in.

jloleysens · 2023-01-11T11:14:22Z

x-pack/plugins/reporting/server/export_types/csv_saved_object/execute_job.ts

+    const searchSourceFields: ParsedSearchSourceJSON = parseSearchSourceJSON(
+      savedSearch.attributes.kibanaSavedObjectMeta.searchSourceJSON
+    );


Think a bit ahead here. As I understand this team "owns" the "search source" object but not the "search" object?

When we have support for "down" and "on-read" migrations of saved objects we can consider pinning this object to a specific version of "search", but also a specific version of "searchsource" to ensure it is always in the shape we expect.

This is nothing to action on this PR, I'd be interested to hear what @pgayvallet or @rudolf think :)

tsullivan · 2023-01-11T18:07:59Z

@Dosant

A scenario I was thinking of and didn't test is if Kibana was upgraded from <7.3 where we had this api, then to <7.17 where we removed it, and then to this branch where we have it again. Have you tested?

I made an attempt to try this test, but I wasn't able to do so successfully. In 7.3-7.8, the API integration test was skipped, and the code seemed to be broken. I'm waiting for some information to find out if consumers that used this were using a fork of Kibana that had this fixed. When I find out, I plan to run this test case (after merging this PR) to make sure the upgrade is seamless for the consumers.

Do you anticipate any issues with Kibana bootstrap?

There are no new dependencies or changes to the build process in this PR. These changes were carefully done to ensure that nothing outside of the restored API can be affected. Let me know if that helps answer this question.

@jloleysens @pgayvallet @rudolf

When we have support for "down" and "on-read" migrations of saved objects we can consider pinning this object to a specific version of "search", but also a specific version of "searchsource" to ensure it is always in the shape we expect.

There are some tough choices that had to be made in this PR, since the feature is a combination of functionality from across different domains. In the 7.17 branch, those domains are not great at exposing the functionality that needs to be shared. As you see in the changes, that led to a compromise to copy code from other domains into x-pack/plugins/reporting/server/export_types/csv_saved_object.

I will draft a design document for a no-compromise implementation of this API. The design will propose having the having the appropriate domains share the functionality that is needed, and handle "on-read" migrations before sharing, which is what I think @jloleysens is suggesting.

On the topic of domains, I would also add that code in x-pack/plugins/reporting/server/export_types does not necessarily need to be part of the reporting plugin. These could be a separate plugin that depends on reporting and calls a method exposed by Reporting, such as reportingStart.registerExportType or something like that. That could help with ownership ambiguity by allowing other teams to maintain reporting code that is aligned with their domain.

tsullivan · 2023-01-11T21:00:45Z

I'm waiting for some information to find out if consumers that used this were using a fork of Kibana that had this fixed.

I got a patch to test running Kibana with, and tested the API in 7.7. That testing led to the latest fix, which would have been a bug present with older saved search objects.

kibana-ci · 2023-01-11T23:34:27Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`reporting`	159	161	+2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`reporting`	40.7KB	40.7KB	+81.0B

Unknown metric groups

API count

id	before	after	diff
`reporting`	162	164	+2

References to deprecated APIs

id	before	after	diff
`reporting`	26	76	+50

History

💚 Build #99412 succeeded a9a6000
💚 Build #99243 succeeded ff8fb9e
💚 Build #98548 succeeded 58864f4
💚 Build #98533 succeeded 6250b2ba69ca5bed2cbcc3d1ec88969c9f0b09c7
💚 Build #97479 succeeded 493f57ddc3d84fd5b3ba84460b8545db04f06374
💔 Build #97474 failed b44804c38dfd97d69166a3fd76dedc94bc8a91c5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

## Summary In 7.17.9, we're restoring a report generation API endpoint to create CSV reports based on saved searches. There has been one PR that serves as the first iteration: #148030 This resolves a missing capability from the first iteration. In the first iteration, POST bodies could allow an additional time range filter, which is merged with any saved filters or queries stored in the saved search object: ``` POST /api/reporting/v1/generate/csv/saved-object/search:${savedSearchId} { timerange: { min: '2015-09-20 10:23:36.052', max: '2015-09-20 10:25:55.744' } } ``` This PR is a second iteration. It allows additional "unsaved state" to be merged with the saved object at the time of report generation. ``` POST /api/reporting/v1/generate/csv/saved-object/search:${savedSearchId} state": { "query": { "multi_match": { "type": "best_fields", "query": "cognac", "lenient": true } }, "timerange": <same as before> ``` ``` POST /api/reporting/v1/generate/csv/saved-object/search:${savedSearchId} "state": { "query": [ { "multi_match": { "type": "best_fields", "query": "cognac", "lenient": true } }, { "bool": { "must_not": { "multi_match": { "type": "best_fields", "query": "Pyramidustries", "lenient": true } } } } ], timerange": <same as before> ``` ### Details In the details of #148030, it was stated: > Does not allow "raw state" to be merged with the Search object, as in the previous code (from 7.3-7.8). Otherwise, the API is compatible with the previous code. This PR pushes a bit back against that limitation. Now, requests to generate a report can accept a `state` field in the POST body. This field contains additional "unsaved state" that gets merged with the contents of the stored saved search object. However, the entire functionality of allowing `sort` and `docvalue_fields` keys in the request, is still not restored from the functionality that was implemented in 7.3-7.8. This limitation exists to minimize the complexity of the restored endpoint. Both of the non-restored keys are related to the sorting of documents. The sorting of documents is controlled by the saved search object only. The user can change the sort of the CSV after downloading the report, in a spreadsheet application or by programmatically working on the file. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

tsullivan force-pushed the reporting/restore-csv-by-savedobject branch 2 times, most recently from 08e6019 to 7b85537 Compare December 22, 2022 19:53

tsullivan changed the title ~~Reporting/restore csv by savedobject~~ [Reporting] Restore the "csv by savedobject Dec 22, 2022

tsullivan changed the title ~~[Reporting] Restore the "csv by savedobject~~ [Reporting] Restore the "csv by savedobject" endpoint for 7.17 Dec 22, 2022

tsullivan force-pushed the reporting/restore-csv-by-savedobject branch 6 times, most recently from b44804c to 493f57d Compare December 29, 2022 22:38

tsullivan mentioned this pull request Dec 29, 2022

[Reporting] Allow Reporting APIs to work when there is no headless browser #148226

Merged

tsullivan force-pushed the reporting/restore-csv-by-savedobject branch from 493f57d to 6250b2b Compare January 6, 2023 22:24

tsullivan added 17 commits January 7, 2023 09:11

partial code restoration

1419dc2

fix sort

a6bbbf4

fix time filter

88605c8

types and test improvement

01bcdd8

simplify conditionals

5f2afb7

fix time zone

283f2f8

update jest snapshot

84caf8a

consolidate the utc default

2f22d65

harden functional tests

4e96424

unit test stubs

7dfc354

add new saved search fixture

652b76b

test for terms filter

a6012ff

harden the no-filter test

96c2162

Allow no post body in the request

8ff68e7

More tests

4ee90f0

add some jest tests

1c89cd8

remove oops

7a60a6c

tsullivan added 5 commits January 7, 2023 09:11

fix telemetry api test

ab695fe

fix telemetry attempt ii

8cf100d

polish

45c6ccd

more tests

a617923

fix usage collection snapshot

58864f4

tsullivan force-pushed the reporting/restore-csv-by-savedobject branch from 6250b2b to 58864f4 Compare January 7, 2023 16:13

tsullivan marked this pull request as ready for review January 7, 2023 16:13

tsullivan added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:skip Skip the PR/issue when compiling release notes Team:SharedUX Team label for AppEx-SharedUX (formerly Global Experience) v7.17.9 labels Jan 7, 2023

tsullivan and others added 3 commits January 10, 2023 09:03

Merge branch '7.17' into reporting/restore-csv-by-savedobject

ff8fb9e

unit test for execute_job

5fc5af7

enable snapshot filtering based on ES version

a9a6000

tsullivan requested review from a team, dokmic and jloleysens January 11, 2023 00:50

Dosant reviewed Jan 11, 2023

View reviewed changes

Dosant approved these changes Jan 11, 2023

View reviewed changes

jloleysens approved these changes Jan 11, 2023

View reviewed changes

export all fields if there is one column named "_source"

bb67608

tsullivan merged commit 7fea0ad into elastic:7.17 Jan 11, 2023

tsullivan deleted the reporting/restore-csv-by-savedobject branch January 11, 2023 23:36

tsullivan mentioned this pull request Jan 12, 2023

[Reporting] API endpoint for exporting CSV from Saved Search ID #148775

Closed

tsullivan mentioned this pull request Jan 23, 2023

[Reporting/CSV/7.17] allow query from state in parameters #149297

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting] Restore the "csv by savedobject" endpoint for 7.17 #148030

[Reporting] Restore the "csv by savedobject" endpoint for 7.17 #148030

tsullivan commented Dec 22, 2022 •

edited

Loading

Dosant left a comment •

edited

Loading

Dosant left a comment

jloleysens left a comment

jloleysens Jan 11, 2023

tsullivan commented Jan 11, 2023

tsullivan commented Jan 11, 2023 •

edited

Loading

kibana-ci commented Jan 11, 2023

API count

References to deprecated APIs

[Reporting] Restore the "csv by savedobject" endpoint for 7.17 #148030

[Reporting] Restore the "csv by savedobject" endpoint for 7.17 #148030

Conversation

tsullivan commented Dec 22, 2022 • edited Loading

Summary

Testing

Dosant left a comment • edited Loading

Choose a reason for hiding this comment

Dosant left a comment

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

jloleysens Jan 11, 2023

Choose a reason for hiding this comment

tsullivan commented Jan 11, 2023

tsullivan commented Jan 11, 2023 • edited Loading

kibana-ci commented Jan 11, 2023

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Page load bundle

API count

References to deprecated APIs

History

tsullivan commented Dec 22, 2022 •

edited

Loading

Dosant left a comment •

edited

Loading

tsullivan commented Jan 11, 2023 •

edited

Loading