[Reporting] Inconsistent output between Export CSV from Discover, and Download CSV from Dashboard

[tsullivan]

Kibana version: 7.x

Terms:

• "Download CSV" - synchronous (keep the request open) way to download saved search data from a dashboard panel.
• "Export CSV" - asynchronous (using the reporting framework) way to generate a CSV export and download it later from the Reporting API

Describe the bug:
Download CSV

• BOM character is not prepended
• Max length warnings are not logged or shown
• Bug with showing/formatting the timefield column
• Lucene syntax bug

Export CSV

• Timezone bug - tz should be a param

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

I plan to work on refactoring the CSV export types to resolve the inconsistent issues, and also prepare the code for removing "Download CSV" as an export type. The reason to do that is as we move to task manager, we don't want the complexity of sync and async exports, we just want them to all be async and use the reporting framework (create job / execute job)

[tsullivan]

Bug with showing/formatting the timefield column

Example: For Metricbeats data, Export CSV output looks like:

"@timestamp","event.dataset"
"June 9th 2020, 11:49:59 pm","DOCKER.DISKIO"
"June 9th 2020, 11:49:59 pm","DOCKER.DISKIO"
"June 9th 2020, 11:49:59 pm","DOCKER.DISKIO"
"June 9th 2020, 11:49:59 pm","DOCKER.DISKIO"
"June 9th 2020, 11:49:59 pm","DOCKER.DISKIO"

Download CSV output looks like this:

"@timestamp","event.dataset"
"[""2020-06-09T21:48:29.772Z"",""2020-06-09T21:48:29.772Z""]","docker.diskio"

[fbaligand]

@tsullivan
I add that "Download CSV" generates "Raw data" export, and "Export CSV" generates "Formated Data".
That is particularly important for numbers, where a custom locale is chosen in advanced settings.

[tsullivan]

Thank you @fbaligand we will work on consistency for the formatting so that both CSV methods show formatted data, and matches what is shown in the UI

[fbaligand]

Great @tsullivan !

[fbaligand]

Hi @tsullivan
I found another difference between "Export CSV" and "Download CSV":
in "Download CSV", the advanced setting "doc_table:hideTimeColumn" is ignored, although it is processed in "Export CSV".

[tsullivan]

Replaced this issue with: #34377

[tsullivan]

Oops, meant to close this.

[maggieghamry]

@tsullivan it appears this issue is still present in 7.9.2, and presumably other 7.9 issues as well. Setting the advanced Kibana settings to "Browser" does not appear to be respected by the CSV reports:

In this example, where the default time zone is EST:

The resulting report does not reflect the time stamp:

Setting the advanced time zone to EST and refreshing does appear to be a workaround for a forced timezone:

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)