Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Data Visualizer] timestamp doesn't support Kibana CSV export format #55586

Closed
blookot opened this issue Apr 22, 2020 · 3 comments · Fixed by #55609
Closed

[Data Visualizer] timestamp doesn't support Kibana CSV export format #55586

blookot opened this issue Apr 22, 2020 · 3 comments · Fixed by #55609
Assignees
Labels
:ml Machine learning

Comments

@blookot
Copy link

blookot commented Apr 22, 2020

Describe the feature: Add the format of Kibana CSV export in data visualizer

I suggest we add MMM dd, yyyy @ HH:mm:ss.SSS to the list of supported timestamp formats in the Data Visualizer File import.

So that when you export a CSV from Kibana you can reimport it via Kibana ;-)

Thank you!

@blookot blookot added the :ml Machine learning label Apr 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

droberts195 added a commit to droberts195/elasticsearch that referenced this issue Apr 22, 2020
The Kibana CSV export feature uses a non-standard timestamp format.
This change adds it to the formats the find_file_structure endpoint
recognizes out-of-the-box, to make round-tripping data from Kibana
back to Kibana via CSV files easier.

Fixes elastic#55586
@droberts195
Copy link
Contributor

@blookot since the basic ask was so simple I've added it for 7.8 in #55609. The initial file analysis now finds the time format.

However, I found that round-tripping a file through the Discover view's CSV export back to ML's file upload still required a bit of manual intervention because the export contains fields _id and _index which are not permitted in ingested documents, so these need to be renamed in the column name overrides.

So there's still a question about what we should do when importing a file that already has _id and/or _index fields.

@droberts195 droberts195 self-assigned this Apr 22, 2020
droberts195 added a commit that referenced this issue Apr 22, 2020
…55609)

The Kibana CSV export feature uses a non-standard timestamp format.
This change adds it to the formats the find_file_structure endpoint
recognizes out-of-the-box, to make round-tripping data from Kibana
back to Kibana via CSV files easier.

Fixes #55586
@blookot
Copy link
Author

blookot commented Apr 23, 2020

Thank you @droberts195 !
just a quick comment: the way we usually export CSV is by creating a saved search having only a few columns selected (out of all the fields), so _id and _index are not exported.
Still there is an issue on numeric fields export, see elastic/kibana#64170
But indeed, my goal is to easily export in csv and share the csv for others to reimport via data visualizer ;-)

droberts195 added a commit that referenced this issue Apr 23, 2020
…55609)

The Kibana CSV export feature uses a non-standard timestamp format.
This change adds it to the formats the find_file_structure endpoint
recognizes out-of-the-box, to make round-tripping data from Kibana
back to Kibana via CSV files easier.

Fixes #55586
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:ml Machine learning
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants