[Sample Data] Update sample log data to include event.dataset values

[jasonrhodes]

The sample data that exists for sample logs currently is not ECS-compliant, but it would help in the logs UI if it at least had the event.dataset value filled in. We should adjust the values, mappings, and tests, to include a selection of dataset values for partitioning, etc.

https://github.com/elastic/kibana/tree/cda3de60aeedb18641c8132b2423a31dcf025d8d/src/legacy/server/sample_data/data_sets/logs

AC:

• Ingested sample log data includes 4-6 different event.dataset values, some much more frequently represented than others

Note:

• Stretch goal: adjust the rate of logs that are ingested so they represent a few "anomalous" bursts or lags (not sure if this is easy, so if it's not, it's not necessary right now).

[elasticmachine]

Pinging @elastic/infra-logs-ui (Team:infra-logs-ui)

[jasonrhodes]

@nreese can you provide any guidance on possible pitfalls around editing the JSON file for the sample log data?

[nreese]

There are several open issues regarding logs sample and ECS

#40397

#40401

I think it would be really useful to redesign the logs sample data set to be in ECS and address all of these issues.

The only pitfalls will be making sure the visualizations, canvas workpads, and maps saved objects are all updated to reflect the new format.

@alexfrancoeur, maybe we should replace the web logs sample data set with ones created from your filebeat-ecs.py script?

[jasonrhodes]

Makes sense. For 7.5 though, logs UI really needs the dataset value. I'm not sure what the timeline would be if we rollin the entire ECS compliance upgrade. :)

[alexfrancoeur]

Longterm, I'd like for at least one of the data sets to be "ECS compliant". Web logs makes the most sense. As Nathan mentioned, we'd need to update all saved objects associated with that demo set as well. If there are certain fields that could just be added to the existing logs, we could do that, but it's not really an ideal experience. I think there are a few options on the table.

• If something is required for 7.5, the quickest way to support this is to augment the existing data set to include new fields required by the logs UI. If this works, feel free to update the existing ndjson file to include these fields. This is the most low touch approach as no dashboards, visualizations, workpads, saved searches and maps need to be updated. The index pattern would though.
• If we wanted to do things right for 7.5, this would require updating the data set entirely to be ECS compliant. In order to do this correctly, we'd have to update all of the saved objects associated with the sample data. I don't believe we have anyone currently focusing on sample data so we'd have to determine an owner here, likely competing with other priorities
• If you were willing to wait a bit longer, it might make sense to wait for the integrations project to be merged in beta ([EPM] Elastic Package Manager Plugin - Tracking #36708). It looks like you opened that issue so I assume you're involved there 😄 I believe these integrations packages are meant to come with sample data as well. Is that still the case?

What do you think @jasonrhodes? How much work are you willing to take on? We've had other teams update the sample data sets in the past, is that something you'd be willing to do?

[tbragin]

@alexfrancoeur @jasonrhodes I'm fine just adding event.dataset to existing log set. It will improve how this looks currently in the Logs UI, and I consider having it pretty much a blocker to shipping our Logs+ML integration in 7.5. Without that value, having sample data really confuses the analysis. Is this something we can fast-track to add for FF today?

[alexfrancoeur]

That works for me. @jasonrhodes feel free to tweak the data set to include the event.dataset field, mapping and values. That shouldn't affect any of the existing saved objects but we should make sure they don't break. I think there are some relevant PR's for modifying the sample data set itself (#36982 - this might just be adding ml jobs actually) as well as adding a navigation link directly to your app (#36702).

[tbragin]

Sounds good. I propose we set it to a single value to match the dataset name: sample_web_logs