[Logs UI] [WIP] Prompt for a data view when log source not configured

[weltenwort]

⚠️ This is work in progress!

📝 Summary

Even though log sources can be configured based on a data view in #92650, the default is still an index name reference. Because we can't assume the existence of any specific data view, the Logs UI can't work without an explicit choice being made by the user. This aims to introduce a quick data view selection prompt when the user visits an unconfigured log source.

part of #120920

✔️ Acceptance criteria

• When the user visits the Logs UI with an existing source configuration of either style, the Logs UI immediately loads as before.
• When the user visits the Logs UI without an existing source configuration, a prompt is displayed in which the user has to choose a data view before proceeding.
• The choice causes the source configuration to be persisted using the selected index pattern reference.
• After the source configuration has been persisted the prompt disappears and the Logs UI continues to behave as before.

🎨 Mock-ups

⚠️ These mock-ups are not meant to be prescriptive, but to set the ACs into context.

🔭 Outlook

Once a data view creation flyout as described in #67711 exists, it could be used here to avoid the need to leave the Logs UI when lacking a data view.

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[miltonhultgren]

Notes from discussion with @weltenwort

Background:

This issue was to be the last step in our planned migration to only using Data Views to configure the Logs UI, deprecating the options to use plain index names. However, we received feedback from users that this would create a barrier to usage due to the fact that you need more permissions to create a Data View (in the case that one hasn't already been used). This means that the usually read only Analyst role would have to rely on someone with higher level permissions to setup the UI for them, an admin user which might not be likely to enter the Logs UI per space that is being created to configure the source.

As a result of this we're back peddling a bit on this plan. We will continue to support the plain index names option (which plays well with our embeddable/internal source configuration strategy). However, we still want to nudge user into the path of using Data Views since it offers us benefits such as linking into other Kibana apps as well as creating column from Runtime Fields.

Next steps:

1. Add support for opting out of Data Views source configuration
Based on #115737 we now allow (and encourage) users to opt into using Data Views for their source configuration, but once they opt in there is no way to opt back out for the given space. If we plan to continue supporting plain index names configuration it makes sense to allow the user to swap back to that option in the Settings page.
See #120082

2. Add dialog to inform users about configuration options
To nudge users into using Data Views we want to make them aware that they should configure their source rather than relying on the implicit default (an in memory stub with some default index names, not stored as a saved object).
This will likely take two paths, one for the Write privilege user and one for the Read privilege user.

a) For the Write privilege user
If the page detects the stub configuration (origin: "fallback") we show a dialog that requires them to configure the source by either using a Data View (recommended) or plain index names. This can be done with the same UI component as found in the settings page if that makes sense. The config is then saved as a Saved Object which can be edited in the Settings page.

b) For the Read privilege user
If the page detects the stub configuration (origin: "fallback") we show a dialog that informs the user that this space needs to be configured by an admin, showing the fallback configuration we will use until then and giving them the option to "Continue with temporary settings". In the same dialog we can show the configuration form we would show the admin with our defaults filled in but leaving the form disabled, just as a reference picture for what the admin should look for.

Once Step 1 is complete, we will revisit this issue to update the ACs in line with Step 2.

Future plans:

In the future we want to integrate the Data View creation flyout in the component used to configure the Source Configuration, to reduce the number of page links out of Logs UI to configure it.

At some point we also need to consider how the "prefer Data Views" path affects our embeddable use cases. Today they can specify the config by an ID but they might want options to specify it with plain index names or Data Views to allow further linking into other Kibana apps. We also need to then solve for the use case of upgrading a source configuration to affect both the embeddable and the log stream UI. This might also be solved by using Transient Data Views once they land.

We also need to consider the future where we might have per user configs in some kind of middle state before it is saved. Both for users who are trying things out before committing the changes, but also for Read privilege users that still want to modify their own experience locally.
This also ties into the future of Saved Views where we need to support multiple Source Configurations per space.

Finally, there was an idea of creating a workflow where a Read privilege could submit a Data View creation request for Write privilege user approval, allowing the Read user to define the configuration as they want but using the Write user's credentials to execute the request.

More info here #119887

[miltonhultgren]

One more note on this, an option that we can consider is to use the Kibana system user to create a default Data View per space. This is what both APM and the Security solution does to make at least one Data View available for usage, the user can still create their own if they have the right permissions.
APM has gotten some feedback on that this is annoying since the admins find Data Views they did not create and don't understand the source of, which they delete and then APM creates them again on page load.

Perhaps it's about naming of the Data View and of informing the user that this has been created due to its absence.

[miltonhultgren]

A note on the Saved Object that gets saved: When creating a Source Configuration, either via the Metircs UI or the Logs UI the Saved Obejct that is stored contains configuration for both apps, using the defaults for the "other" app.
So if you create your configuration in the Logs UI, we include the defaults for the Metrics UI, and vice versa.
It may be desired to split these into two individual Saved Objects.

This means that if we wanted to prompt for setup in the Metrics UI, but the Logs UI has already been configured we would not be able to detect the "fallback" state anymore. Yet the user has not explicitly configured the Metrics UI.

We can keep supporting reading from the old shared Saved Object, and only change so that we write to new split Saves Objects.

[jasonrhodes]

@weltenwort / @miltonhultgren should we close this ticket in lieu of the other source config work now being started?

[weltenwort]

No, it's step 2 of the proposed task breakdown in the epic.

[miltonhultgren]

We should likely show this prompt on all Logs UI pages so that the user does the configuration before using any page.
Since the prompt will ask the user the same info that is put into the Settings page it would make sense to also show this on the Settings page but we're likely to remove the Settings page as we lift things out from it into the Log Stream page as per #124011.

[weltenwort]

The ACs intentionally don't limit this to only the logs stream page. There's probably not harm in showing it on all logs UI pages.

[smith]

Updated description and title to say "data view" instead of "KIP."

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)