[GS] Feature results provider

[alexfrancoeur]

Building off of the work done with #65222 and #65633 I'd like to track the need to add search results that are one level deeper than applications.

We have a number of features that fall under larger applications that I believe should be easily accessible through search. Some examples include Index Management. When searching for Management, I believe you will only see the overarching stack management application and not any of the features related to it. The same could be said for our Security solution for example. Maybe a user wants to navigate directly to Cases?

We currently have a feature directory that's main purpose today is to populate the home page. There are a number of features missing, but as @cqliu1 suggested, we could potentially repurpose this to provide more granular search results for some of the larger applications with key features associated with them.

Feature mapping

Below you’ll find an initial stab at features that may be beneficial to return as part of the Kibana’s new navigational search.

Global

• Home

Kibana

• Not many new additions, most views are populated by applications in navigation (and search results)
• Machine Learning
  • Overview (same entry point as Machine Learning app)
  • Anomaly Detection
  • Data Frame Analytics
  • Data Visualizer
  • Settings

Enterprise Search

• No new additions, all views are populated by applications in navigation (and search results)

Observability

Overview

• No new additions, all views are populated by applications in navigation (and search results)

Logs

• Stream (same entry point as Logs app)
• Anomalies
• Categories
• Settings

Metrics

• Inventory (same entry point as Metrics app)
• Metrics Explorer
• Settings

APM

• Services (same entry point as APM app)
• Traces
• Service Map

Uptime

• Overview
• Settings

Security

Overview

• No new additions, all views are populated by applications in navigation (and search results)

Detections

• Detected signals (same entry point as Detections app)
• External Alerts
• Manage signal detection rules

Hosts

• All hosts (same entry point as Hosts app)
• Authentications
• Uncommon processes
• Anomalies
• Events
• External alerts

Network

• Flows (same entry point as Network app)
• DNS
• HTTP
• TLS
• Anomalies
• External alerts

Timelines

• Timeline (same entry point as Timeline app)
• Templates

Cases

• Cases (same entry point as Cases app)

Administration

• No new additions, all views are populated by applications in navigation (and search results)

Management

Dev tools

• Console (same entry point as Dev Tools app)
• Search profiler
• Grok Debugger
• Painless Lab

Ingest Manager

• Overview (same entry point as Ingest Manager app)
• Integrations
• Configurations
• Fleet
• Datasets
• Add agent

Stack Monitoring

• Elasticsearch
  • Overview
  • Nodes
  • Indices
  • Logs
• Kibana
  • Overview
  • Instances
• Beats
  • Overview
  • Beats
• Logstash
  • Overview
  • Nodes
  • Pipelines
• APM
  • Overview
  • APM servers

Stack Management

• Ingest
  • Ingest node pipelines
  • Logstash pipelines
  • Beats Central Management
• Data
  • Index Management
    • Indices
    • Data streams
    • Index templates
    • Component templates
  • Index Lifecycle Policies
  • Snapshot and Restore
    • Snapshots
    • Repositories
    • Policies
    • Restore status
  • Rollup Jobs
  • Transforms
  • Cross-Cluster Replication
    • Follower indices
    • Auto-follow patterns
  • Remote Clusters
• Alerts and Insights
  • Alerts and Actions
    • Alerts
    • Connectors
  • Reporting
  • Machine Learning Jobs
    • Anomaly detection
    • Analytics
  • Watcher
• Security
  • Users
  • Roles
  • API Keys
  • Role Mappings
• Kibana
  • Index Patterns
  • Saved Objects
  • Spaces
  • Advanced Settings
• Stack
  • License Management
  • 9.0 Upgrade Assistant

Open questions

• How granular do we want this feature results provider to be? Do we want to make the search results "actionable" navigating directly to CRUD views. For example, rather than simply navigation to cases we could navigate to "Create new case". It definitely streamlines navigation, but unsure if that level of granularity is useful. For those trying to reduce as many clicks as possible - it might be
• If we don't have feature controls for a number of these items, are they worth showing in search? I don't know if we want to streamline navigation to an error page
• Is there a programmatic way to get most of these features / sub-features or do teams need to manually register new ones? If we're adding a description to each, I'd imagine it's manual

[elasticmachine]

Pinging @elastic/kibana-core-ui (Team:Core UI)

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[alexfrancoeur]

I updated the original description to include mappings of in-app features that might be useful to return as part of Kibana's navigational search. This is basically a manual mapping of what's available today in 7.x and would love some feedback from the different teams responsible for these areas. When we're ready to start working on this one, I think a good next step will be to engage with the different solution and management teams to make sure this is a good starting point.

[ryankeairns]

While it doesn't belong to a specific category, I think it would be useful to add the Home page as well (I'll add to the description)

[joshdover]

To answer the "where should we capture this data" question, I think it makes sense at this point to consolidate the feature catalog registry with app registration. They're intrinsically linked and it would allow the application GS provider to use the same data source for both sets of items.

One tricky aspect is that for applications like Stack Management and Dev Tools which allow other plugins to register sections dynamically, not all sub-features will be known at setup. This may require an Observable-based API like:

core.application.register({
  id: 'management',
  features$, // management plugin would update this as new management apps are registered
  async mount() { ... },
});

The ApplicationService's applications$ Observable should be able to hide this detail by unwrapping this Observable and producing a single Observable of a static structure.

Later, we can adapt the feature directory page to use this data as well and then do away with the old feature catalog registry. The feature catalog registry has confused many a Kibana developer, so that seems like an additional win to me.

[ryankeairns]

Adding to that, @alexfrancoeur has been pushing for removal of the directory page altogether. It is no longer up to date, for one, and there are suspicions that it is used infrequently. We need to confirm, but that is likely the direction we are headed with that page.

[joshdover]

Josh to investigate what data we have in FeatureCatalogue and if it meets requirements for this feature

[tbragin]

@alexfrancoeur +1 on this enhancement from the observability perspective.

[ryankeairns]

Dropping a thought for us to discuss once this work is picked up...

This issues covers a lot of items, all with great potential. That said, we will only be able to tackle a subset of these items for 7.11. Perhaps the Feature directory proves to be the most attainable, but the Stack Management side menu items would also make for nice set of universally used content.

Looking forward to chatting further.

[joshdover]

@ryankeairns Take a look at #83380 and let me know what you think. I've added a generic option where plugins can register new 'subLinks' for their application which will automatically be picked up by search.

I decided against using the feature catalog/directory data since it is mostly out of date anyway and would require more work to integrate with the existing app results. Seemed to make sense to just build the 'ideal' pattern separately and then work on integrating the apps with it. It should be fairly trivial for apps to now add new locations into search with this pattern.

In this initial PR I've already integrated the Stack Management sections with search and things seem to be working nicely.

[alexfrancoeur]

I decided against using the feature catalog/directory data since it is mostly out of date anyway and would require more work to integrate with the existing app results.

That makes sense to me. @ryankeairns one thing we discussed earlier is that we can remove the app directory link from the home page and Kibana overview page once the feature results provider is in place. With these changes, are we good to remove the link? I realize the home page is powered by it at the moment, so I'm fine with simply removing the link from the home page.

[ryankeairns]

I think it's safe to remove the link, yeah. Between what we have now and what will come from the Stack Management sub links... I don't see anything else very useful in the directory list.

[alexfrancoeur]

++ agreed, is that a change your team could make? Or should we just couple with @joshdover's PR?

[ryankeairns]

++ agreed, is that a change your team could make? Or should we just couple with @joshdover's PR?

Kibana Design can remove it. I'll create an issue and open a PR.

[joshdover]

With the initial work in place in #83380 and Stack Management already integrated in that PR, these are the remaining areas that need to integrate with this feature. This should be quite simple and in most cases should be just a matter of adding a searchDeepLinks array to your core.application.register call. You can see the documentation and included example here: https://github.com/elastic/kibana/blob/master/docs/development/core/public/kibana-plugin-core-public.app.searchdeeplinks.md

EDIT: I've opened separate issues for these teams linked to this issue below.

• Observability - @jasonrhodes
• Security - @XavierM
• Stack Monitoring - @chrisronline
• Machine Learning - @peteharverson
• Fleet / Ingest - @jen-huang
• Dev Tools - @cjcenizal
  • Since this one is dynamic, depending on which dev tools are enabled (OSS vs. X-Pack) this one will need to leverage the appUpdater$ API similar to how management does this here: https://github.com/elastic/kibana/blob/master/src/plugins/management/public/plugin.ts#L51

Ideally, we'd like to get this all in for 7.11. In theory, this should be very simple. If I can be of any assistance please do let me know.

[alexfrancoeur]

❤️ thanks Josh! @tbragin @hmnichols - if we're able to add support across solutions, we can quickly navigate to sub-links within solution applications. Making discoverability and MTTR that much faster. For example, with Observability, that means cmd+\ and typing in traces could navigate you directly to the Traces tab in APM.

[cjcenizal]

@joshdover This is awesome! Great work man! Full transparency though: ES UI doesn't have the capacity to work on this for 7.11. But we can budget time for it in 7.12.

[alexfrancoeur]

@joshdover with FF around the corner and the short week this week, are the plans still to add support for other applications in the next release?

[joshdover]

@alexfrancoeur I think links in other apps will have to be delayed. I just spent some time looking at what it would take to add it to a few apps myself and it's actually a bit more work than I was thinking. The main reason is that we need to make sure to keep the deep links in sync with the actual routing code which would require reorganizing code in several plugins to make sure the route paths & translate labels are the same. This reorganization needs to be done carefully so that we don't end up pulling in a lot of code into the main plugin bundles.

Luckily we got one of the most useful ones in for 7.11: Stack Management.

I think I will close this an open a new issue for each team to add their appropriate deep links as listed in the top of this issue. Does that work for you?

[joshdover]

I've opened issues for the individual teams that need to integrate with this feature. Closing this!

[alexfrancoeur]

Woo!