[skip-ci] [Discuss] Management apps organization

[cjcenizal]

Problem statement

• Our apps map poorly to the existing buckets in Management, forcing apps to be shoe-horned in places where they don't quite fit. For example, Upgrade Assistant is a tool for upgrading your entire stack, not just Elasticsearch.
• The existing buckets don't guide a user towards the app they're looking for. People don't come into Management thinking "I want to do something with Kibana" and then says, "Ah yes, Kibana Index Patterns are what I was looking for!" People come with a problem in mind ("I need an alert when my data changes"), looking for a solution ("Alerting!"). Bringing me to the third point...
• Sometimes, a single problem ("I need an alert when my data changes") has multiple solutions (Alerting, Watcher). But the existing buckets force these solutions apart, making it difficult for users to identify and compare similar options.
• The existing buckets also force a very uneven distribution of apps. Beats, Logstash, and Machine Learning each contain a single app, while Elasticsearch contains 10.

Proposed solution

Let's use the navigation to create a mental model for finding solutions based on the problem a user wants to solve. And let's organize these problems along the spectrum of answering questions about data. When thinking about data, users ask these kinds of questions:

• Ingest. How is data entering the cluster?
• Cluster. Once it's ingested, what is the data's state at any given point in time? And how can we control that state over time?
• Alerting. How can we detect changes in and glean insights from the data?
• Access. How do we control who can see the data? (Renamed from Security to avoid confusion with SIEM).

The remaining apps that don't fall into these buckets relate to either:

• Kibana. Managing Kibana as an app.
• Stack. Managing the Stack as a whole.

Implementation

The image below shows three columns. On the left is the management navigation as it appears today. On the right, we have our desired end-state, which will combine/modify our applications and buckets them based on what the user wants to do. In the middle is an intermediate state in which we bucket our applications as they exist today around what the user wants to do.

The intermediate step shows us moving our existing apps into these buckets without making any major changes to the apps themselves, like renaming or combining them.

From there, we can gradually start combining apps and breaking them apart, as we work towards the desired end-state:

• Extracting Index Templates from Index Management, renaming Index Management to Data Sources, and moving Index Patterns into Data Sources (because people creating Index Patterns will want to know which indices they're capturing and we can easily cross-link).
• Combining Index Lifecycle Policies, Rollup Jobs, and Snapshot and Restore into a single Scheduling app.
• Combining Cross-Cluster Replication and Remote Clusters into a single Cross-Cluster app so that people who want to set up CCS will gravitate there to set up the remote cluster necessary for CCS.

Benefits and opportunities

Evolving buckets into apps

Aside from giving people a structure to guide them towards the tools they're looking for, this new grouping also creates the possibility for evolving the buckets into specific apps, e.g. an "Ingest" app.

By creating an app that's focused on a specific type of problem, we can create the opportunity for tailored onboarding flows, e.g. getting started with data ingestion when you have no data, or getting started with lifecycle management or snapshots (or proposing moving the data to Cloud) once you have a lot of data.

One possible drawback to this is that in many companies there may only be a handful of stack managers. These individuals will use the full range of management apps, from configuring Beats to creating ILM policies. Breaking Management up into separate apps might impede app discovery and cross-navigation.

Opportunities for new apps

Grouping apps the way that's been proposed opens the way for adding new apps that solve the unaddressed aspects of the problem that the bucket is focused on. For example, a "Cluster" bucket gives us a place to add diagnostic tools for checking on cluster health, or a Tasks management app.

Considerations

• How do these buckets relate to our subscription matrix? Will a user be able to associate these buckets and apps with the buckets and features in the matrix?
• Do we need a management landing page? What would go on this page? Possibly a dashboard or digest of all of the buckets/apps?
• Can we use feature control to allow admins to curate the management experience?

[VijayDoshi]

I love the idea of organizing around tasks or "jobs" - from an end user's point of view it resonates much more than a product name centric model (IMO). We'll need to go through the details on wordage and groupings if we move forward; however, in general I am in support of this direction. I'd love to see further exploration and ideally some initial user testing - find some admins and get their feedback.

[arisonl]

These are great ideas and approach to explore. One parameter to factor in as we are thinking this approach is the design's interaction with Spaces. Some of the functionalities are space-specific, e.g. what is currently under the Kibana umbrella, while others are global, e.g. what is currently under Elasticsearch. While not ideally, this gets reflected in the current buckets (but it should not necessarily drive the design, just maybe inform it). As an FYI, we also want to potentially make some of the space-specific functionality accessible/manageable globally, e.g. admins being able to manage all saved objects or alerts from one place across spaces. Here is a related discussion that might be relevant: #37285. And of course, the menu is dynamic: for the most part, you see listed the functionalities that your roles have access to.

[gchaps]

I really like the idea of reorganizing our management apps based on the task that users perform. Also, the idea of creating onboarding flows. This also works well with the task-based approach we are moving toward in the docs.

Another benefit of the proposed structure is that it is so much easier to scan. By using more plain English terms, users can get to where they need to go faster.

I agree with what @VijayDoshi says about thinking through the details of wording and groupings. Always a difficult thing to do. For example, to me "Security" is more clear than Access. Also, Machine Learning might work better grouped with the other visualization apps and not be part of this at all.

I think the Cluster section will be tricky. Data Sources and Scheduling are pretty generic terms. For Snapshot and Restore, I was expecting to see something more along the lines of "backups".

Overall, though, I think this is a good direction for both the apps and docs.

[peteharverson]

In general I like the changes proposed here, grouping the items around the type of action the user wants to perform on their data.

Echoing the comments above, some of the wordings for the groups might need some more thought. Specifically, looking at the proposed group for Machine Learning, might a heading of 'Insight' work better than 'Alerting'? ML and alerts are both ways of getting insights, but not the other way round. 'Insight' would also work well for Reports.

The group that the Transforms item is in makes sense to me. The key thing is it belongs in the same group as 'Rollup jobs' (or the 'Scheduling' item in the second iteration).

[mostlyjason]

I also like this idea of organizing by the job users want to perform. I think it helps with discoverability because they will see there are sometimes multiple ways to accomplish a job (logstash or ingest node, watcher or alerts). Over the long term, it'd be nice to consolidate these into a single experience so users don't have to guess which to use.

[ppf2]

This is a great idea :)

I think it can be helpful to provide a listing of the items under these new "groupings" in some kind of mouseover dialog so users can quickly see what items are covered under the main headings (without having to go to another page, or do some kind of expansion to get to them, etc..), and have a way to quickly navigate to these specific items from the mouseover dialog.

For example, users may not immediately know to click on "Scheduling" for snapshot/restore and ILM.

[sebelga]

I think it is a great initiative to improve how apps are organized! Thanks for opening the discussion @cjcenizal !

I do have some concerns about Index management being removed. I can't tell on the right column where I should go to manage and see my indices.
For me, index management should grow and combine other apps. A user that wants to manage his indices would go to a single place to do it all (Create index, index templates, ILM, CCR).

For example: CCR and Index management. I currently find it strange to create an index in CCR. I would like CCR to be merged into Index management:

• User click "Create index"
• In the index creation form there is a toggle on top: "Replicate from remote cluster". If activated, the form changes to the current Follower index form.
• No need to have a separate table for follower indices

I would then keep the Index Templates app inside Index Management and have 2 more apps added to this super app: "Auto-follow patterns", "ILM"

For example, users may not immediately know to click on "Scheduling" for snapshot/restore and ILM.

I also think that Snapshot and restore should not be under "Scheduling" as we can manually trigger a snapshot. It would belong more to a "Documents (or Data)" section that has a subsection "Snapshots", another subsection "Restore".

Although having a "Scheduled" widget somewhere that gathers everything being scheduled (a snapshot, indices about to change lifecycle phase...) would be quite awesome I admit 😊

So I would group more by domains of action than what type of action (recurring, scheduling) the user wants to do.

[mattkime]

Overall I really like this idea, some nit picks -

I think the name Data Sources is a little too abstract, it would be better to mention Indices.

I do like the idea of combining Index Management and Index Patterns although Index Patterns is going to go through considerable changes. But thats something we can deal with when appropriate.

[cjcenizal]

Closing in favor of #65796. If you have any feedback or would like to raise any concerns please share your thoughts there.