Generic UI search API #61657
Comments
joshdover
added
Team:Core
|
Pinging @elastic/kibana-platform (Team:Platform) |
|
Pinging @elastic/kibana-core-ui (Team:Core UI) |
|
Regarding global architecture (actual required savedObjects dataSource(s) implementation will have it's own discussion) dataSource registration and architecture
In an ideal world, However, we already know that some searchable objects ( The first consequence is that we will need to allow A corollary from this first statement is that searching from the server-side will not provide all results, as it would not be possible to 'query' the client-side for client-side-registered sources. We got an architectural decision to make here: Option A: only allow to register dataSources from the client-sideData providers would only be able to register dataSources from the client-side. If the datasource requires to perform calls to the server-side to retrieve their results, it will be considered an implementation detail. Pros:
Cons:
Option B: allow to register dataSources from both client-side and server-sideAs most of the currently identified datasources (AFAIK, all of them except the Pros:
Cons:
I'm leaning to option B, but it's not a strong opinion, and would really like to have the other's vision on that. POC interface for the dataSource providersThis is a base, naive contract to open the discussion, please comment on that. // are fields missing here?
interface GlobalSearchResult {
// the main text/title to display in search results UI
title: string;
// the type of result. also displayed as line2 in search result UI
// I.E `dashboard`, `application`
type: string;
// EUI icon name to display for the search result.
// Optional, will display a default (or no icon?) if not provided
icon?: string;
// The url to navigate to this result
url: string;
// optional, priority/ordering of the result
order?: number;
}
interface GlobalSearchDataSourceResponse {
success: boolean;
// only if success == true
results?: GlobalSearchResult[];
// only if success == false
error?: string;
}
// type instead of just a `(term: string, options: {}): Promise<GlobalSearchDataSourceResponse>` function
// to allow potential addition to the `dataSource` contract in the future.
type GlobalSearchDataSource = {
search(term: string, options: {}): Promise<GlobalSearchDataSourceResponse>;
};
// would be used like
coreSetup.globalSearch.addDataSource(myDataSource);Remarks / Questions:
In current POC implem, I just added a In stage 1 of GS, we will only have objects contained in the kibana instance we are searching for, however in next stage we will have to display result from other clusters' kibana instances, and therefor be able to redirect to an url outside of current kibana's domain. Navigating to an internal object should probably be performed using Would something like interface GlobalSearchResult {
// ...
navigateTo: { absUrl: string } | { application: string; path?: string };
}suit the need?
In current POC, I just added a As we are going to fetch results from multiple dataSources, I'm not really sure how to properly introduce a global/centralised scoring mechanism, as it would relies on the underlying dataSources to score their results, not to the GS service/aggregator, which mean distinct dataSources would have different scoring mechanism, therefor making them biased and useless. I don't see any good option on that problem, so any insight is welcome.
Last point, should I create a RFC for that issue? If so, should I do it right now or after we find answers to these problematics? ping @elastic/kibana-core-ui |
|
Thanks @pgayvallet , I'll bring this up at our sync today. @kobelb your input would be appreciated here as well! |
dataSource registration and architectureDon't really have an opinion here but do have a hazy recollection from an early meeting with @stacey-gammon that a client-side only implementation ended up being the leading idea. Stacey, do you remember this any better than I do? Have any thoughts on @pgayvallet's proposal? 👆 POC interface for the dataSource providersHow to handle redirecting to a result's associated page?
Results scoring / orderingHaving a "global" level of scoring would be cool but I'm not sure that it's necessary. I think it'd be totally reasonable if the results were grouped by their type so all that the aggregator would then need to do is order the types themselves. If we could do some sort of scoring on which type is most likely to be the one we want, that could be neat but I think also just having a predefined hierarchy would be fine. Anything else?
|
|
@myasonik Thanks for the input
TBH the hybrid server/client implementation is mostly driven by the need to able to use GS for external APIs such as ChatOps. I really don't know how much using GS outside of the UI is a real / realistic need and if the degraded search results would be acceptable. If it's not, my option A is probably the way to go. Not really sure who should have the last word on that...
I see two issues here however:
If we choose option A regarding dataSources (only allow to register them from the client-side), we no longer need the
with a example implementation of This would allow dataSources providers to have more control on how they should handle access to the result, while definitely breaking the possibility to have a server-side API counterpart. This is a high-impact decision.
This would be very doable. The main question I see here is: is ordering by type sufficient? I'm thinking mostly of results from other spaces or even other kibana instances. Should Maybe we should order by wdyt?
I'm mixed on this one. The major downside I see with that is that 3rd party plugins would not be able to register dataSources for
Ideally, any field added at a later time would be optional, to allow BWC with the existing dataSources. This is why we need to look a little further than stage 1 here. |
@pgayvallet and I discussed this over Slack and we decided it's probably best to go with the hybrid approach, though preferring datasources to be registered on the server side when possible:
I think we may be able to get away with just using absolute URLs, while still supporting navigating to other spaces and doing in-app SPA navigations. This idea is contingent on being able to parse out the application id from the URL, but I think this algorithm would be quite simple:
Why this should work:
Is there any case I'm missing here?
I think this creates a non-trivial amount of work later when we do want to support chatops. I think having some constraints on what results can do is a useful design decision for future requirements. |
|
Thanks for tagging me @myasonik. Here are my thoughts:
I'm leaning towards option B to avoid BWC issues down the road and pave the way for chat ops. Urls & storageUrlGenerators are registered client side, and the plan for now is to keep them that way, but to have the links accessible on server side via a simple helper: // On server side
createGotoLink(urlGenerator) =>
`${getDomain}/${getBasePath}/goto/${urlGenerator.id}?state=${risonEncode(JSON.stringify(urlGenerator.state))}`You wouldn't need that though until actually generating the links to give to an end user. The following code could exist on server or client: DashboardPlugin{
setup(core) {
core.globalSearch.addFinderDestination({
search(term: string, options: {}): Promise<GlobalSearchDataSourceResponse> => {
const [coreStart, ] = await core.getStartServices();
const dashboards = coreStart.savedObjectClient.find('dashboard', term);
return dashboards.map(dashboard => {
title: dashboard.title,
score: dashboard.score // Can you reuse the score returned from ES here instead of "order"?
urlGenerator: {
id: DASHBOARD_URL_GENERATOR,
{
id: dashboard.id,
}
}
})
}
});
}
}I need to think through this a little more, but I want to be sure we are thinking through backward compatibility here. With the URL Generator service, what is backward compatible is the id + state. NOT the url returned from it. I just gave a presentation on persistable state registries. I need to think more about where URLS fit into this picture. Like, what if someone created a plugin that supported pinning search hits from this service? The pinned hit would be stored in a saved object with a url string with no way to support migrations from the owner of that url. If you stored the URL generator state, there is a migration path. Thinking through a couple options:
^^ that doesn't exist yet but it should to support bwc server side url links, because the app will grab the correct generator, decode the state, and use the
OrderRegarding order. Maybe instead of order there should be a concept of "match score"? Just a thought. TerminologyIt may be too late in the game to change the terminology of "Global Search", though I do think this is going to be confusing. We have search services inside the data plugin. Maybe like "Global finder services" or "global resource locator" (... or, universal resource locator? 😄 ) Even if you are sold on Global Search, I really think you should steer away from "data source" because it's already used by canvas "data sources" and used heavily inside the ingestion manager. Some suggested alternatives:
The one given is that every item must have a url destination, right? |
NavigationI like @stacey-gammon's first two options much more than using absolute URL strings. Relying on a bunch of string manipulation to pull logic out of it just tends to spider with condionals over time so having the actual backing data feels much safer. No preference between the first two options but I think taking the stance that we will not support persisting URLs will come back to bite us so I'd avoid thinking of URLs as single use actions. Ordering results
I've got no idea. Almost seems like a product question? But also one that we might need data on from users... Also seems like an answer that is likely to change over time and/or how big the customer is. How much flexibility could we bake into our ordering strategies?
I'm not sure I follow on how this is different... What falls into a single if not a single type?
Wouldn't that not be possible across multiple ?
|
We have this kind of thing in use already in quite a few places, but I'm actually thinking we should discourage its use. Check out #62815 for more details. Although I'll reiterate here because that issue doesn't really touch on the type mappings. You can see how URL generators does this here: https://github.com/elastic/kibana/blob/drilldowns/examples/url_generators_examples/public/plugin.tsx#L31 If you can type check this though, you know the id. E.g. you are doing something like // If you use type mappings, this will be automatically typed and ensure
// that the state is of the shape the `FOO` generator requires
urlGenerator.get(FOO).createUrl({ fooBar: 'hi; })Because otherwise you are doing something like this:, which you can't type check anyway: // We can't type check this, just have to hope where-ever this persisted data came from is
// using the right state.
urlGenerator.get(id).createUrl(state)So, in the former case, what I talk about in that issue, is that we should discourage direct access with the plugin if you know, at runtime, the type you want, and instead, get it off the registrators plugin directly. e.g. Then you don't need typescript mappings at all. |
|
The hybrid approach is likely what we'll end up with long-term. As long as it doesn't drastically expand the scope and throw off timelines, implementing it from the start seems reasonable. Cloud UI 2.0 will be making various customizations to Kibana, including augmenting the global search bar with Cloud specific datasources. At the moment, the plan is to have all integration with Cloud APIs be performed client-side, so either approach will work in this regard. Spaces essentially hijack the base path to include the space url prefix. Internal to the spaces plugin, there's a utility function to augment a path with the space url prefix. Unfortunately, there isn't any functionality currently that creates a URL to navigate to an application in a different space. However, as long as we know the space, this is definitely possible. For the initial implementation, we'll likely want to only search within the existing space. The saved-object API only works for a single space at the moment, and can't be used to search across spaces. Long-term, this is definitely something that we'll want to support though. The url generators service is currently in the share plugin which appears to be focused on supporting the share context menus. If we start utilizing it here, it feels like it should be moved elsewhere. One of the first datasources which we'll want to search is saved-objects. We don't want to have to do a With regard to ranking the search results, switching from an |
|
On further reading, we potentially should not be trying to use the
There's also quite a lot that goes into scoring these documents that emulating the score is potentially quite challenging. For the initial implementation, if we can figure out an amenable design which displays the results grouped by the datasource, it'll let us avoid having to figure this out... |
|
I won't pretend to have entirely followed ALL of that 😄, but as far as grouping results according to some prefixed order goes - I think we could make that UX work. As we've envisioned this feature, two common use cases typically come to mind:
If we accept these as our core use cases, then forcing an order of results that returns applications, then saved objects, then anything else (we can hash out what other categories are left) seems like it would still provide the desired outcome we're after. That said, the ability to define a type in your search would allow users to bring anything to the top. Suppose you could search something like Dashboard: My favorite dash which would, in effect, make saved objects the top (only) result type. If we don't want to get into too much complexity handling user typed strings, perhaps they could even scope their search via a dropdown - in this scenario you'd change 'All' to 'Saved objects', as an example - then input your string in the search input. I don't know if that helps with scoping down the complexity, but there is flexibility in the UX that I am more than happy to explore. @bmcconaghy @alexfrancoeur do you have any thoughts? |
SavedObject types can already register a function to generate a URL to link to that object. We could easily leverage this for the saved objects data source. This definitely has some overlap with the URL Generator service, and it may make sense to consolidate these mechanisms or provide a bridge of some sort. I'm just trying to think about how we could get a working MVP done with less effort.
Depending on how this implemented, there isn't necessarily a problem with using absolute URLs. I would expect that a pinned search result is a full result or backed by some object, rather than being a raw URL. So a pinned result may have the form of That said, using URL generators may be useful when linking to other clusters where the version of the destination cluster may differ from the cluster the user searched from. But once again, depending on how those data sources are implemented, we could actually call the This pattern only works for saved objects though. I'm not sure what else we may want to search for, but if there are other things that are not saved objects that we need to return, using the URL generators may be necessary. But I don't think we necessarily have to use that pattern from the start. I would be more comfortable leveraging URL generators once that pattern has had more time to mature and there are generators implemented for all the things we need. |
|
So, It seems like the biggest questions is regarding the actual url format / mechanism. @stacey-gammon biggest issues/questions I have with using the
We could choose to go for absolute urls and use the DashboardPlugin{
setup(core) {
core.globalSearch.addFinderDestination({
search(term: string, options: {}): Promise<GlobalSearchDataSourceResponse> => {
const [coreStart, ] = await core.getStartServices();
const dashboards = coreStart.savedObjectClient.find('dashboard', term);
return dashboards.map(dashboard => {
title: dashboard.title,
score: dashboard.score // Can you reuse the score returned from ES here instead of "order"?
url: createGotoLink(DASHBOARD_URL_GENERATOR, { id: dashboard.id })
})
}
});
}
}However with the implementation you provided createGotoLink(urlGenerator) =>
`${getDomain}/${getBasePath}/goto/${urlGenerator.id}?state=${risonEncode(JSON.stringify(urlGenerator.state))}This kinda go against @joshdover suggestion to parse the url to be able to use Or would Also, not sure how that would work with space change ?
TBH I'm kinda afraid about
Because of reverse-proxy and middleware that can be present in complex production environment, I think we cannot be certain that the domain in the a server-generated result's url (using wdyt? Also
Because we can define a custom route for an application kibana/src/core/public/application/types.ts Line 225 in 48a33ab We would need to check every registered app to see if it matches. But this is likely not a real issue. |
Great points @pgayvallet. Seems like returning the url string and using I imagine the const link = createGotoLink(id, state);
const query = url.parse(link).query;
application.navigateTo('goto', { path: urlGenerator.id, query })
// or
const path = createGotoLink(id, state, { relative: true });
application.navigateTo('goto', { path })For spaces, don't we want it to look up the object inside the current space only? So the space prefix should be automatically applied based on whatever space the user is in when looking for results? |
As long as However, for v1 needs and the SO types we need to be searching for, I think I really like @joshdover idea of using the already existing Last point I'm thinking of, if that if the results exposed to the public GS API should all have absolute urls, we could still allow dataSources to return either absolute or relative urls, and have I.E class SomePlugin {
setup(core) {
core.globalSearch.addDataSource({
search(term: string, options: {}): Promise<GlobalSearchDataSourceResponse> {
return [
{
title: 'someObject',
// relative - will be consolidated to https://current-kibana-instance/base-path/my-app/path-to/some-object
url: '/my-app/path-to/some-object',
},
{
title: 'objectFromAnotherCluster',
// abs - will be kept as it is
url: 'https://kibana2/path-to/objectFromAnotherCluster',
},
];
},
});
}
}wdyt?
For v1 we will only be returning results from the current space. For next versions we do want to be able to search for objects outside of current space, and therefor need to be able to generate absolute urls containing the space. |
It may be time that we introduce a When it's not set, the server can just return absolute paths without the protocol, domain, and port. For example |
I think it makes sense.
One other option would be to construct this Would be something like (naive implem, there are some trailing slash issues when no basePath, but it's to get the idea): const publishAddress = `${getServerInfo().protocol}://${httpConfig.host}:${httpConfig.port}/${httpConfig.basePath}`I feel like this would answer the need in maybe 90% of the deployments while allowing ops to still configure the |
|
Watching the CAH recording and I see that App Search allows the user to tweak the priority of different types of results, so for example, an engineer could give extra weight to github results. Might be worth asking someone on that team how they are solving this problem of ordering results from mixed sources. |
|
As this is a complex feature, I created an RFC with a proposal for the initial version of the |
|
Closing as all sub tasks have been addressed. |
To power Global Search (#57576) and future features like ChatOps, we should provide a general framework for searching for various objects, apps, and other locations in the UI that a user may want to find quickly.
This framework should allow plugins to provide different "data sources" of which may be backed by Elasticsearch indices or other items (eg. application links).
The search framework should be able to take a search inputs, call each data source, and aggregate results into something sensible.
Tasks
globalSearchx-pack plugin #66293SavedObjectresult provider - [GS] SavedObject results provider #65222Applicationresult provider - [GS] Application results provider #65633The text was updated successfully, but these errors were encountered: