EuiInMemoryTable - provide callback for search

[nreese]

EuiInMemoryTable is great and covers almost every use case for providing a responsive table. The only missing usecase is when the number of items gets really big. It would be nice to still use the EuiInMemoryTable (because its so simple to use and responsive for users) but just to be able to provide a search callback that gets called when the search is modified. That way, new items can be fetched when needed and up to the discretion of the consumer.

cc @uboness @cjcenizal

[uboness]

I would argue that this goes beyond the responsibility of this table as it's built to handle all items in memory (the previous version of the table provided a callback to load items remotely and everybody shouted over it). We need to keep clear boundaries of components and if we decided that this table and all its internals are for in-memory data, we need to keep it as such.

For you use case, I'd not use the internal search functionality of the table, and instead have an external search bar that is responsible to load the data set and then populate the table with it (which effectively means the internal search control is not used and should not be configured).

[nreese]

If I don't use the built in search bar then I won't get all of the fancy search stuff like drop downs and stuff. I want all of that and the ability to fetch new items. As a consumer of the table widget, I want to wire as little as possible. I like the model of just passing an array to the table and letting the table take care of paging, sorting, and filtering. Then the consumer can sit back and fetch new items as needed. Its not pure but nothing ever is.

[uboness]

If I understand correctly, what you want is that the search bar filters will filter the data in memory but the search will effectively replace the data. That won't work with the table nor the search bar. The search in the table is just a search bar... but the filters and the search box in the search bar are one... you can't separate them - they're both working on the same query model. In fact, the filter effectively modify the the query - the same query that the search box modifies... so if I understood you correctly, it's not possible to do what you want with the current components.

[nreese]

I want a call back that gets called any time the search state changes - this includes search bar and filters. The callback should get the search state as a parameter. The callback will decide if an external search is needed and do so with the search state. In the event that the callback does an external search then the EuiInMemoryTable component will be updated with new items property so it won't even care. It will just happily render itself with the new items.

The only thing EuiInMemoryTable has to do is call the callback if provided. If the consumer decides to take any action, updates will be passed to EuiInMemoryTable just like any property change.

[uboness]

ok... so basically you want to "intercept" the default search and if you should decide so, you'll execute a new search yourself... otherwise (if you do nothing), it'll fallback on the default internal behaviour of searching in-memory. is that it?

[nreese]

exactly

[uboness]

I thought about it more and even built a small poc of it, and my conclusion is that we should not do it with the search bar. The problem here is that the filters and the search box in the search bar are tightly coupled (intentionally) - they're working on the same query construct that serves as their model (any changes to the filters, or to the query in the box directly changes the query... and those changes are immediately reflected on both). Now... building this interception mechanism is definitely feasible, but in order for it to work, it will force the interceptor (external search) to respect the exact same search semantics as the table has internally, which effectively means that for the same query, the exact same search results should be returned by the internal search or the external search. But this is something that we can't really (and shouldn't really) commit to... there is a big difference between searching in-memory at the client compared to searching with ES (the latter has a notion of relevance and boosts). And I don't think we should bind the two.

I would keep all actions in the EuiInMemoryTable in-memory and not try to combine it with external filtering functionality. What we could do is add support for "filters only" to the search bar - such that it'll be possible for the search bar to only display filters (these will always filter in-memory), then, along with #443 you'll be able to add your own search box as a component in the toolbar which will execute the search externally.

[cjcenizal]

it will force the interceptor (external search) to respect the exact same search semantics as the table has internally, which effectively means that for the same query, the exact same search results should be returned by the internal search or the external search

@uboness Trying to understand this -- why do we need to force external search to work the same way as internal search? Can you help me understand the problem from a user's point of view, or the consumer's point of view?

[uboness]

I tried to explain it above... It's one query object that would sometime execute externally and sometimes internally... So the results need to be consistent (it doesn't make sense from the user perspective that when executing it on ES for example the user will get one set of hits and then when adding a filter (which would be done internally)the filter will be applied on a different results set... For example.. Lets say the user searches for X... That's expected externally and returns results R1.... Then the user adds a filter "name:Y"... That would create a query "X name:Y" which executed internally... But for it to be consistent X needs to match all items... But nothing guarantees that.

As I mentioned above... You need to decide what you want.. An in-memoy component (in-memoy table) that works on in memory data set or an more generic component that can work on any data set (basic table)... Creating something in between (something that sometimes acts as in memoy and sometimes doesn't) makes little sense...

[cjcenizal]

I see. Why not just consider search & filters to be part of one query, which must either be executed remotely or locally, but not both? This will avoid the situation you're describing. I don't think it makes sense to search remotely and the filter locally.

[cjcenizal]

@nreese I spoke with @uboness this morning about this. We want to confirm that in your use case, you want to completely override the search & filtering behavior, right? In other words, you don't want a mixed use-case where the search is performed remotely and the filtering is done locally.

If this is correct, then we're all on the same page and we can move forward with implementing this.

[cjcenizal]

Just spoke with @nreese and he confirmed the above is correct. @uboness we're good to go.