Indicate dashboard search panel limit of 500 results

[jimmyjones2]

When including search results in a dashboard it took me a while to realize I was not seeing all the results, just the first 500. I just presumed there were exactly 10 pages of results! Would be good to have some kind of notification at the top of the panel to say "showing first 500 results" or something similar? In discover you get such a message when you scroll to the bottom,

[jccq]

Jimmy, I would feel like recommending to remove the page count alltogether as it is indeed misleading. instead have an endless scroll search result (results appear as one reaches the bottom) or a nice "See 50 more button at the end of the current results" to make them appear.

[timroes]

What is the reason, why there are only 500 search results? Could this be configured or just use pagination in ES to retrieve ALL records while switching through pages?

[rashidkpc]

The problem with the pagination support in Elasticsearch is that it does not have the ability to pick up where it left off. So you can't say {from: <some doc id>} which means that the results in the infinite scroll would appear out of order, or duplicated, if events continued to come in and things get push to the next page. This becomes even worse when sorting by anything but timestamp. This is the reason we queue up 500 results. It is already configurable as advanced settings discover:sampleSize

[pecegit]

It does support "from": N, "size": S. Search tables on the dashboard have paginations and as far as I can tell it shows 50 records at the time with pagination controls. It should of course use "from": (page - 1) * 50, "size": 50 and use hits.total to calculate pages count. It's currently returning all 500 records and paginate client side. In my particular case, showing a few fields from a Twitter index it it drops the query from 3126ms to 300ms not to mention traffic and whatever load the pagination thing adds on.

[burtonator]

Why can't we just use a scroll ID .. .my understanding is that those results sets are deterministic?

[phagunbaya]

Guys, is there any way to see all the logs matching the search query (beyond first 500 rows) ?

[jimmyjones2]

@phagunbaya see advanced options

[stacey-gammon]

Design team (@cjcenizal @snide @alt74), any thoughts on how to present this message?

This is the same styling used in the discover tab but there it only shows up at the bottom once all 500 results have been scrolled through. It looks a bit ugly at the top here. How about I put the notification only on the last page so it's not so in your face?

[snide]

@stacey-gammon in that usage, it's fine to only be on the last page (assuming they paged through all of them. Can even be below the last result.

[stacey-gammon]

Based off feedback on #10827, I think we need to put some more consideration into the UI for this.

Putting the warning on the last page doesn't help the initial confusion that it looks like there are only 10 pages of results, and putting it on the first page (at least with that warning banner) is a bit too intrusive.

Unless the design team has any quick ideas, this is starting to fall outside the time limit for the low fruit label.

cc @jimmyjones2 @trevan @cjcenizal @snide

[cjcenizal]

The other issue in #4161 was that on the first page you don't know that the results have been limited. Could there be some kind of indication on the pager perhaps, or at the top of the first page?

I think we just need some indication that 10 pages is an artificial cap. Just some small visual signal, which can be explained with additional text if the user needs more info. How about we use an icon and a tooltip:

Hovering over this icon can show the help text in a tooltip: "These are the first 500 documents matching your search. Refine your search to see others."

[uboness]

hhhmmm... lightning bolt at the end of pagination bar? doesn't play well with me tbh...

why do we need to show pages at all? why not just < and > buttons (like in tables) and when they navigation to the last page we show a subtle message next to the disabled >?

showing pages is useless and (as we can see here) only confuses. (assuming we also show total count)

[cjcenizal]

@uboness Are you suggesting we show three things: left arrow, right arrow, and a number that represents the total count of documents?

And yes I agree we need better icons.

[jimmyjones2]

Showing the actual total would be very useful and clear up the issue of assuming the number of results based on pages.

I also like the idea of an icon, maybe and exclamation mark triangle or an info circle.

[uboness]

@cjcenizal yea.. These 3 things... I think it'll be cleaner and more informative

@jimmyjones2 we can explore an info icon next to the text by the disabled next button

[stacey-gammon]

Whoa, I swore I wrote up a long comment here yesterday, but I guess I never submitted it. Or I accidentally posted somewhere else. Anyway, to summarize:

We don't have the total count of documents/objects because we set a size limit (see #8044). I don't think there is a way to get the total count without removing that size limit...

We currently have two types of pagers and both have problems with giving incorrect information.

We could do something like 1 to 10 of many when the size we request is equivalent to the count of rows we get back. Gmail used to do something like this, but now that I'm re-checking, it looks like they give an accurate count now.

Anyway, I'm removing the low fruit label, because this is really a larger issue with all our pagers and tables.

[uboness]

We don't have the total count of documents/objects because we set a size limit (see #8044). I don't think there is a way to get the total count without removing that size limit...

I assume that the size limit is on the query side as part of the request to es... If that's the case you should have access to the total count... If it's not the case, I guess the question is why isn't it?

[stacey-gammon]

Maybe I am misunderstanding. By total count, are you referring to the total count of documents that were returned by es, or the total count of documents that would have been returned by es if we didn't add a size limit to the query? If the former, then we do have that number but it's misleading. If the latter, then I am just not aware that this number is returned by es.

[uboness]

total count is the total number of documents that match your query, and ES always returns it.

[stacey-gammon]

Ah, I see "hits": { "total": xxx, ... } Awesome. In that case, sgtm!

[stacey-gammon]

Reusing the toolbar and pager components and styling from the landing pages, how does this look @cjcenizal @snide?

In action:

[jimmyjones2]

@stacey-gammon Looks great!

[cjcenizal]

@stacey-gammon Could you try using a regular Bar component instead of the Table-specific one? The gray background looks a little out of place here.

[uboness]

what @cjcenizal said... we need to ease up on the grey... OTT 👍

[cjcenizal]

Also, maybe try aligning the content to the left, in the order of buttons, count, then warning:

[ < ] [ > ] 301-500 of 8,268 (Your search results are limited, etc.)

[uboness]

Also, maybe try aligning the content to the left, in the order of buttons, count, then warning:

[ < ] [ > ] 301-500 of 8,268 (Your search results are limited, etc.)

not sure about that... if all the tables use right alignment for pagination, we shouldn't break it... keep UI patterns consistent

[cjcenizal]

Yeah, it's just an idea. I agree consistency in our UI patterns is important. The problem I think we need to solve here is that it doesn't make sense to have the warning on the left. Users will need to read it after interacting with the arrows, and after reading the count information for content.

This table also doesn't fit into the same pattern as the tables on our landing pages (which are used for searching and bulk-managing user-created content). Those tables surface controls which aren't appropriate for this table. So I think it's worth trying a new pattern here to see if it makes sense.

[stacey-gammon]

mmm, does look better without the grey. here it is with the pager stuff on the left and the warning on the right.

[joshbronson]

Sorry to comment on an old thread. This probably belongs in 4933, but that was closed in favor of this bug, so hopefully this is the correct place to say this:

Why not use a scroll ID, as a @burtonator recommended above, in order to paginate over an arbitrary number of documents?

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html

Every documents should appear exactly once in the paginated list, if I'm understanding it correctly.