[Lens] Provide a datasource for individual documents

[wylieconlon]

This might need to be a totally separate datasource, or part of the default datasource.

While using individual documents, we generally have a 10k document limit. We might need to use the scroll query to paginate when the number of documents is over 10k

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[nreese]

The maps app recently switched from hard coded 10k limit to using the index setting max_result_window to determine the maximum number of documents to pull, see #53500 for details. Maybe lens could do the same to be consistent? Maybe this route could be moved into Kibana proper so all apps have a consistent way of pulling index settings.

[ghudgins]

a bit unclear about the user story for this issue - is it displaying a table of individual documents? should we do this task or provide better integration with discover

[thomasneirynck]

a bit unclear about the user story for this issue - is it displaying a table of individual documents?

I think this gets at the heart of Lens now:

• (1) It is an UX to build aggregations
• (2) It's a collection of visualization-types

The work on supporting multiple query-languages (e.g. SQL, ESQL) in Lens will start chipping of at (1). Lens will start visualizing data of arbitrary provenance (not necessarily backed by an agg), which could include showing individual docs as a table, as a scatterplot, ...

[stratoula]

Closing this as we have added ES|QL as a new datasource and the support of a datatable computed from external resources