[WIP] [DataGridPro] Server side data source lazy loading

[arminmeh]

🚧 Work in Progress 🚧

Part of #8179
Resolves #10857
Resolves #10858

WIP Preview: https://deploy-preview-13878--material-ui-x.netlify.app/x/react-data-grid/server-side-data/lazy-loading/

Action items in progress:

• Make initial end index dependent on the viewport Use page size for the initial data load
• Refine/fix issues when rows positions are changed after
  • Sorting
  • Filtering
• Handle empty data set
• Update documentation and add more examples
  • Include [data grid] TanStack Query integration demos #14227
• Improve caching
• Lazy loading in combination with grouped rows / tree grid
• Throttling requests
  • With a fixed time
  • Configuration
• Update premium grid to use new processors/hooks
• Check if lazy loading can be combined with infinite loading
  • Support infinite loading
  • Support switching between viewport and infinite loading
• Error handling
• Tests
• Prepare release notes

[mui-bot]

Deploy preview: https://deploy-preview-13878--material-ui-x.netlify.app/

Updated pages:

• docs/data/data-grid/server-side-data/lazy-loading.md

Generated by 🚫 dangerJS against cc4423d

[MBilalShafi]

Really nice initial implementation 👍
Thank you for picking this up. 🙏

[MBilalShafi]

It would reset all the rows if some batch fails.

This raises another point, how should we handle the errors for lazy loaded rows? Should a retry UX be introduced to let the users retry the failed batches?

[arminmeh]

Is there anything in place for the regular server side data?
I guess we can use the same on both, but leave the existing rows in place for lazy loading

[MBilalShafi]

Is there anything in place for the regular server-side data?

We do have one for tree-data lazy loading, for the plain server-side data there isn't one because it always loads all the rows for the current page which could use a full-page error overlay (check Make the requests fail and re-fetch rows).

But for lazy loading and infinite loading, a full-page error overlay doesn't make much sense (not even for the first page) since it will block the users from seeing other loaded pages or triggering the load of other unloaded pages.

Considering how the plain lazy-loading works, coming up with a nice error UX for it would be tricky. I haven't seen any application with a perfect UX in this regard.

I can think of a few possibilities.

1. Do nothing, rely on the user scrolling out of the viewport and back on the same area of the viewport to retry fetching the rows.
2. Change some styling on the skeleton rows. Like the background color of the skeleton row to be red and adding a new row on top of the failed row section with a retry button (that will be auto-removed when the user a retry call has been sent, whether by clicking the retry button or scrolling back to this section of the viewport).
3. Provide a demo with the unstable_onDataSourceError by implementing a persistent Snackbar approach that includes a Retry button (we'll possibly have to rethink the API then too, to allow the user to request a specific portion of the data, let's say rows from index 100 to 109)
4. We keep track of failed pages in the state, allow re-fetch using an API, and have a button on top of the demo saying Refetch Failed Batches

@KenanYusuf I'd be interested in what you think about this since you recently touched the skeleton rows part.

[KenanYusuf]

Sorry for the late reply here, missed the notification.

3. Provide a demo with the unstable_onDataSourceError by implementing a persistent Snackbar approach that includes a Retry button (we'll possibly have to rethink the API then too, to allow the user to request a specific portion of the data, let's say rows from index 100 to 109)

I think this is a good option, it's non-disruptive, and allows users to carry on interacting with the data grid despite the fetching error. Updating the API to allow for this would allow users to customize the error experience too.

[MBilalShafi]

As discussed already, documenting here for further discussion:

I think the only difference as an end user between lazy loading and infinite loading would be the presence of rowCount.

For example, in an implementation, the rowCount is not available in the beginning, or it’s an unknown row count (-1), the Data Grid works in an infinite loading mode, but at a later point, the rowCount becomes available. At this point, the Data Grid has to compute the total viewport height and shift to a lazyLoading mode. This can also happen the other way around.

Could we remove the line between both cases based on this assumption, and handle them together?

[arminmeh]

I think that it is definitely possible to have them together
we will definitely have to add an explanation in the docs that one server side data feature replaces two client site features

in order to allow switching between modes based on the rowCount, infinite loading would have to work differently than what we have now

let's say that you load initial page and scroll fast to 3rd page or more. This will leave some skeleton rows in between the content, so switching to infiniteLoading will not make it possible to load those rows anymore (with the current way of working)

the solution could be that they both still load skeleton rows in the middle and that the only actual difference between them is what is being rendered after the last row (with content). This will remain as it is now

in the infiniteLoading mode, additional requests stop in two cases

• rowCount is returned and it matches amount of the currently loaded rows (this will also switch to lazyLoading and have the grid filled)
• response returns empty array. this can be used if the backend does not have the capability to return the count

[MBilalShafi]

we will definitely have to add an explanation in the docs that one server side data feature replaces two client site features

The idea was to deprecate the existing features with the introduction of the data source variants and remove them eventually. Because it doesn't make much sense to have these features used on the client side.

I'm confident that even today, most of the implementations on the users' side on top of the current lazy loading and infinite loading are binded to server-side logic on the user's side.

Even if there is some use-case on the client-side (for example the users want to lazy load a large JSON file present locally), it should be possible by using the Data source interface (implementing the getRows method on the client-side).

What do you think?

let's say that you load initial page and scroll fast to 3rd page or more. This will leave some skeleton rows in between the content, so switching to infiniteLoading will not make it possible to load those rows anymore (with the current way of working)

For lazy-loading => infinite loading transition (setting of rowCount prop to undefined or -1) I'd consider the existing rows stale. I'd reset the rows, move the user to the top, and send the request for the first page since we are unsure about the row count at this point, it might be less than already loaded lazy-loaded rows.

While a transition infinite loading => lazy-loading (setting of rowCount prop from undefined or -1 to some finite number) might require some further processing based on the provided rowCount value.

There could be 3 possibilities in this regard. (consider pageSize/batchSize = 10 in the examples)

rowCount > loaded rows
Let's say the loaded rows are 1-60 and the rowCount is provided 100, we just need to add 40 skeleton rows below the current rows and keep the existing rows rendered.

rowCount == loaded rows
If 60 rows are loaded and the rowCount is set to 60, we remove the 10 skeleton rows (rendered due to infinite loading) and consider all the data to be fetched.

rowCount < loaded rows
I'm doubtful about this case. It could mean one of the following:

1. The data set is changed: In this case, we need to reset the rows, move the user to the top, and send the request for the first page again.
2. The data set is modified, and some records are deleted from the end: We can simply discard the extra rows) in this case (for example, 1-60 are loaded, rowCount provided is 50, we internally delete rows 51-60, move the scroll to page 5 (let's say row 41) and consider all the rows to be fetched at this point
3. The data set is modified, and some records are deleted from arbitrary places: We'll have to reset the rows and re-fetch the data from the beginning again)

I'm not sure how we differentiate between these cases and handle them (in terms of the public API)

[arminmeh]

I have added the support for infinite loading as discussed.
if rowCount is undefined or -1, infinite loading is in place
otherwise, lazy loading kicks in.

Since we want to have them combined I thought of using the same name (lazy loading) for both and differentiate them as different modes of lazy loading (viewport and infinite)
WDYT?

I still have to check the scenario where the modes are changed after initial load