[data grid] Improve Server-side support with new data source

[cherniavskii]

Summary 💡

The server-side data source will be implemented in a step-by-step manner, the issues it will be covered in are mentioned in the list below.

• A TypeScript interface with clear requirements of what the user has to provide for the data grid to have the necessary data. (Handled in [DataGridPro] Add data source interface and basic documentation #10543)
  • Start with a Collection interface where you provide the interface for the data collection that could be used by Toolpad in their server-side function. The goal is to have the configuration of the data source in a single place where the users can translate the pagination, filter, sorting models, etc. to whatever you have on the server (HTTP request, DB query, or something else)
  • It should handle the fetching first, and we can extend the interface with updating functions later.
• [data grid] Implement server-side data source in tree data #3377
• [data grid] Implement server-side data source in row grouping #10859
• 🚧 [data grid] Implement server-side data source in row lazy loading #10857
• 🚧 [data grid] Implement server-side data source in row infinite loading #10858
• [data grid] Implement server-side data source with nested data lazy loading #14527
• [data grid] Implement server-side data source with aggregation #10860
• [data grid] Implement server-side data source with pivoting #10861
• [data grid] Support edit functionality with server-side data source #13261

Motivation 🔦

To make the integration of the data grid with users' data easier. Today we have https://mui.com/x/react-data-grid/row-updates/#lazy-loading but it could be a lot better.

Examples 🌈

• https://marmelab.com/react-admin/DataProviders.html
• @Janpot has an idea for the createCollection() function in the Toolpad

Shaping page

https://www.notion.so/mui-org/Data-source-proposal-db58b8b749154420a407f231c5528c1d

User requests

• https://mui.zendesk.com/agent/tickets/15724

[cherniavskii]

We have recently explored supporting lazy loading for tree data and row grouping, and this seems to be extending the original description of the data source.

With the data source, we want to support at least these use cases:

• Row grouping: lazy-load children rows.
  Currently, row grouping only works on the client side. This means that all the rows have to be fetched in advance. This is not scalable for large datasets.
• Tree data: lazy-load children rows.
  Same as with row grouping, tree data only works on the client side. We have a demo allowing lazy load children rows, but it's not a part of the core package. We need to support this natively.
• Hooking up server-side pagination, sorting, and filtering
  While it's possible to use server-side pagination, sorting, and filtering, each of these features has to be handled separately. We need to provide a way to hook up all of these features at once. It will also be beneficial for us if we have a centralized way of fetching data from the server (when not using client-side features).
  This effectively replaces paginationMode="server", sortingMode="server" and filterMode="server".

It’s preferable for the data source to be optional in the use cases that are currently supported by the data grid (like server filtering, sorting, pagination, etc.).

Benchmarks:

• https://www.ag-grid.com/javascript-data-grid/server-side-model/
• https://js.devexpress.com/Documentation/ApiReference/Data_Layer/CustomStore/
• https://www.telerik.com/kendo-react-ui/components/grid/data-operations/odata-server-operations/
• https://blazor.syncfusion.com/demos/datagrid/remote-data?theme=bootstrap4
• https://www.infragistics.com/products/ignite-ui-angular/angular/components/grid/remote-data-operations

[Janpot]

Just for cross-reference: We now have an implementation in Toolpad which supports pagination, filtering, sorting and all CRUD operations.

[avioralm]

@cherniavskii Any update on this development ?

[BrentFarese]

Seems like this is a priority for V7? Does the MUI team have any guidance on when this will land/be usable? I ask b/c our team is considering server-side pagination, sorting, filtering in the upcoming months and we'll have to roll our own solution or use this solution if it becomes available.

We are Premium users. Thanks!

[MBilalShafi]

Hey @BrentFarese, thank you for reaching out.
The first implementation of the server-side data source is already in progress and will include server-side pagination, sorting, filtering support, tree-data lazy-loading, and other related improvements.
I can't say how soon this will be available, but I can assure you it's one of the top priority areas for the team and hopefully, it will soon be available.

Meanwhile, may I know what specific features you currently use or plan to use in the future, do you also expect any advanced use cases like aggregation, row-grouping, etc. to be utilized in your project with server-side support?

CC @avioralm

[BrentFarese]

Hey @BrentFarese, thank you for reaching out. The first implementation of the server-side data source is already in progress and will include server-side pagination, sorting, filtering support, tree-data lazy-loading, and other related improvements. I can't say how soon this will be available, but I can assure you it's one of the top priority areas for the team and hopefully, it will soon be available.

Meanwhile, may I know what specific features you currently use or plan to use in the future, do you also expect any advanced use cases like aggregation, row-grouping, etc. to be utilized in your project with server-side support?

CC @avioralm

@MBilalShafi we don't plan to use advanced features for the initial implementation, just pagination, sorting, filtering support. I know timelines shift all the time, but is this work expected to land in 1-2 months or is it more like 4-6 months? Just gauging whether we wait for it or roll our own. Thanks!

[MBilalShafi]

is this work expected to land in 1-2 months or is it more like 4-6 months

1-2 months is the more likely period for the first implementation (including server-side filtering, sorting, and pagination), however, I'm curious to know if meanwhile there's a barrier for you to keep using the server-side features already supported by the Grid.

The following features are already available in the Grid:

• Server side sorting
• Server side filtering
• Server side pagination

[BrentFarese]

is this work expected to land in 1-2 months or is it more like 4-6 months

1-2 months is the more likely period for the first implementation (including server-side filtering, sorting, and pagination), however, I'm curious to know if meanwhile there's a barrier for you to keep using the server-side features already supported by the Grid.

The following features are already available in the Grid:

• Server side sorting
• Server side filtering
• Server side pagination

No there is not a barrier to us using the existing server-side features. We might have to do that depending on when we implement server-side logic and when MUI releases the initial version. Would like to avoid having to migrate tho so we'll follow releases closely. Thanks!

[avioralm]

@MBilalShafi We are using the pro plan in our company and it is getting harder and harder to implement new tabular data features without the data source.
Can you share some time estimation for this feature?

[MBilalShafi]

Hi @avioralm, thank you for reaching out.

I'd be interested to know more about specific features / use-cases you want to solve with the data source.

The first implementation of data source is expected to land in Q2 2024, however it will cover a subset of features with other advanced ones to follow up.

Here's the list of Server-side features targeted in first implementation of the data source.

• Filtering
• Sorting
• Pagination
• Loading plain data with data source
• Tree data lazy loading

Does that cover your use case?

[avioralm]

@MBilalShafi
Yes it is fully covers my needs
especially the data tree implementation

[BrentFarese]

@MBilalShafi any update on the initial release? We're doing pagination now using the "old" way. Any specific issues/PRs we should follow? Thanks!

[MBilalShafi]

Hey @BrentFarese thank you for your continued interest in the set of features related to backend integration, you can subscribe to this first phase PR which is currently under review: #12317
You can check the in-progress documentation [plain data, tree-data], play around with the demos, and give some early feedback if you'd like.

[sebastianfrey]

Hi @MBilalShafi,

We've been using the DataGrid in our app since it was first released. Our app binds various server-side data sources with different interfaces from 100 rows to several million rows. For this reason, we have relied on the server-side filtering and sorting modes in combination with Infinite Loading from the very beginning to implement server-side data.

Due to @cherniavskii comment that the new data source mechanism will replace the previous server-side modes, I am now a little unsure whether there is a suitable migration path for our approach.

This leads to two questions:

1. Is this approach - Infinite Loading with server-side filtering and sorting - also supported with the new data source mechanism?
2. Is the new data source mechanism released as major release? I hope so, because this is a major change for us and probably some other customers too.

[MBilalShafi]

Hey @sebastianfrey,

The currently supported filtering, sorting, pagination are going nowhere, at least for now. The data source is supposed to be an opt-in feature, if not used, the Grid will behave the same as of today.

Coming to your questions, yes, the infinite loading will be subsequently supported with the data source, but it will most likely be hidden behind a flag, to be backward compatible.

If we even plan to replace the current implementation with the new one, it will happen no sooner than the next major release (expected in March 2025), and there will be a proper migration path available for the breaking features before that.

How does that sound?

[sebastianfrey]

Hi @MBilalShafi, thanks for reaching out and clarifying things. Yes that makes sense, as soon as the new data source feature is released I will make some tests involving infinite loading.