Realm Query Pagination

[dotjon0]

Description

We have checked the Flutter docs at https://www.mongodb.com/docs/realm/sdk/flutter and can not see anything around pagination.

We have Realm collections/models where some individual collections/models have 100,000+ documents. Obviously when displaying the documents in this collection/model in the Flutter UI, there is no option but to paginate for memory/performance reasons. Is there a recommended approach for pagination in Realm? Cursor based pagination is the preferred option (that preserves the current Realm query query/sort/etc) if possible as this is likely the most memory efficient way.

How important is this improvement for you?

Dealbreaker

[nielsenko]

RealmResults (or RealmList) will handle it under-the-cover. We don't actually load the full result into memory, but access it element by element as you iterate over it. Hence you can just do the unthinkable and don't paginate.

[dotjon0]

sneaky, I like it @nielsenko ;-) Thanks for your fantastic and super fast help as always!

[nielsenko]

A small example..

import 'package:flutter/material.dart';
import 'package:realm/realm.dart';

part 'main.g.dart';

@RealmModel()
class _Stuff {
  late String id;
}

final realm = Realm(Configuration.local([Stuff.schema]));

Future<void> main() async {
  for (int i = 0; i < 1000; ++i) {
    await realm.writeAsync(() {
      realm.addAll(List.generate(1000, (index) => Stuff('$index')));
    });
  }
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    final query = realm.all<Stuff>(); // BIG! result set
    return MaterialApp(
      home: Scaffold(
        body: ListView.builder(itemBuilder: (context, i) {
          return ListTile(
            title: Text(query[i].id),
          );
        }),
      ),
    );
  }
}

[dotjon0]

o wow thanks so much @nielsenko that is very helpful, was not expecting an example, thanks!

[nielsenko]

@dotjon0 Anything new here?

[n-doton]

Hi @nielsenko,

When working with a large collection containing 180k documents the performance is poor. It appears to work as intended when scrolling using gestures or a scroll wheel, but scrolling with the right-hand scrollbar causes significant performance issues in our application.

We used the above example with 180k documents with 5 fields. It is also worth noting that we're using Device Sync.

Do you have any suggestions we could try to improve this?

Thanks in advance

[nielsenko]

I'm a bit surprised that the method of scroll would make a difference. Could you share a bit of your code?

[nielsenko]

I don't think that is realm related, try running the following:

import 'package:flutter/material.dart';

Future<void> main() async {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      debugShowCheckedModeBanner: false,
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: Scaffold(
        body: Scrollbar(
          child: ListView.builder(
            itemCount: 1 << 30,
            itemBuilder: (BuildContext context, int index) {
              return ListTile(title: Text('Item $index'));
            },
          ),
        ),
      ),
    );
  }
}

[n-doton]

@nielsenko Ah yes, you're correct.

Thanks for your help, I'll explore some solutions and will hopefully find a fix soon!

[nielsenko]

I can make it perform if I set an itemExtent on the ListView. I think the Scrollbar widget needs to know the height of every potential child in order to calculate the scrollbar offset correctly, which obviously is going to be bad for performance if you have to measure 200K items.

[nielsenko]

The following example (including realm) works like a charm for me:

import 'dart:isolate';

import 'package:flutter/material.dart';
import 'package:realm/realm.dart';

part 'main.g.dart';

@RealmModel()
class _Stuff {
  late int id;
}

final realm = Realm(Configuration.local([Stuff.schema]));

Future<void> main() async {
  Isolate.run(() {
    // just fill the db with stuff
    var j = 0;
    while (true) {
      realm.write(() {
        for (int i = 0; i < 1000; ++i) {
          realm.add(Stuff(++j));
        }
      });
    }
  });
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    final items = realm.all<Stuff>();
    final controller = ScrollController();
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: Scaffold(
        body: StreamBuilder(
          stream: items.changes,
          builder: (context, snapshot) {
            return Scrollbar(
              controller: controller,
              child: ListView.builder(
                controller: controller,
                itemExtent: 50,
                itemCount: items.length,
                itemBuilder: (BuildContext context, int index) {
                  final item = items[index];
                  return ListTile(
                    title: Text('Item ${item.id}'),
                  );
                },
              ),
            );
          },
        ),
      ),
    );
  }
}

[nielsenko]

.. even with 20 million items :-)

[n-doton]

Fantastic, this works perfectly in our repo too. Thanks again for your help!

[dotjon0]

.. even with 20 million items :-)

amazing!