store: Retry registerQueue on failure

In addition to the test this adds, I wanted to manually test it
end-to-end, to help be sure this covered the scenario where this
retry is known to be needed in the wild:
 * The app is offline for a while, perhaps because the device
   is asleep.
 * The app comes online, tries polling, and finds the event queue
   has expired, so it attempts a re-register.
 * Before that completes (which after all takes several seconds
   if the realm is a large one), the app goes offline again.
 * That request's response therefore never reaches the app,
   and so when it eventually comes back online it needs to retry.

Step 1 is annoying to carry out literally, because it means
waiting 10 minutes for the event queue to expire.  To work around
that, I sabotaged the getEvents binding function to use a wrong
`queue_id` value:

    'queue_id': RawParameter('wrong' + queueId),

so that the server would always respond with a BAD_EVENT_QUEUE_ID
error, just the same as if the queue had expired.

Then to take the app offline and online again, I just turned
airplane mode on and off on my device.  Because I used a
physical device connected to my computer over USB, that caused
no interference to my watching the logs on the console.

In my manual testing, the retries worked perfectly: no matter
how many times I turned airplane mode on and off, or with what
timing, the app always returned to getting a fresh queue and
polling it for events.

Fixes: #556

Author: gnprice

lib/model/store.dart

@@ -605,7 +605,7 @@ class UpdateMachine {
     final connection = globalStore.apiConnectionFromAccount(account);
 
     final stopwatch = Stopwatch()..start();
-    final initialSnapshot = await registerQueue(connection); // TODO retry
+    final initialSnapshot = await _registerQueueWithRetry(connection);
     final t = (stopwatch..stop()).elapsed;
     assert(debugLog("initial fetch time: ${t.inMilliseconds}ms"));
 
@@ -628,6 +628,22 @@ class UpdateMachine {
   final String queueId;
   int lastEventId;
 
+  static Future<InitialSnapshot> _registerQueueWithRetry(
+      ApiConnection connection) async {
+    BackoffMachine? backoffMachine;
+    while (true) {
+      try {
+        return await registerQueue(connection);
+      } catch (e) {
+        assert(debugLog('Error fetching initial snapshot: $e\n'
+          'Backing off, then will retry…'));
+        // TODO tell user if initial-fetch errors persist, or look non-transient
+        await (backoffMachine ??= BackoffMachine()).wait();
+        assert(debugLog('… Backoff wait complete, retrying initial fetch.'));
+      }
+    }
+  }
+
   Completer<void>? _debugLoopSignal;
 
   /// In debug mode, causes the polling loop to pause before the next

test/model/store_test.dart

@@ -149,7 +149,6 @@ void main() {
       addTearDown(() => UpdateMachine.debugEnableRegisterNotificationToken = true);
     }
 
-    // ignore: unused_element
     void checkLastRequest() {
       check(connection.takeLastRequest()).isA<http.Request>()
         ..method.equals('POST')
@@ -173,6 +172,35 @@ void main() {
         users.map((expected) => (it) => it.fullName.equals(expected.fullName)));
     }));
 
+    test('retries registerQueue on NetworkError', () => awaitFakeAsync((async) async {
+      await prepareStore();
+
+      // Try to load, inducing an error in the request.
+      connection.prepare(exception: Exception('failed'));
+      final future = UpdateMachine.load(globalStore, eg.selfAccount.id);
+      bool complete = false;
+      future.whenComplete(() => complete = true);
+      async.flushMicrotasks();
+      checkLastRequest();
+      check(complete).isFalse();
+
+      // The retry doesn't happen immediately; there's a timer.
+      check(async.pendingTimers).length.equals(1);
+      async.elapse(Duration.zero);
+      check(connection.lastRequest).isNull();
+      check(async.pendingTimers).length.equals(1);
+
+      // After a timer, we retry.
+      final users = [eg.selfUser, eg.otherUser];
+      connection.prepare(json: eg.initialSnapshot(realmUsers: users).toJson());
+      final updateMachine = await future;
+      updateMachine.debugPauseLoop();
+      check(complete).isTrue();
+      // checkLastRequest(); TODO UpdateMachine.debugPauseLoop was too late; see comment above
+      check(updateMachine.store.users.values).unorderedMatches(
+        users.map((expected) => (it) => it.fullName.equals(expected.fullName)));
+    }));
+
     // TODO test UpdateMachine.load starts polling loop
     // TODO test UpdateMachine.load calls registerNotificationToken
   });