Messages sent while offline disappear after connection is regained if the user closes/reopen app - Reported by: @Santhosh-Sellavel

[isagoico]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

1. Make sure you are offline.
2. Send some messages to any user.
3. Refresh the page on the web. Close & open the app on mobile.
4. Now you can still the messages.
5. Now enable internet
6. Repeat step 3 or switch between chats.

Expected Result:

Messages that were sent while offline should be sent to the other user

Actual Result:

Messages are displayed on chat but disappear once the user regains the connection.

Workaround:

None, sent messages while offline are lost.

Platform:

Where is this issue occurring?

• Web
• iOS
• Android
• Desktop App
• Mobile Web

Version Number: 1.1.8-8

Reproducible in staging?: Yes
Reproducible in production?: Yes

Logs: https://stackoverflow.com/c/expensify/questions/4856

Notes/Photos/Videos: Any additional supporting documentation

WhatsApp.Video.2021-10-21.at.6.49.54.PM.mp4

Expensify/Expensify Issue URL:

Issue reported by: @Santhosh-Sellavel
Slack conversation: https://expensify.slack.com/archives/C01GTK53T8Q/p1634765287021500

View all open jobs on GitHub

[MelvinBot]

Triggered auto assignment to @johnmlee101 (Engineering), see https://stackoverflow.com/c/expensify/questions/4319 for more details.

[MelvinBot]

@johnmlee101 Whoops! This issue is 2 days overdue. Let's get this updated quick!

[MelvinBot]

@johnmlee101 Huh... This is 4 days overdue. Who can take care of this?

[MelvinBot]

@johnmlee101 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

[MelvinBot]

@johnmlee101 10 days overdue. I'm getting more depressed than Marvin.

[MelvinBot]

@johnmlee101 12 days overdue. Walking. Toward. The. Light...

[kidroca]

"Add Comment" requests are persisted to storage when we're offline
But when we regain connectivity they are read, removed from storage and kept only in memory
Making a refresh/exit before they are complete would make the app "forget" about them

1. Persisted request getting cleared when they're added to the in-memory queue

App/src/libs/Network.js

Lines 55 to 59 in b12eb5f

	// Merge the persisted requests with the requests in memory then clear out the queue as we only need to load
	// this once when the app initializes
	networkRequestQueue = [...networkRequestQueue, ...persistedRequests];
	Onyx.set(ONYXKEYS.NETWORK_REQUEST_QUEUE, []);
	didLoadPersistedRequests = true;

2. Persisted requests getting cleared by the queue itself

App/src/libs/Network.js

Lines 215 to 220 in b12eb5f

	// We should clear the NETWORK_REQUEST_QUEUE when we have loaded the persisted requests & they are processed.
	// As multiple client will be sharing the same Queue and NETWORK_REQUEST_QUEUE is synchronized among clients,
	// we only ask Leader client to clear the queue
	if (ActiveClientManager.isClientTheLeader() && didLoadPersistedRequests) {
	Onyx.set(ONYXKEYS.NETWORK_REQUEST_QUEUE, []);
	}

The request are only moved in memory, they are initiated but not completed - making a browser refresh (or closing the app) after the above code has emptied persistent storage would lose the data when the request didn't complete

[kidroca]

In theory it should be possible to recreate this with shorter steps like

1. Make sure you are offline.
2. Send some messages to any user.
3. Regain connectivity
4. Very shortly after refresh or close the page / quit the app
   When you open the app again you should see the messages briefly and then they would disappear (These were the optimistic messages created and stored. They get cleared since we receive the latest data from the server and it is supposed to replace them with the actual messages, but since requests never made it they're just removed)

Considering that it's very specific window of time that will cause the issue it's less likely to happen often

IRL cases that this bug can happen are:

• User have been offline for an extended period of time and has accumulated many requests. They decide to quit/terminate the app, but by coincidence they regain connectivity right before they quit
• User with accumulated requests regains connectivity. Something causes the app to refresh - like an app update
• User with accumulated requests regains connectivity. Something causes the app to crash
• User with accumulated requests regains connectivity. Something causes a logout

All of these are the case of: A persisted request is moved to memory and executed but something unexpected happens

---

IMO we're only trying to "bend" the network queue to fit this "process persisted requests" functionality
Due to the cyclic nature of the default queue and that it has no context of what's already running we're forced to clear persisted requests otherwise they'll get repeated - Onyx.set(ONYXKEYS.NETWORK_REQUEST_QUEUE, [])
Instead a separate queue should be processing persisted requests

• it would not be interval based - it's only needed when we have persisted requests - it's triggered when we regain connectivity
• it would clear requests from storage as they complete, and not when they are moved to the queue
• the default queue is relieved

[johnmlee101]

Looking now!

[johnmlee101]

Hmm, I would also say that a separate queue should have a maximum timeout, so we should keep that data with the requests. However, I like the idea. Do you know which files would need to be modified, or some psuedo code to represent this change?

[kidroca]

I think the only change to acomodate the suggestion would be in https://github.com/Expensify/App/blob/main/src/libs/Network.js

1. We still use the general queue to persist retryableRequests when we're offline
2. We remove any other handling related to persisted requests, this code
3. When we're back online we process whatever is stored in persistent request storage

Onyx.connect({
    key: ONYXKEYS.NETWORK,
    callback: (network) => {
        // when we transition from offline to online trigger the persisted request queue
        if (isOffline && !network.isOffline) {
            startPersistedRequestsQueue();
        }

        isOffline = network.isOffline;
    }
});

function startPersistedRequestsQueue() {
  // If the queue is already running we don't have to do anything
  // it will sort itself in case more requests are persisted after it started
  if (persistedRequestsQueueTask) {
     return;
  }

  processPersistedRequestsQueue();
}

function processPersistedRequestsQueue() { 
  // Onyx.get is necessary in the event that persisted requests are not yet read when we call this function
  // I'll mention alternatives outside the example
  persistedRequestsQueueTask = Onyx.get(ONYXKEYS.NETWORK_REQUEST_QUEUE)
      .then(persistedRequests => {
          // This sanity check is also a recursion exit point
          if (_.size(persistedRequests) === 0 || isOffline) {
              persistedRequestsQueueTask = null;
              return;
          }

          const tasks = _.map(persistedRequests, request => {
              // processRequest - reusable logic common to both request queues (should return a promise)
              return processRequest(request)
                  .then(() => removeFromPersistedStorage(request))  // when a request completes remove it from storage
                  .catch((error) => {
                       // Decide whether to discard the request despite the error, or keep it for a retry
                       if (!shouldRetryRequest(error)) {
                           return removeFromPersistedStorage(request);
                       }
                  });
            });

            // More request could have been persisted by now
            // If that's the case we'll handle them with a recursive call
            // Otherwise the recursion will exit at the recursion exit point
            return Promise.allSettled(tasks)
                .finally(roessPersistedRequestsQueue)
    })
}

// Have something unique in every request and use it to distinct which one we ought to remove
function removeFromPersistedStorage(request) {
  return Onyx.get(ONYXKEYS.NETWORK_REQUEST_QUEUE)
        .then(persistedRequests => {
           const nextPersistedRequests = _.filter(persistedRequests, r => r.id !== request.id);
           return Onyx.set(ONYXKEYS.NETWORK_REQUEST_QUEUE, nextPersistedRequests);
        });
}

Additional requirements like a request timeout can be implemented in processRequest though it might not be necessary - request would timeout on their own
If for some reason we need to abort the queue from outside, additional APIs can be exposed

---

This queue would start as much request as possible in parallel, the underlying environment would prioritize how they're actually executed, but we can assume a pool of 5-10 requests running at the same time

We might be fine with a simpler queue that processes one request at a time, then removes it from storage and reads the next one and so on until we run out of processed requests.
This way we can also order and execute requests based on a timestamp

---

Alternatives to Onyx.get

Currently applied workaround - connect and disconnect

App/src/libs/Network.js

Lines 77 to 83 in ccdf729

	// Client becomes online, process the queue.
	if (isOffline && !val.isOffline) {
	const connection = Onyx.connect({
	key: ONYXKEYS.NETWORK_REQUEST_QUEUE,
	callback: processOfflineQueue,
	});
	Onyx.disconnect(connection);

• we lose the ability to promise chain and we'll have to deal with flags for the recursive part of the queue

Top level Onyx.connect call for everything that we need

Not as reliable as the Onyx.get version - we're basically hoping that everything we need is already updated by the top level connect when we use it
Here we're discussion an alternative to top level connections: #6151

[kidroca]

Hmm seems like I misunderstood the timeout part, you mean like persisted requests getting stale after a day or so?

We can include some stale request filtering logic in there

[johnmlee101]

Ah yeah sorry, if things become too stale we should ignore. I'd imagine there's a situation where you're offline with a backlog of messages and you only check days weeks or months from then and having it auto-post wouldn't be useful.

So yep! Stale request filtering.

[johnmlee101]

I like this though! I'm going to mark as external, get a manager to create the job then I'll have them assign you.

[MelvinBot]

Triggered auto assignment to @stephanieelliott (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.

[johnmlee101]

Sorry for the shuffle there. @stephanieelliott once you create the job can you assign kidroca?

[kidroca]

Onyx.get fate is still undecided - I've posted alternatives to highlight their problems - none are suitable
Until this problem is solved: #6151 I don't think my suggestion can be aplied

[MelvinBot]

Triggered auto assignment to @SofiedeVreese (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.

[MelvinBot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @rushatgabhane (Exported)

[MelvinBot]

Triggered auto assignment to @roryabraham (Exported), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[stephanieelliott]

I'm about to go OOO, so reapplied the label to assign this to another CM team member (thanks @SofiedeVreese!)

[kidroca]

Can we agree the issue is complete: #5987 (comment)
The PR was merged 22 days ago: #5987 (comment)
And payment in Upwork is due

[SofiedeVreese]

@roryabraham just want to get your thumbs up for releasing payment for this GH (5987) which seems to have been actioned via the PR here?

[roryabraham]

Yeah, we should release payment here 👍

[SofiedeVreese]

Sorry for the long wait there @kidroca I've just released payment.

@Santhosh-Sellavel I'm going to add you to the Upwork job for your reporting bonus now too.

[MelvinBot]

Current assignee @rushatgabhane is eligible for the Exported assigner, not assigning anyone new.

[MelvinBot]

Current assignee @roryabraham is eligible for the Exported assigner, not assigning anyone new.

[Santhosh-Sellavel]

Thanks! @SofiedeVreese

[kidroca]

Thanks 🎊

[SofiedeVreese]

ok reporting bonus paid out too!

All is paid out, removed Upwork posting and closing GH.