Could the endpoint queue send newest locations first?

[wir3z]

Question around the Endpoint Queue handling. If the phone is out of data connectivity range, it starts to spool up locations to send out in the endpoint queue. If that outage was for days (think going on vacation), the phone ends up with a lot locations, that seems to take days to finally offload all of them.

I looks like it starts at the oldest location in time and works to the current time. Would it be possible to reverse that order to offload the newest first. That would address issues around automations requiring location updates.

I don't know if that breaks a lot of other logic with the queue handling, just a thought.

• App build number: 2.5.2
• Android version: Android 14
• Device: Pixel 7
• Installation source: APK

[growse]

Think this has come up before.

The biggest challenge (I think) is technical - the library we use to persist the queue to storage (tape2) only supports queue semantics and not stack ones. If a message is dequeued for sending, and that fails, we actually push it back onto the head of the queue (bit like a stack).

Because tape2 doesn't support this, we fake it with two tape2 queues, one large and one higher priority with length 1 representing the "last failed message". Messages that fail get offered back to the small queue, rather than the big one.

If we wanted full stack semantics, the problem you'd have is how to deal with failures. If you're offline, a message that fails to send should be put back on the stack but as new locations arrive from the device, they should be stacked ahead of it.

You can of course build a stack out of multiple queues, but you end up doing a lot of IO on each operation.

One option is to vendor the tape2 code and add our own push/pop.

Open to suggestions.

[wir3z]

Ahhh I see says the blind man! I figured there was something that made this harder.

For the next failed message, is that always the the most current in the "queue of 1"? If so, that probably checks the box enough and might be easier to just deal with this on the server side where messages that are too old just don't go through the same location processing until it catches up.

[wir3z]

Going to basically answer my own question above. It seems to start interleaving the latest from both queues. On the server side, I suppressed dealing with incoming presence events for older locations (just use them for location history, etc).

What about a button on that status page where you could force clear the queue if someone got into a bad state with a ton of locations to offload? The data junky in me cringes at purging data, but it would address some of these situations.

[growse]

There's a bunch of preferences where if you change them, it clears the queue. Notably, switching between MQTT and HTTP clears the queue.

So in the rare case you need to wipe it out, could you quickly just toggle between those? Saves having a big shiny button that people might press by accident?

[wir3z]

It looks like it retains all the HTTP settings when you do that. Is that correct? That might be the ticket right there for these corner cases.

Only catch22 is it's not intuitive unless you knew that was the behavior.