Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Messages get stuck after retries on SailfishOS #8536

Closed
4 tasks done
schmeat opened this issue Jan 18, 2019 · 21 comments
Closed
4 tasks done

Messages get stuck after retries on SailfishOS #8536

schmeat opened this issue Jan 18, 2019 · 21 comments

Comments

@schmeat
Copy link

schmeat commented Jan 18, 2019


Bug description

This only affects one of my chats but it effectively locks me out of talking to the person because it keeps retrying without actually sending. All of my subsequent messages are stuck in queue and never get sent or delivered. Message sending/delivery work fine in the same chat on desktop though, so only stuck on mobile.

Steps to reproduce

  • Send a message before network cuts out (elevator?)
  • Observe retry loop forever

Actual result: Messages are stuck in retrying loop
Expected result: for my messages to send

Screenshots

Note there are stuck messages above the one that sent correctly (from desktop)

photo_2019-01-17_19-26-05

Device info

Device : OnePlus ONEPLUS A6003 (OnePlus6)
Android : 9 (1812250000, ONEPLUS A6003_22_181225)
Memory : 46M (44.27% free, 512M max)
Memclass: 256
OS Host : ubuntu-14
App : Signal 4.32.5

Link to debug log

https://debuglogs.org/bfeb4b01b7a80edce0fca69e5fd797d2c7321d0dce6fd708d1fdde17d6c8e0da

@greyson-signal
Copy link
Contributor

Looking into it. In the meantime, try force-stopping signal and re-opening it.

@schmeat
Copy link
Author

schmeat commented Jan 18, 2019

I initially did force stop and it did not work at the time. However, I tried it now (after sending desktop messages) and it seems to have properly failed now.

I'm not sure if it's the time that's passed or the desktop messages that did that? or neither? 🤷🏼‍♂️

@greyson-signal
Copy link
Contributor

You had a message that experienced some network failures, and was therefore scheduled to be retried in the future, but it wasn't being retried, blocking future messages from sending (messages need to be sent in order within a conversation). Looks to be a problem with WorkManager.

@late-latte
Copy link

Not sure if it's related, but my client can call, receive calls, and receive encrypted text messages, but I can't send any text message and send or receive media messages. All they do is stuck in retrying loops.
Website latest release 4.31.8, without Play Services/GCM.
Lineage 15.1 Official Build 17/01, Asus Max Pro M1.
I'll send debug log later if you needed it, but there's nothing indicative of a problem so far.

@strybe
Copy link

strybe commented Jan 18, 2019

Same problem here. I have two chats where sending messages is somehow blocked. Both started the problem after an unsuccessful attempt to send a picture or small video file.

Chat 1
It started with failed sending of picture+text to Nexus4 device. I think both devices had 4.30.8 installed.
Almost everything was tried to get it working again (lots of messages send on both devices, switching networks on and off, enable/disable disappearing messages, restarting encryption on both devices, forced stop of Signal, reboot device, update Signal to 3.31.6).

Receiving messages always worked. Also typing indicators and calls in both directions. Notification of enabled disappearing messages showed same problem as sending messages of all kind (picture, voice, text).

Logs are after experiments with typing indicators and sending different messages from both devices. Problem started around 18-12-06, so this probably is not in the log anymore.

Jolla (not sending):
https://debuglogs.org/d56939e1379304edf99e5c10110c5aa62c3111c39a2db82e6669321dfedc334e

Nexus4:
https://debuglogs.org/ada0c2123639c5836f71979be5c8e913bdd694852555057169149da279d968cd

Chat 2
It started with sending a small video file+text while on WiFi. Sending was stuck with rotating circle for a while. Lost patience and deleted unsent message. Next try with text message failed. Tried restarting Signal, and network on/off. Test with receiving messages from other device successful

Logs are short after problem started (19-01-13), from both devices.

Jolla (not sending):
https://debuglogs.org/215f648a27b50fb79c69df3c28df3cb978ee7eeb5302d1b858cbc6edab466f91

Samsung:
https://debuglogs.org/749a3ad56bcd45681b27c2a16c881217148533296281e9629bb93d7c2e835be5

I know Jolla and SailfishOS are not supported, but since Signal is working good for other chats it might be helpful somehow nevertheless.

@greyson-signal
Copy link
Contributor

I'm talking with the WorkManager devs, and apparently this may be an issue with manufacturers being too aggressive with their battery optimizations. Can you folks check if you have any 'battery optimizations' enabled on your devices? @schmeat I know OnePlus is pretty aggressive about this sort of thing. Our support article on notifications has some details on how to go about checking this.

The technical details behind this are that after a message fails to send, it's scheduled to be retried on the Android system's JobScheduler. Some manufacturers heavily restrict the usage of JobScheduler. Now, jobs that an app submitted are supposed to be highly-prioritized when the app is foregrounded, but apparently these devices aren't doing that, possibly because of some overly-aggressive battery optimizations. This results on the related messages not getting sent.

Now, I personally think that if the app is foregrounded, WorkManager should be running the job on it's own, independent of the JobScheduler, so I'm making that suggestion and will try to work with them to resolve this.

Thanks!

@schmeat
Copy link
Author

schmeat commented Jan 19, 2019

I have "adaptive battery" turned off for Android P and I also have signal white listed for battery optimizing, but I don't know what my phone is actually doing due to that OP bug lol

https://www.androidpolice.com/2019/01/18/oneplus-phones-are-discarding-androids-battery-optimization-whitelist-a-problem-as-more-apps-update-to-target-oreo/

It says not optimized but I can't really trust it now 🤷‍♂️

@PaulXiCao
Copy link

PaulXiCao commented Jan 20, 2019

Same issue. App is stuck in retry loop, but still receives messages. Only effects 1 chat.

Debug log: https://debuglogs.org/2575d415b57d8078b19931fc154f114e0f505dc58e142551677d33574cfcdc63

Device : Samsung S7 edge (SM-G935F)
Android : 8.0.0
OS Host : debian testing
App : Signal 4.32.7

EDIT
Restarting the phone canceled the retry loop and re-sending the messages worked.

@greyson-signal
Copy link
Contributor

After doing more digging, this could be a bug that was introduced in WorkManager beta02. I'm seeing problems with it in my testing. I'm going to be moving us back to beta01. Let me know if the issue comes up again in 4.32.8. Thanks!

(Note: If the message is still stuck when you do the update, you may still have to force-stop and/or restart your device to kick it in gear. But hopefully no new occurrences come up in 4.32.8)

@akutuzov
Copy link

akutuzov commented Jan 28, 2019

@greyson-signal @strybe Do you know of any way to 'force-stop' Signal on Jolla/Sailfish? I've run into exactly the same issue with messages not being sent to one particular contact from my mobile. The only way to fix this was to completely reinstall the app, with new keys generated, etc.
Several days after reinstalling the same happened again. I upgraded the app to the version 4.32.8, but the messages still are not being sent to the 'problematic contact'. Rebooting the phone or restarting Android support does not fix the problem.
So, is there any way to 'reset' the schedule and purge the stucked messages from within the Signal? I would really hate to reinstall it again from scratch.

Thanks for this thread, it was very helpful.

@strybe strybe mentioned this issue Feb 2, 2019
4 tasks
@strybe
Copy link

strybe commented Feb 2, 2019

See also #8559, problem still exists.

@akutuzov For SailfishOS you can go to Settings -> Apps -> Signal -> Force Stop or you stop the Android Support or restart the phone. The problem was not fixed for me by these measures.

In both chats my last message remained in the state with spinning circle. In one chat the problem resolved itself notifying about dozens of unsuccessful attempts to send. After that, sending was working again. The other chat still remains stuck.

@akutuzov
Copy link

akutuzov commented Feb 2, 2019

@strybe same here, all conversations that were stuck before upgrading to 4.32.8 are still stuck, whatever I do.
Seems very weird, considering that the same conversations work perfectly with Signal Desktop.

@greyson-signal
Copy link
Contributor

@akutuzov This is an Android client issue, not a server issue. That's why you don't have any problems on desktop. The library we use to schedule message sends had a bad release that was present in 4.32.7, which caused messages to get stuck. This was fixed in 4.32.8, and any stuck messages should typically be able to get 'unstuck' by restarting your phone or force-stopping signal.

However, it also appears you and @strybe are running SailfishOS. After a message failure, messages are scheduled to be retried on JobScheduler, which is an Android system component. I'm going to guess that SailfishOS has some bug, or otherwise undesireable behavior, in their JobScheduler implementation. I'm basing that on the premise that the only users I've seen who haven't been able to recover are either running rooted devices without Google Play Services or SailfishOS.

That said, I'm not trying to be dismissive. WorkManager (the library we use for scheduling message sends and stuff) has been a real pain point lately, and I'm looking for ways to resolve some of these issues we're running into.

@akutuzov
Copy link

akutuzov commented Feb 2, 2019

Thanks @greyson-signal, you are doing a great work!

Yes, I wouldn't be surprised if Sailfish had a bug in Dalvik (their Android compatibility layer). That's why I asked is there any way to 'reset' the message queue for a particular conversation from Signal itself.
I suspect it should in theory be possible, since if a conversation with a contact X is stuck, I am able to create a new group chat including only me and X, and it works fine, messages are sent without any problems. Would be nice if instead of this one could simply 'reset' a conversation.

@strybe
Copy link

strybe commented Feb 2, 2019

Thanks @greyson-signal for your time and effort!

This was fixed in 4.32.8, and any stuck messages should typically be able to get 'unstuck' by restarting your phone or force-stopping signal.

There is already an option "restart encryption" in Signal chats. I tried this and it did not solve the problem. Maybe the behavior of this option could be expanded to also 'reset' the conversation or message queue as @akutuzov said.

I'm basing that on the premise that the only users I've seen who haven't been able to recover are either running rooted devices without Google Play Services or SailfishOS.

For me, some logs in issue #8559 look like they are normal Android devices with GCM enabled.

@akutuzov
Copy link

akutuzov commented Feb 5, 2019

I confirm that even after upgrading to 4.32.8 new 'stuck' conversations continue to appear. The symptoms are the same: try to send a message (usually a large one, like a picture) at a moment when something is temporarily wrong with the network, like wifi reconnecting to another access point, etc. After that this message is forever in the pending state, and it is impossible to send anything in this conversation.

@greyson-signal I suppose this issue should be reopened then?

@greyson-signal
Copy link
Contributor

At this point this issue seems to be limited to SailfishOS and rooted devices. I don't see logs of 4.32.8 devices in #8559 that aren't rooted, but please point them out if you see them.

Maybe the behavior of this option could be expanded to also 'reset' the conversation

This currently isn't a simple thing to do. WorkManager doesn't make it easy to resubmit an entire work chain.

I'll re-open this issue, but I'm retitling it to be specific to SailfishOS.

@greyson-signal greyson-signal reopened this Feb 5, 2019
@greyson-signal greyson-signal changed the title Messages stuck in retry loop Messages get stuck after retries on SailfishOS Feb 5, 2019
@akutuzov
Copy link

akutuzov commented Feb 5, 2019

This currently isn't a simple thing to do. WorkManager doesn't make it easy to resubmit an entire work chain.

Does it allow to open a new work chain? Like it happens if one creates a new group chat, for instance?

@Furocious
Copy link

I've had this problem for some time now (it started with the beta version where typing indicators was introduced) and I am using Android 6.0.1 (MotoG3 without root). Currently running version 4.33.5 and I have tried everything except wiping the app data. Curiously, this is happening only on one device. I have another unrooted Android 6.0.1 device (Xperia z3c) and only the MotoG3 exhibits this problem.

It usually starts after trying to send a gif or image file. Another Signal user (unrooted also) has also indicated that they had problems trying to send image files to me.

In essence, this problem is not limited to SailfishOS.

@greyson-signal
Copy link
Contributor

Merging this into #8602

@schmeat
Copy link
Author

schmeat commented Mar 9, 2019

I got this again on 4.35.1

https://debuglogs.org/f3a0ecc2ceb108fb52aa88a0740b9b826eff33b815d4744d78a4ad4b1d8d5d42

Same device info as above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

7 participants