Eliminate loading screen, load new messages in the background

[TurkeyMan]

• I have searched open and closed issues for duplicates

---

Bug description

I sleep my computer when it's off.
I often wake my computer from sleep, and then Signal plays catch-up with all the activity from my phone or other clients.
Every single message received while my computer was asleep shows a pop-up notification, and makes the new-message 'ding-a-ling' noise.
Today, I have hundreds in the catch-up queue. I've been sitting here for over 5 minutes so far just hearing my computer ding and spamming notifications about messages I received the last couple of days.

Does this happen if my computer is off rather than asleep? I can imagine perhaps, since the client was launched before the messages were received, that when I wake the computer, it thinks the messages were received during this active session?

Steps to reproduce

• Run the desktop client (I'm using electron, but i think chrome works too)
• Sleep the PC
• Do activity with your phone
• Wake the PC, prepare for a torrent of activity...

Actual result: Torrent of catch-up activity; very annoying
Expected result: Signal should know my computer was off, and not give me notifications that have already been read on other devices.

Platform info

Operating System: Win10 64
Browser: Electron
Signal version: [...where is the version information on the electron client?]

[scottnonnenberg]

First, you can get the version information from the top of the debug log.

Sadly, we have a bit of a problem implementing this today. We don't know how many messages are waiting for us in the queue on the server, so we don't know a massive torrent is coming. And today we have nothing attempting to detect full application sleep.

So perhaps we can talk about the expected user experience. Would you be okay if the loading screen came back when we were able to detect that we had 20+ messages to pull down? Or would you like the UI to be changing real-time as you navigated around (unread counts going both up and down, new messages showing up unread, then being marked as read, etc.), just no notifications?

[TurkeyMan]

I reckon if I were designing the experience, I'd probably do something like; when there is data to fetch from the server, fetch all of it in the background until it's fully synchronised. Then, now that all new messages are available locally, apply the UI update in one hit? Ie, all the new messages would appear in the UI at the same time.
There are 2 problems this would mitigate:

1. The contacts on the left are ordered relative to most recent interaction, and that bounces around like crazy when synchronising
2. Each new message produces a notification and a noise. If the program fully synchronises before emitting notifications, it can estimate that I don't want more than [magic number] notifications in one hit, or maybe even just emit one notification saying "many new messages".

It seems fetching from the signal server is surprisingly slow, so perhaps a piece of UI to indicate that a synchronisation is ongoing in the background would be a good hint to users that there is background activity and they should expect new messages to appear soon.

Now the reason I suggest to maintain a usable UI even though there is pending background activity, is that the user may want to author+send new messages while the background sync is ongoing. I can't imagine why that would be impossible...? Is it possible to send a message immediately even if the client hasn't yet 'caught up'? If not, then the client should obviously take my authored message(/s) and send after sync has concluded.

[skulumani]

Is it possible to send a message immediately even if the client hasn't yet 'caught up'? If not, then the client should obviously take my authored message(/s) and send after sync has concluded.

Frequently I'll try to send a message while the app is still synchronizing in the background. I will inadvertently send a message (current time) and then the previous messages from the contact will arrive on the desktop. This will create a situation where the newest sent message is displayed before the older messages from the contact.

The sent message will only have 1 check mark, signifying it's been sent and not received. Most of the time I'll notice this and be unsure if the recipient has the message (due to some logic to prevent out of order messages on the Android?) and have to resend once everything is updated.

Now my desktop and android message timelines have diverged and it's confusing for a short time until everything gets synced back up.

[deutrino]

I noted in the Chromium app that the syncing process is hidden behind the curtain when the machine has been off, but not when it has slept, as is noted above, but I wanted to clarify that bit.

Either way it's annoying, and the biggest annoyance is the glacially slow speed. I was away from my computer for over a week using Signal the whole time on my phone, and it literally took nearly an hour to sync when I turned the computer back on. I know there are protocol limitations influencing this, but it's really not something that could be explained to 99% of users when every other messaging app Just Works in the same scenario.

So if I were project manager, the absolute highest priority would be making the sync happen as quickly as possible. I don't know how to do that without leaking information - would the server volunteering "hey, there are 500 messages in the queue" be too much of a leak? - but hopefully it is possible.

As far as UI, seeing the entire song and dance and hiding it behind the curtain are both bad. The former makes the desktop client difficult to use for the duration, and the latter makes it impossible to use. I suspect there aren't any easy solutions to this, but for now, it would be somewhat less unpleasant if messages sent AFTER the date of messages being received as part of a sync, would stay on the bottom of the scroll.

[TurkeyMan]

but for now, it would be somewhat less unpleasant if messages sent AFTER the date of messages being received as part of a sync, would stay on the bottom of the scroll.

This.

The former makes the desktop client difficult to use for the duration, and the latter makes it impossible to use.

The latter makes it impossible to use under the assumption that the sync will take an hour... if the sync was 'fast' then it it's not a problem at all, or is there some other problem with the experience that I'm not thinking about?

[deutrino]

Apologies if this is duplicate info. I wanted to characterize the behavior I'm seeing in Electron a bit better. I'm a pretty heavy Signal user; between my contacts and the one messaging group I'm in, it's typical for me to exchange hundreds of messages per day. This is a major pain point for me.

I'm experiencing the problem in this ticket 100% reproducibly on Electron. It is always glacially slow to sync a message backlog when the Electron client hasn't been running while the messages were exchanged on other devices. The UX varies depending on whether the client was shut down or running but with the computer asleep. If the client was shut down, the "loading messages" interstitial appears. If the client was running on a sleeping computer, the messages are replayed, generating huge numbers of notifications and making the app essentially unusable for messaging contacts for whom there is a backlog being replayed. This has a severe negative impact to the usability of the app during the replaying, which can take tens of minutes.

STEPS TO REPRODUCE:

1. Use the Electron client and the Android client. These function as expected when both are running.
2. Shut down the Electron client, or leave it running but sleep the computer on which it is running.
3. Use the Android client normally.
4. Start the Electron client, or wake the computer on which it is running.

WHAT ACTUALLY HAPPENS:

If Electron client was shut down:

5. The "loading messages" interstitial pops up.
6. Messages load very very very slowly, VERY slowly, taking tens of minutes for an evening's worth of chatting with a couple friends. I am unsure if the number of messages the app is claiming to load is accurate - someone mentioned in a related ticket that they suspect it is not.
7. Eventually the app finishes loading the messages and functions normally.

If Electron client was running but computer was asleep:

5. The app UI appears normally (no interstitial, etc).
6. Messages load very very very slowly as described above.
7. Notifications display (Linux Mint 18) for most or all messages as they replay, which is very annoying.
8. Each message is put into the contact pane as it would be if it had just arrived in real time (i.e. not as part of a backlog). This makes the app difficult or impossible to use for messaging contacts with whom messages were exchanged when Electron was sleeping, as it is not possible to see what you've sent or to see replies from your contact once backlogged messages come in and the newer messages scroll out of view.
9. Once the messages are replayed, the app functions normally.

WHAT I EXPECT TO HAPPEN

(Please note, I know there are fairly fundamental issues behind this problem, I write these from the "it should Just Work™" user perspective.)

A. The app should sync the entire backlog of messages in a reasonable time - I would pick 30 seconds as a goal to shoot for, because I suspect that users other than just myself will start to get more and more annoyed beyond about 30 seconds.
B. If A cannot be accomplished, app should not spew messages into the panes of individual contacts when the client returns after a computer sleep, as this makes chatting with said contacts nearly impossible.
C. If A cannot be accomplished, app should not spew notifications for messages as they are synced.

Regarding "what I expect to happen," I understand there are protocol issues in the way of point A. If I understand correctly, part of this is that the desktop app doesn't have a way to know that the messages are part of a backlog, which makes points B and C quite difficult?

Even implementing point C alone would be a considerable improvement - because Electron is popping up hundreds of notifications, the aggravation is leaking outside of the app itself and negatively impacting the usability of my entire computer.

Anyway, hopefully a little more detail is helpful, let me know if there is any logging or testing I can help with. Thanks!

[scottnonnenberg]

@deutrino Thanks for the additional detail. You'll be happy to see that Electron v1.0.27 implements your option C: #1507

[deutrino]

I just had my laptop powered down for about 2 hours.

• During that time, I exchanged approximately 75 messages while using Android (I went back and counted)
• Upon booting my laptop, the Electron client took over 17 minutes to sync
• The "n so far" message count crawled slowly up to 220 (about 75 * 3??) before the sync was complete.

https://gist.github.com/deutrino/3e771ef1f00eec7a96a9bafe6ac2caee

This is really, really bad. And not even remotely out of the ordinary for that volume of messages.

Edit: Forgot to add, the signal-desktop process with the lower pid shows constant CPU usage the entire time, usually about 20-25% of a core. Also, I am starting to wonder more about #1495, I am a really heavy user so maybe that plays a part.

[scottnonnenberg]

@deutrino I took a look at your log, and there's a lot to digest.

First, there are a bunch of these errors, which makes me wonder about the health of your database. How big is your ~/.config/Signal directory?

Error: Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing.
    at Driver.update (file:///opt/Signal/resources/app.asar/js/components.js:21629:44)
    at Driver.execute (file:///opt/Signal/resources/app.asar/js/components.js:21587:22)
    at ExecutionQueue.execute (file:///opt/Signal/resources/app.asar/js/components.js:21941:29)
    at next (file:///opt/Signal/resources/app.asar/js/components.js:22021:34)
    at child.sync (file:///opt/Signal/resources/app.asar/js/components.js:22027:13)
    at child.sync (file:///opt/Signal/resources/app.asar/js/components.js:20052:28)
    at child.save (file:///opt/Signal/resources/app.asar/js/components.js:20259:18)
    at file:///opt/Signal/resources/app.asar/js/signal_protocol_store.js:800:36
    at SignalProtocolStore.addUnprocessed (file:///opt/Signal/resources/app.asar/js/signal_protocol_store.js:798:20)
    at Object.add (file:///opt/Signal/resources/app.asar/js/libtextsecure.js:37148:48)

Those are all in the session right before the slow load you're talking about, but they do help set the context. Not long after the start of the slow startup in question, there was 30-second period where nothing was logged:

INFO  2017-10-26T01:44:42.885Z queueing envelope +[REDACTED]108.1 1508972281775
INFO  2017-10-26T01:45:16.369Z queueing envelope +[REDACTED]935.1 1508972281775

And from then on there is general slowness, which I have to imagine is due to some sort of database difficulty. Either that, or there's generally a lot of stuff going on in the process otherwise. You can see that there are only about five log entries per second after that. Might have something to do with all the message XXX expired entries that come after that. Come to think of it, the 30s hang above could be all that work to grab all the cached envelopes (of course, it was only 11 in your case).

This is an interesting section - the message from entry is a message from someone else, and the sent message to is one synced from one of your other clients. There are twelve seconds between them, and I'm not sure why. Could be even loop saturation from all that expired message querying. It's also interesting to note that the next message expires 2017-10-26T01:53:04.848Z is logged out at 2017-10-26T01:53:14.596Z, 10 seconds after it should have fired. Seems like the whole thing is bogged down.

INFO  2017-10-26T01:53:05.770Z message from +[REDACTED]935.1 1508975673900
INFO  2017-10-26T01:53:06.036Z next message expires 2017-10-26T01:52:42.756Z
INFO  2017-10-26T01:53:06.036Z loading expired messages
INFO  2017-10-26T01:53:06.914Z message 1508377984513 expired
INFO  2017-10-26T01:53:09.947Z next message expires 2017-10-26T01:53:04.848Z
INFO  2017-10-26T01:53:09.948Z loading expired messages
INFO  2017-10-26T01:53:14.596Z next message expires 2017-10-26T01:53:04.848Z
INFO  2017-10-26T01:53:14.598Z loading expired messages
INFO  2017-10-26T01:53:17.052Z sent message to +[REDACTED]005 1508975673900 from +[REDACTED]935.1 1508975673900

I've got two competing theories: a) the database is starting to buckle under the weight of your usage, and b) You're a very heavy disappearing messages user, and that is contributing to database saturation or general event loop saturation.

Some more information on your database size, and then some CPU profiles would be useful. You can open the devtools and collect a javascript profile during a slow startup, and that might help us understand things a little bit better. I'll also take a look at our expiring message handling.

[deutrino]

I have disappearing messages set to 1 week with almost all my contacts, including the ones I message a LOT. ~/.config/Signal is 584M.

I don't have time to poke more at it right this second but here is some further system data. Also, the hard drive is glacially slow, but I looked and nothing was thrashing the disk while it was loading. I will probably have several opportunities in the next few days where my laptop will be offline while I am using Signal on Android, so hopefully I can gather more data.

CPU:       Dual core Intel Core i5-4210U (-HT-MCP-) cache: 3072 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 9578
           clock speeds: max: 2700 MHz 1: 1118 MHz 2: 1186 MHz 3: 1100 MHz
           4: 1575 MHz
Info:      Processes: 265 Uptime: 21:20 Memory: 6232.6/7887.7MB
           Init: systemd runlevel: 5 Gcc sys: 5.4.0
           Client: Shell (bash 4.3.481) inxi: 2.2.35 

[scottnonnenberg]

I just renamed this issue to reflect its current meaning - the loading screen, shown on launch of the app, should be eliminated, and messages can be loaded in the background.

[Lesik]

I find the UX solution of a different instant messenger quite elegant, simply replace the "Signal" text in the top-left with "Syncing..." or something like that.

[scottnonnenberg]

@Lesik That assumes you are able to determine what subset of your contacts have messages needing to be sync'd. The signal server presents us with a simple queue of messages, and we don't know who they are from until we pull them down.

[Lesik]

@scottnonnenberg There seems to be a misunderstanding, my proposal does not require to know what contacts have new messages. I simply suggested that instead of a full-screen loading screen, the "Signal" text in the top-left corner to be replaced with "Syncing...", as other messengers do.

Current UI:

My proposal:

[scottnonnenberg]

@Lesik Thanks for the clarification. The standard problem arises, then, of sending a message in a conversation when there are pending incoming messages - things end up out of order. Perhaps the right tradeoff is the top-level notification you mention, then a popup if you try to send a message when that's showing. You can navigate and compose messages all you want...

[breznak]

Thanks Scott for moving the discussion here.
You guys here are solving the problem to eliminate the loading screen and making the UI more responsive.

My question is on implementation level, why does it take that long to download & decipher the few hundreds (thousands) new messages on the sync? It could be what..a few MB, that's a matter of a minute max on any modern connection and hardware. So what is causing the slow-down we observe? Also, are the messages locally cached? To me it seems I download them all over each time Signal is restared.

Other than that, as I understand it now, all the messages from any of the "linked devices" are kept on the OWS servers (hopefully end-to-end encrypted :) ) and waiting for the sync to the remaining linked device. Then they are deleted? What happens if the linked device never connects anymore, is there a EOL time? Is the server side infrastructure open-source as well? It would be nice to have an option in the app to "wipe all of my data from the servers"...

[scottnonnenberg]

@breznak - Remember, it's not just download. It's download, decrypt, and processing of each message. Including potentially large attachments, which then also need to be decrypted. Of course, that's not a complete explanation, because we could still do better.

Here's the server source code: https://github.com/WhisperSystems/Signal-Server

Regarding, 'wiping your data.' We don't believe it to be a high risk, because the messages are encrypted. Only from/to/timestamp is known to us. Most of the time there aren't many messages there anyway, because clients generally stay up to date, and there's a limit to the messages we save. But if you're concerned about it, you might consider unlinking a desktop client that you don't really use anymore.

[deutrino]

Even given all that work, it's still really really REALLY slow. To the point of, as discussed, tens of minutes or worse for a client that missed a couple hundred messages.

What is the fundamental cause of that - a round trip for every message? The more people adopt the desktop app, the more this pain point is going to get noticed and become a black mark when people decide whether or not to recommend the app.

[scottnonnenberg]

@deutrino Please provide logs when things take a long time. It's very help for us to see what happens on machines other than our own.

Digging a bit deeper, I think some of the undue delay is because of IndexedDB churn - we save to the database before we decrypt, then again, after we decrypt, then finally after we've fully processed the message. But I think our biggest opportunity for improvement here has to do with how we schedule all this work - for protocol reasons we need to decrypt in strict incoming order, but there are other things we can do outside that ordering - downloading attachments, and processing the messages on the application side. Once we break those up into separate queues, I think the whole process will go a lot faster.

Note: I'm not far from locking this conversation, because I think we've covered all the bases. I really don't want to get into bunch of repeated "why is it so bad?" questions. Feel free to add emoji responses, of course - that helps us understand how important things are to y'all.

[deutrino]

Okay. I did provide logs back in November but will try to provide more as, having taken a glance back at that part of the thread, it looks like there was a bunch of unhelpful noise in those logs.

As you know I'm a heavy user with a longstanding desktop install. I'll start from that baseline and then later if you need me to reinstall etc, I'm happy to do that - I'm traveling so turnaround time might be a bit slow but I have a major interest in helping to improve this any way I can. I can gather logs when waking my laptop from sleep with a running client, and/or when starting the client after it's been offline a while, let me know if you have any preference.

I'd be happy to move this to a separate issue, whatever works for you. I have a major interest in seeing improvements to this sync process and would have helped more already, it's just the usual herding of cats problem for me :)

[janvlug]

See also: #2733

[imrekalo]

it's a lot faster now for some reason, even though i had my laptop turned off.

[scottnonnenberg-signal]

@imrekalo Glad to hear you've seen an improvement. Is that a change that came along with the v1.18.0 release?

Also, in the future, please include a debug log whenever reporting performance issues (or really any issue!).

[imrekalo]

@scottnonnenberg-signal i shouldn't have counted my chickens before they were hatched...

tonight my desktop app had to load more than a few messages and it took very long to sync, with signal desktop being unresponsive and cpu running around 90%. when it was responding, i could see that it takes the conversations one after the other and updates them, the messages coming in very slowly. here is the debug log: https://pastebin.com/J7TvtBJq
didn't wanna paste the whole thing here. i also copied the last lines from before sending my laptop to hibernation. yes, i'm running 1.18.0.

please let me do if there's anything i can do, test, whatever. i'm eager to help.

[thecyberd3m0n]

is it possible to transfer messages from mobile client using LAN connection, not from servers? It would be so much faster, and could be also encrypted

If my phone is on 192.168.1.101, and PC on 192.168.1.102, why Signal Desktop must download it from servers?

[MartinX3]

A local discovery service and if it doesn't find the partner, it would speak with the server. :)
Syncthing does the same.

[Liquidmasl]

Pretty sure not the downloading part is time intensive but the decryption is

[MartinX3]

It depends on your CPU, SSD/HDD, Motherboard and Internet speed.

[uUJsd7PX]

How about syncing in the background and updating UI (adding/removing messages) as new information is received?

[aquamammal]

Still waiting for this shit to load up, after finishing this whole thread, lol

[ChrislyBear-GH]

@aquamammal

As far as I have understood it, it's because the encryption works by chaining messages together. So if you have very long message threads it takes forever to decrypt everything from the beginning of time until the present.

And if you only casually open your Signal Desktop from time to time you'll always have to wait until it takes so long you'll just grab your phone anyway.

And they'll tell you that it's not a bug but a feature.

smh

[stephanemagnenat]

I'm still having this issue with 1.25.1. The average CPU load of signal-desktop is less than 20%, and overall my computer is 65% idle. I have gigabit fiber connection to internet, and only some (30-ish, maybe 5 images) messages since last time I started signal desktop. After 5 minutes only 150 messages were loaded (it seems more messages than only the new ones), with a throughput of loading about 30 messages per minute. I of course do not know the technical details of the protocol, but it looks to me that some improvements should be possible.

[disconn3ct]

Same issue, here's more datapoints if you want.

v1.25.1 Linux, running on Chrome OS, Slate tablet (beta channel 75.0.3770.61, with keyboard) using the standard upstream Linux container and the official Linux install docs for Ubuntu/Debian. Debug log from a few minutes after startup.

As a vote in favor of dumping electron in favor of ANYTHING else, its also using a significant amount of memory just idling along. Freshly launched it is over a gig resident, and by the time it has been up a few days it starts to push everything out of ram until it is restarted (at the huge battery/time cost of the startup sync.) And that hits on my osx client as well :(

[haary]

Same with 1.25.2 on Arch Linux. Loading took 10 minutes.

[jmrodri]

Using v1.25.3 on Fedora Linux, I had to change the group to only load 1 week's worth of messages. It still takes a lot longer than say Telegram or other desktop chats. I've gotten to the point where I start it in the morning and go do other things.

[ChrislyBear-GH]

As a vote in favor of dumping electron in favor of ANYTHING else...

When you're designing an application and selecting a platform for it, if your choice falls on Electron you've made a huge mistake or missed something along the way. Electron is always the wrong choice, unless you want to build a Chrome flavoured web browser.

[MartinX3]

Yes, my experience as developer is that electron doesn't give you the performance and the resource usage you need.

[Aspire1Inspire2]

Here is the log file.
Start time is still very slow comparing to other messaging apps.

Yes, it is improved. But the normal bar of "starting IM app" is higher.
Thanks for your great work.

[hpvd]

Slow Startup not only a problem of speed -> loosing messages #3636

[ghost]

Another user here with continuous, long, slow startups and laggy interface due to Electron. However, if I disconnect my internet, start Signal, and then reconnect, startup is magically speedy.

v1.27.4 debuglogs

Please drop electron. Qt Quick or even Java Swing or Java FX for cross-platform use would be preferable.

As for the slow loading messages, the mobile apps don't have such a slow startup, so why does the desktop app? Maybe it's possible to replicate what they're doing on desktop.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.