Improve message download speed - startup is slow

[scottnonnenberg-signal]

If a desktop instance has not been started in a long time, it may have up to 1000 messages in its server queue. It can take minutes to get through all of these messages. Desktop needs to process messages faster to improve startup times.

[MartinX3]

Do you need something from us to work with? :)

[scottnonnenberg-signal]

Nope, we've got plenty to work from!

[JanZerebecki]

The more pressing issue it that it shouldn't block startup as in typing up a new message (it doesn't on mobile, though speed improvement of a big queue would be beneficial there, too).

[MartinX3]

@JanZerebecki
but another issue is, that a message you typed will not be the last one in the list, if you get older messages synced after sending it. :)

[Pilvinen]

This is absolutely ridiculous. You need to redo the design of Signal from scratch in such a manner that the startup doesn't take 10 minutes. And who the hell needs 1000 previous messages? The desktop app is horrible to use. I read the previous - closed (wtf?) - issues on this and understand what the issue is all about - but I hope that you realize that the desktop app is 100% unusable like this.

This is not the 80'ies anymore, we're not used to Commodore 64 loading times.

You've got to do better than this. It's absolutely ridiculous.

[janvlug]

@Pilvinen I understand your frustration, but your wording is not of any help.
Interesting enough, I have a flatpak Signal Desktop running on three system, and only one of them has the extremely slow startup time. So the issue is not omnipresent.

[Aspire1Inspire2]

I always have 1000 messages to load when opening the app. It gives me 19 mins loading time last time.

It would be awesome if we could choose how many to load on start up, instead of loading 1000 every time.

Thanks for your work!

[4mig4]

I do have the same problem here on Linux desktop app 1.20.0

If you need more info please let me know what you want me to look at

[Sajo8]

@Aspire1Inspire2 Choosing how much you want I think isn't good; I'd rather a set amount load, (maybe the last few hundred) automatically and the rest load on demand when a user scrolls up, in set increments of something like 100 messages for that convo/group convo

[janLo]

I don't understand how it can always load abot 1000 messages when I definitely have exchanged a few hundred at max. since I started signal last.

[binarious]

@janLo @Aspire1Inspire2 As said in the other threads about this: This is not only about received text messages. These "messages" contain metadata and other application-relevant information that has to be synced. Therefore it probably doesn't work to limit the amount of messages.

I personally don't get why the UI has to wait for this "messages" to be received. Can't the UI be updated when these messages arrive in the background?

[scottnonnenberg-signal]

I appreciate that everyone is frustrated, but comments 'why can't it just be like X' or 'I don't understand why X' really don't contribute to the solution. Please limit your comments to things which might help us make progress on this. Specific performance metrics, debug logs, etc.

[binarious]

@scottnonnenberg-signal I was asking, because I wanted to know if there are technical limitations which lead to this design decision. If the answer is no and you're willing to accept PRs then I'm happy to help.

[scottnonnenberg-signal]

@binarious I appreciate your enthusiasm. Large changes, even architectural changes, will be required to fix this. It's highly unlikely that a third-party pull request will be able to help. The only way that would work is if I'm in close contact with the author before and during the creation of that pull request. Even then, there's a lot to learn about this application.

[darrellenns]

@scottnonnenberg Are there any particular metrics/logs that would be helpful? I'm guessing this is a client-side issue (since the mobile clients are not affected). Do we need to do some profiling of the client? Or do we already know where the performance issues are originating?

[scottnonnenberg-signal]

@darrellenns We have a really good idea where the problems are. But additional logs from situations where your desktop has just downloaded 1000 messages might still be useful. Especially on Linux.

[maxzinkus]

Perhaps a fixed number of messages, or a fixed range of time (e.g. past 30 days, maybe customizable in settings) can be eagerly loaded and decrypted at startup, and the rest decrypted lazily as needed?

[scottnonnenberg-signal]

@maxzinkus Since we haven't yet had this discussion on this particular thread, I'll explain. We must process all messages in order for encryption reasons. And, keep in mind that not all messages are actually messages written by your contacts - they can be state updates for groups, for read status from your other devices, etc. We must process all messages to have consistent state, and we must process them in order.

Could we show the UI while downloading all of those messages? Maybe, but it can result in quite odd situations, for example a sent message shown above older incoming messages in the conversation thread. For now, we're going to focus on improving download time.

[maxzinkus]

Ah, that makes sense - the entire KDF chain has to be unspooled. Thanks for the reply.

[veeara282]

@scottnonnenberg-signal Is there any way to optimize this process? I imagine that simply compressing it wouldn't help.

Noob question: Is this because every message in a conversation is encrypted using the previous message?

[MarkAtwood]

Do you still need a log? Why doesn't local caching work? It takes 20 minutes for Signal Desktop to start on my Mac. If I quit out of it and start it again, it takes that long again for it to redownload 1000 messages again.

[4mig4]

I think enough people complained (including myself) about this problem and that themselves know this is a problem in the Desktop App that they are working on fixing. Surely the team understands that people prefer to use a keyboard on a desktop PC while they are using their Desktop PCs than using their Smartphone. The problem is easily replicated on any platform MAC , LINUX or Windows I believe.

[scottnonnenberg-signal]

@MarkAtwood Your situation does appear to be quite odd. Unless you're receiving messages so quickly that you do have 1000 messages to download every time you start up. Note that this number always includes communication between devices that aren't shown to you as messages. You might consider entering a new bug.

[scottnonnenberg-signal]

All: This Github issue isn't about debating whether or not performance is important. It's important, that's why we created a separate issue to track our work on it.

Please really consider whether your comment is warranted here, or should be a discussion on the forums (https://whispersystems.discoursehosting.net/), or should be a separate bug entirely. I don't want to lock this conversation, but I may have to.

[lanort]

I have uploaded another debug log for you. , if you need it. It took about 25 minutes to load the app. The highest number I saw for the loading message was over 1.000 (I think around 1.050 or so).

[pjedge]

In the case where a mobile device is also online, would it be possible for the desktop instance to bootstrap off processed messages from a mobile phone somehow to improve startup speed?

[scottnonnenberg-signal]

@pjedge No, not possible.

[veeara282]

Why not spin it off into a background process that downloads and decodes new messages whenever the computer is idle, even when the app is not running?

[thesilentonetoo]

~15 min startup log Win7

[malkhamis]

There is an issue that kinda concerns me from a security prospective.. Consider the following scenario, which is what I usually do to circumvent the issue of the UI blocking me from using Signal until every thing is synced:

1. A specific chat, called XXX in Signal mobile is set with disappearing messages time of 1 hour
2. Signal Desktop was previously run inside a virtual machine
3. The virtual machine running the Signal Desktop was closed by "Saving the State" of the VM. I do it this way for two reasons: first, I use Fedora, which is RPM-based and, second, I don't want to wait forever for Signal Desktop to sync gazillion messages. This way, when I start the VM again, the UI will be present and the sync will happen while I am free to send messages
4. In chat XXX, a message is sent
5. An hour later, the message in chat XXX disappears
6. The VM running Signal Desktop is now launched after the message I sent earlier in chat XXX disappeared from Signal mobile.
7. All of the suddent, I could observe all the disappeared messages from Signal mobile showing up for a split of a second in Signal Desktop and then disappearing immediately.

This certainly means the disappeared messages still exist in the Signal mobile database and are never cleared. Indeed, I have very few messages in my chats, but the cache size is several gigabytes in my phone! Clearing the cache reduces the time needed for Signal Desktop to launch

[scottnonnenberg-signal]

@malkhamis That's a discussion best left to the forums, since it only peripherally deals with performance and is not about a bug but about your concern about how we've done something. https://whispersystems.discoursehosting.net/

"This certainly means the disappeared messages still exist in the Signal mobile database and are never cleared"

I'm really not sure how you came to this conclusion - when you make your forum post, provide some more reasoning for it.

[malkhamis]

and is not about a bug but about your concern about how we've done something.

@scottnonnenberg So you want to tell me that it is ok for disappeared messages to somehow reappear and that is not a bug??

I'm really not sure how you came to this conclusion - when you make your forum post, provide some more reasoning for it.

I have laid out how I came up to this conclusion!!! I am not sure what else you were expecting!
V
V
V

• A specific chat, called XXX in Signal mobile is set with disappearing messages time of 1 hour
• Signal Desktop was previously run inside a virtual machine
• The virtual machine running the Signal Desktop was closed by "Saving the State" of the VM. I do it this way for two reasons: first, I use Fedora, which is RPM-based and, second, I don't want to wait forever for Signal Desktop to sync gazillion messages. This way, when I start the VM again, the UI will be present and the sync will happen while I am free to send messages
• In chat XXX, a message is sent
• An hour later, the message in chat XXX disappears
• The VM running Signal Desktop is now launched after the message I sent earlier in chat XXX disappeared from Signal mobile.
• All of the suddent, I could observe all the disappeared messages from Signal mobile showing up for a split of a second in Signal Desktop and then disappearing immediately.

Anyway, this might be a bit off-topic, but I thought it is worth mentioning hoping that this issue is considered along any decision for fixing the performance issue. Cheers

[lesurp]

Could we simply have an option to disable the sync? It's annoying not to be able to lock the app, meaning my private messages are virtually available to all my coworkers during my lunch break. An option such as this would improve both security and usability.

[scottnonnenberg-signal]

This conversation is now locked to contributors.

@plesur That is a separate feature request, which is already being tracked in this project.

@malkhamis That data you've presented absolutely does not mean that messages are kept in the signal mobile database and never cleared. Please go over your logic once more, try to avoid phrases which are unnecessarily inflammatory (like 'disappearing messages are never cleared'), and then take that conversation to the forums.

[scottnonnenberg-signal]

If you've previously commented here, you're interested in performance. v1.22.0-beta.1 was released yesterday, and has a number of performance enhancements which should make startup much faster: https://github.com/signalapp/Signal-Desktop/releases/tag/v1.22.0-beta.1

If you're not already on the beta, you can find out how to install it here: https://github.com/signalapp/Signal-Desktop#install-the-beta

If you're currently on the production build, you can switch from production to the beta by shutting down, then renaming the Signal Desktop data directory from 'Signal' to 'Signal Beta,' then starting up the Beta.