Poor sync performance under high count of unread articles

[godfuture]

Problem
(1) Syncing of articles seems to be slow. This is not noticeable when a "full sync" was done and only updates are received later on. But the full update itself (client is empty, server holds around 20000 unread articles) takes a lot of time (around 8h with 54mbits connection bandwith).
(2) When the app crashed (or force closing) during full update, only newly (since initial sync start) added feeds on server are fetched on next sync. What are the symptoms of this problem? The count of unread articles stucks at maybe 7000 articles and starred items are empty. Some hours later, the sync only fetches few articles that were published since last sync.

What is transmitted from server to android client? JSON, XML?
How is this data stored in client? DB, XML, own data structure?
How could the sync be done faster? Introduce data Compression, change sync strategy?
Why does an interrupt of the "full update" break the sync of existing, outstanding feeds?
Is the android client done to handle such an amout of feeds? If not, what could be done to avoid problems?

Server: IIS 7.5, OC 6.0.3, PHP 5.4
Client: NewsApp 0.6.6, Android 4.4.2, CM11

[David-Development]

Thank you for the report. I already noticed this issue last weekend when I synchronized with my ownCloud (6500+ unread feeds). I'm already working on this issue. Probably I can release an update within the next days.

The data is transmitted via JSON, and it's stored in an SQLite Database. No data compression is used for the transmission.

[David-Development]

Okay.. I improved the code and it's possible now to sync ~5800 articles in 4 minutes and 30 seconds. I think this is an acceptable result, isn't it? The update is not finished yet but I'm going to release it as soon as possible (probably this weekend).

[godfuture]

As you already use JSON, the transmitted amount of data shouldn't be blown up compared to XML, so quite impressive improvement. Did you test the second problem as well? When you force close the app while syncing (at least initially) you will have problems getting the rest of unsynced feeds to client.

Cheers

[David-Development]

I know that issue number 2 exists but it's not that easy to find a clean solution for the problem. So I'm going to fix issue maybe later. But it hasn't such a high priority for me because when you clear the cache you can start from scratch if the sync fails.