Sync'ing Stallment

[jleandroperez]

Reference: #351

@martinoneutrino let's move over to this issue, regarding the sync'ing stallment.

I've been reviewing the logs you've provided (thanks for that!). Few comments below:

• I've noticed you're using custom keys, and the keys are not unique across buckets. This could cause side effects when dealing with deletions (Ref. Update SPCoreDataStorage RemotelyDeletedKeys Mechanism #302).
• Are all of the relationships in your model marked as Optional?
• Would you please confirm that you're not accessing the writerManagedObjectContext directly?
• I've noticed that, apparently, the requestVersion:forObjectWithKey: method is called during initial sync. Just to be sure, would you please disable that behavior and test again?
• There are above 400 message errors saying Simperium error: transform diff for a ghost member. Are you using just one device?. If so, we'll need to find the cause for that. That code should not be executed in a single device scenario.

I'll take a deeper look, and report back any new findings.
Thanks!

[martinoneutrino]

• The legacy keys are (previously assigned) UUIDs. I have been using the same initial database to test syncing using different Simperium logins, so perhaps that is the cause. Generally I Delete Users on the server before each sync, but not always.
• Yes, I have made all relationships optional.
• Yes, I've never used writerManagedObjectContext.
• I'm not familiar with that call. How should it be disabled?
• Yes, I am using just one device, but my initial database is coming from the same source, including the (legacy generated) unique keys.

[jleandroperez]

@martinoneutrino hello there,

Upon closer examination of the logfile you provided, we've identified a race condition. Should be fixed now in develop, would it be possible to run a new test?.

Are you sending the app to BG after 5 minutes?. If the app is in foreground, the websocket shouldn't get reconnected. If possible, before running your test, add an extra log sentence here so we can follow the heartbeat.

Regarding the Ghost Transform errors (there are many of those in the logs):

2014-09-19 18:08:47.233 Food Storage[32824:34f] Simperium error: transform diff for a ghost member (ghost <SPGhost: 0x20334e60>, memberData (null)) that doesn't exist (guid):

Is it possible you've either changed the Data Model, and that account has legacy data stored in the backend?

Thank you!

[martinoneutrino]

I don't recall sending the app to BG, and don't believe I did. I have an additional log I'll send; it has the connection closing after nearly 30 minutes, with heartbeats every 30seconds during that time.
I haven't changed the Data Model; I have been doing a full Delete Users every time I have started a new sync test. (Every new test I do also generates a random account.)

[martinoneutrino]

I tried it with commits 96da65c and 8790721 (over v0.6.9), and the result was basically the same (continuing heartbeats), although this instance never finished.
I will try it again with the current develop branch.

[jleandroperez]

@martinoneutrino many thanks for the tests + extra logs!. Please, whenever you try with the latest develop, it'd be super helpful if you could export the logs again, with the latest code.

Thanks!

[martinoneutrino]

I also ran the devel branch, but I don't know if it actually finished correctly or not, since there is no pending changes count.

[jleandroperez]

@martinoneutrino hey there,

We've closed issue #9, and exposed an async call that grants you access to internal sync'ing stats.
(Please, check SPBucket.h, in develop). That way you can easily check if there are local pending changes to be sent (or enqueued!).

I've been through the logs you provided, i would agree with you!, this time the WebSocket connection remained open for several hours, so it's highly likely the end of stream we've been observing was due to a network glitch.

I'm sorry to say that i missed a flow, and the race condition was still present (my bad). May i bother you once again with a quick test? should be 100% good now!.

Thank you!

[jleandroperez]

@martinoneutrino would you please confirm if this issue is fixed for you?

Thank you sir!

[martinoneutrino]

Yes, it worked and completed successfully.
I can't say that I'm terribly excited about the callback (yet). The function is fine; the execution ends up being lacking. I'm going to change my code a little and call it less frequently, to see if I can get better results. (Currently I call it after each didChangeObjectForKey, which might mean about 100 calls over 10 seconds; unfortunately, the 100 callbacks occur later and together over about 4 seconds, and almost all have the same result.)

[jleandroperez]

@martinoneutrino thank you for testing the fix!.

Regarding the Sync Status method, since it's an expensive call (it involves checking the number of elements in three threadsafe collections), adding a bucket level callback would be a terrible idea in terms of performance.

Are you using some sort of locking UI + spinner?

[jleandroperez]

@martinoneutrino since the Sync'ing stallment issue is fixed, i'm closing this ticket.

If needed, please, feel free to open another issue. Any suggestions will be welcome.

Thank you sir!