Certain relationships fail to resolve when initially downloading entire data set

[jcmosc]

I have managed to reproduce this with a data set and a known from/to object pair.

So far I have narrowed this down to a a condition in SPRelationshipResolver.m line 170, where fromObject is nil somehow. If I examine the logs up to this point it I can see the expected from object in question being inserted into the managed object context, before the missing relationship is resolved.

I think it has to do with the fact that the relationships are resolved per batch, and each batch uses a different thread safe child context (?), and all the dispatch_async calls causing out-of-order behaviour. Maybe by the time the relationship tries to resolve the previous batch hasn't completed saving the newly inserted object into the persistent store coordinator??

I have various logs, but they're lengthy ...

[jleandroperez]

@jamesmoschou hi there James!,

Would you please verify if you've got the patches pushed in these issues?. There were a couple glitches in the index-batch-processing logic that caused buckets not being processed in batch, plus, the logic to hit the index finished processing was flawed, and the delegate chain was getting called before the right time.

• Issue Race condition in index-batch processing #239
• Issue Index entities are not being processed in batch #238

Thanks!

Update: After checking + rechecking the flows, the commits for 238 + 239 should minimize the occurrence rate of this bug. But i'm not quite sure if they would completely fix it, i'll be debugging this!

[jcmosc]

@jleandroperez Hi Jorge,

Yes, I've merged in those updates to our branch. It seemed to improve the situation, but not fix it completely.

Let me know if you need anything else from us, cheers!

[jleandroperez]

@jamesmoschou hi there James!

May i ask you if the object that's causing you issues has a bidirectional relationship? (or it just doesn't have the inverse set?).

Thanks!

[jcmosc]

@jleandroperez Hi Jorge

It's a to-many relationship A -->> B where it's receiving B and trying to resolve the relationship back to A before receiving A.

[jleandroperez]

@jamesmoschou thanks James!

I've sent, yesterday, this pull request

If / whenever possible, would you please confirm if the problem is gone?.

The PR isn't 100% ready yet: i'll be adding a stress test to trigger the NIL glitch, and some things will probably change. But the raw idea is to prevent any race conditions between Root / Child order of insertion (+ any weird threading issues).

Thanks!

[CapoChino]

Hi @jleandroperez,

What's the status of this bug/pull request? I hit this bug pretty regularly and don't want to ship my app with this bug. Should I also test your pull request or do you have some changes in the works?

Thanks!

[jleandroperez]

@CapoChino hey there!. No new progress on this bug, for now. We'll be addressing this one very shortly (for the time being, the only fix is the pull request mentioned above!).

I'll make sure to keep you posted, thank you!

[CapoChino]

Hi @jleandroperez,

I've done some testing with the issues/250-null-relationships branch. It seemed to fix the immediate problem in this bug. My database in the cloud downloaded fully and correctly to a clean device.

However, I'm having a couple problems, which might relate to changes in this branch.

Problem 1) When downloading is happening, it's possible that the user creates some changes to the db. If an object is changed during syncing, it sometimes ends up with some of the relationships set to nil. The setting of the relationships to nil is registered as a change by the server, suggesting the faulty change is being generated by the client. Perhaps this is the SPRelationshipResolver getting confused by having both incoming (remote) and local changes?

Problem 2) When I'm creating then removing objects, I get the following errors:

2014-07-19 18:05:34.157 RaceTrace[90486:8e2b] Simperium Received Error [The operation couldn’t be completed. (SPChangeProcessor error 2.)] for object with key [8505db5512cf410787f52bb307ff0c8e]

then

2014-07-19 18:05:34.235 RaceTrace[90486:8a0b] Simperium warning: couldn't apply change due to version mismatch (duplicate? start 33, old 0): change {
ccids = (
47e13436d42b476bbf0fcd8db4252d0f
);
clientid = "ios-47272d436ffd40278343779bf8a22ed6";
cv = 53cb15dea5cc2b6e709b4822;
ev = 34;
id = 393fae6da62c4fdd967eb7ed7f00ad37;
o = M;
sv = 33;
v = {
childCount = {
o = r;
v = 104;
};
...

and finally:

2014-07-19 18:05:34.355 RaceTrace[90486:5b43] *** Terminating app due to uncaught exception 'NSObjectInaccessibleException', reason: 'CoreData could not fulfill a fault for '0xb6ee810 x-coredata://DDBE7885-F897-4F06-8239-77EB50FE3B2E/MyEntity/p547''
*** First throw call stack:
(
0 CoreFoundation 0x036a21e4 __exceptionPreprocess + 180
1 libobjc.A.dylib 0x02d038e5 objc_exception_throw + 44
2 CoreData 0x00d25beb _PFFaultHandlerLookupRow + 2715
3 CoreData 0x00d25147 -[NSFaultHandler fulfillFault:withContext:forIndex:] + 39
4 CoreData 0x00d24d23 _PF_FulfillDeferredFault + 259
5 CoreData 0x00dce363 _sharedIMPL_vfk_core + 147
6 CoreData 0x00dd01f1 __generateAccessor_block_invoke + 33
7 RaceTrace 0x001a238e -[SPMemberEntity simperiumKeyForObject:] + 78
8 RaceTrace 0x001a2ef4 -[SPMemberEntity setValue:forKey:inDictionary:] + 148
9 RaceTrace 0x001675b1 -[SPMember diffForAddition:] + 177
10 RaceTrace 0x001c1d99 -[SPDiffer diffFromDictionary:toObject:] + 1417
11 RaceTrace 0x0019495e -[SPChangeProcessor processLocalObjectsWithKeys:bucket:] + 2350
12 RaceTrace 0x00196dfb -[SPChangeProcessor enumerateQueuedChangesForBucket:block:] + 1643
13 RaceTrace 0x001e4958 __43-[SPWebSocketChannel sendChangesForBucket:]_block_invoke321 + 184
14 libdispatch.dylib 0x030927b8 _dispatch_call_block_and_release + 15
15 libdispatch.dylib 0x030a74d0 _dispatch_client_callout + 14
16 libdispatch.dylib 0x03095047 _dispatch_queue_drain + 452
17 libdispatch.dylib 0x03094e42 _dispatch_queue_invoke + 128
18 libdispatch.dylib 0x03095de2 _dispatch_root_queue_drain + 78
19 libdispatch.dylib 0x03096127 _dispatch_worker_thread2 + 39
20 libsystem_pthread.dylib 0x033d6dab _pthread_wqthread + 336
21 libsystem_pthread.dylib 0x033dacce start_wqthread + 30
)

Perhaps the incoming deletion change (or another change after it) is trying to fault in the object that's already been locally removed? I found this "TODO" in SPChangeProcessor.m to be very suspicious:

    // TODO: If this isn't a deletion change, but there's a deletion change pending, then ignore this change
    // Change was awaiting acknowledgement; safe now to remove from changesPending

[jleandroperez]

Hello @CapoChino,

Thanks for testing the branch!. Few questions/comments below.

1. Nil Relationships after updates:

   I've been testing with a sample app, and i couldn't manage to reproduce this. Couple questions:

   • Would it be possible to capture the logs (outgoing + incoming)?.
   • Are you working on a derivedMOC?
   • Any chances you're saving an instance of the object, with a NIL relationship, while the relationship has been already established on the main MOC? (that would generate a changeset, nuking the relationship).

2. Log messages:

   1. Error 2 (SPProcessorErrorsReceivedZombieChange):
      This means that the backend confirmed a change, while the object was already nuked locally.
      This should be treated as a warning, and should not have a negative impact.
   2. Version Mismatch:
      Would it be possible to capture the logs? (ADD message sent + server response)

3. Crash:

   • The TODO you've spotted is actually an improvement, and should not be the source of the crash.
   • We're using a Mutex to deal with deletions. Whenever a deletion changeset comes through, any other operation should finish. (This is also valid for the relationships resolver).
   • It would appear that the object is being processed for Addition (while it was already nuked). Is it possible you're using Simperium's writerMOC?
   • Would you please describe the steps (+ Entities + Relationships involved)?

Again, many thanks in advance!!

[jleandroperez]

@CapoChino hey there!

Relationships Resolver branch was just merged into develop. If you're still seeing nil relationships after the initial dataset sync, please, feel free to reopen.

As for the rest of the glitches you've mentioned, likewise, if after this merge you still see any of them, please, feel free to open a new issues, as required. We'd be more than happy to help you out!

Thank you!