Equal not dominates and Merge

[martinsumner]

In a standard PUT situation, when fetching the local object to compare with an inbound object, it would be unexpected to find the objects have an equal vector clock.

Currently, with the leveled backend an efficiency is gained by doing a HEAD request as part of the PUT, and only when the vector clock of the inbound object does NOT dominate the local object a GET will be required. When a GET is required the objects are merged, rather than the local object simply being replaced.

See test - in situation where vclocks are equal, neither are dominant.

equals_or_dominates_test() ->
    A = vclock:fresh(),
    A1 = vclock:increment(a, A),
    A2 = vclock:increment(b, A1),
    A3 = vclock:increment(a, A2),
    B4 = vclock:increment(b, A3),
    B5 = vclock:increment(b, A3),
    ?assert(vclock:dominates(B4, A3)),
    ?assertNot(vclock:dominates(A3, B4)),
    ?assert(vclock:equal(B4, B5)),
    ?assertNot(vclock:dominates(B4, B5)),
    ?assertNot(vclock:dominates(B5, B4)).

There are two handoff scenarios though where vclock equality is normal:

• When hinted handoffs are running after an outage where the read_repair_primaryonly is enabled (the default setting). In this case objects that were read, but not updated, during the outage will be handed off from the fallbacks - and each will require a GET - Object Merge - Discard (as the inbound vector clock does not dominate the local clock, but the merged object is equal to the local object) not the more optimal HEAD - confirm clock equality - Ignore.

• When an ownership transfer has failed, then restarted. All the objects that had been passed in the aborted transfer will be re-sent in the re-started transfer, and all will require a Get/Merge/Discard (as above). As the cost of the GET is potentially more expensive than the HEAD, then there may be a scenario where ownership handoffs never complete as they always fail due to the cost of re-visiting previously handed-off objects, leading to continuous aborts.

There is uncertainty though about how safe it is assume that true = vclock:equal(VA, VB) means that ObjA == ObjB. For normal API-driven PUTs, the scenario is sufficiently rare and unexpected to warrant additional processing - but is this the case in handoffs?

Current consideration is that this may be too risky to change as a default for the standard PUT path. Just because the scenarios where clock equality may occur without object equality are not obvious - it doesn't mean they do not exist. Would ignoring inbound objects on vector clock equality (to locally stored object) be an acceptable risk specifically for handoffs?