"producer double-confirming known range" error when testing failover

[jchung00]

Tag: dawn-v4.2.0.
We have two nodes, same producer key and config, and one of them starts with production paused. They both run on the same machine.
Block producing node 1 start producing when the nodes start. Block producing node 2 syncs fine. We run curl -sL http://127.0.0.1:<node-1-port>/v1/producer/pause and curl -sL http://127.0.0.1:<node-2-port>/v1/producer/resume.
Block producing node 2 begins producing with no problems.
Then, we run curl -sL http://127.0.0.1:<node-2-port>/v1/producer/pause and curl -sL http://127.0.0.1:<node-1-port>/v1/producer/resume.
This causes the producer double-confirming known range assertion exception. We checked that this is where the assertion is defined in EOS.IO:

eos/libraries/chain/block_header_state.cpp

Line 158 in 3b70b57

FC_ASSERT( itr->second < result.block_num - h.confirmed, "producer double-confirming known range" );

We also noticed that block producing node 2 can't resync with the chain even when we run it with the --hard-replay-blockchain option.

[gleehokie]

Thanks for the report, I'll bring this to the devs attention.

[wanderingbort]

I will create a more specific issue for this and reference this issue in it. One of the concepts not ready for fail over is the watermark calculated by the producer_plugin. This watermark is used to determine how many blocks to confirm with each produced block. For the particular scenario you describe we can fix this by scanning incoming blocks for producers who are also local producers. That way your backup node will have accurate watermarks Inaccurate watermarks trick on node into double committing past blocks which is a considered Byzantine failure and is why you cannot easily recover

…

On Sat, May 26, 2018, 2:46 PM Greg Lee ***@***.***> wrote: Thanks for the report, I'll bring this to the devs attention. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3442 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACYR4nEsvGMdedtoJLqZWOaWoHncuVy4ks5t2aMMgaJpZM4UO7Ca> .

[noprom]

Look forward to the fix.

[wanderingbort]

We moved this to 1.1 for a few reasons:

1. while this feature is intended to eventually bloom into support for failover its original purpose was not designed with that in mind and there may be several other harder to find issues with having a hot-standby producer for availability.
2. The watermark calculation is conservative when a process starts. So, if we are simulating a dead node where that original node was restarted this would not have occurred. It is only when we are doing live-to-live-to-live node hand-offs with no process restarting that we would see this bug.

This is not to say it isn't something we need to fix. This will get better in short order. However, I wanted to enumerate some of the reasons why we have slipped it to version 1.1

[abourget]

Bart, I have a suggestion. The resume should ask for a "proof of resignation" from another node that was signing with the same key, and the resume wouldn't kick on if that proof wasn't valid.

Ex:

• node no. 1 produces blocks
• operator wants to handoff to node no. 2
• sends a signal to node no. 1 with pause, which returns some data about its last state, watermark or whatnot.. could be base64-encoded JSON or binary or whatever
• send a signal to node no. 2 with resume, along with the payload from pause. The resume operation wouldn't unlock without a message from the previous producer that it is safe: the last block signed, a notice that production has stopped over there with the key used.
• same thing if we want to come back.. and perhaps the watermark or whatever data could be passed back if we want live-to-live-to-live handoff..

Can you imagine something like this ? This would be to lower the risk of double-signing blocks.

Some special payload could unconditionally resume a chain, if you didn't stop production from a node previously. That would be explicit, and you'd need to check you didn't have a node previously.

What do you think ?

[abourget]

sort of an out-of-band sync'ing of node production :)

[abourget]

Also, I'm seeing that many nodes with the same keys loaded will all "counter-sign" all blocks. It doesn't fork or anything, and it's probably signing the same digest everywhere.. but I would have expected all signing to stop if you pause production. Would that make sense ?

[EOSBIXIN]

when I test in 1.0.6 I suffer the same bug after call resume&pasue API in second time to switch master & slaver.

the error message:
on_incoming_block ] 10 assert_exception: Assert Exception
nodeosd_1 | 2018-06-27T09:44:39.503449026Z prior != by_id_idx.end(): unlinkable block

[jchung00]

Any update on this issue?

[abourget]

Was it solved?