Cluster Inconsistent State Classification

Some minor blocks of a root block in the master cannot be found in slaves

• Severity: CRITICAL
• Expected Cluster Behavior: Shutdown
• Possible Reason: Machine/Container crashes (and fdatasync of rocksdb is not properly implemented), cluster is incorrectly implemented
• Fixes:

1. Manually remove the root block in the master (tools to be provided); or
2. Automatically scan and detect the root block with missing minor blocks (tools to be provided)

The root block of a minor block in a slave cannot be found in the master

• Severity: CRITICAL
• Expected Cluster Behavior: Shutdown
• Possible Reason: Machine/Container crashes (and fdatasync of rocksdb is not properly implemented), cluster is incorrectly implemented Fixes:

1. Manually remove the minor block in the slave (tools to be provided); or
2. Automatically scan and detect the minor block with missing root block (tools to be provided)

The minor block header of a minor block in a slave cannot be found in the master

• Severity: WARNING
• Expected Cluster Behavior: Cluster should continue to work
• Possible Reason: Cluster shutdown before the minor block is committed
• Related fixes: https://github.com/QuarkChain/pyquarkchain/issues/487

The root block in a master cannot be found in the slave

• Severity: WARNING
• Expected Cluster Behavior: Cluster/Slave will fail to sync minor blocks
• Possible Reason: Cluster shutdown before the root block is committed
• Fixes:

1. Manually add the root block to slaves (tools to be provided)
2. Automatically scan and detect the root block in the master while it is missing in slaves (tools to be provided)

The cross-shard transaction deposits cannot be found in the target slave

• Severity: CRITICAL
• Expected Cluster Behavior: Cluster will redo the block and broadcast the x-shard tx deposits again
• Possible Reason: Cluster shutdown before the minor block is committed
• Fix: https://github.com/QuarkChain/pyquarkchain/pull/493