hbbft Integration Meta-Tracking Thread

[c0gent]

This will be updated to contain the latest information about the status of hbbft integration into Parity.

• Get hbbft fully functioning within Parity in a forked version. This may include changes which break Parity for non-hbbft networks.
  • Add a second (tokio runtime) event loop (in addition to the existing tokio-core reactor + thread pool). This is a temporary workaround until tokio-core can be completely replaced (below).
  • Update deprecated dependencies needed within hydrabadger/hbbft.
    • log & env_logger updates: Update log -> 0.4, env_logger -> 0.5. openethereum/parity-ethereum#9294
    • Update ring v0.12 -> 0.13 ecosystem wide.
    • Replace use of tokio_core (deprecated) with tokio
      • jsonrpc: Update hyper to version 0.12. paritytech/jsonrpc#303
        • parity_tokio_ipc: Migrate from tokio_core to tokio. paritytech/parity-tokio-ipc#9
          • tokio-named-pipes: Migrate from tokio_core to tokio. NikVolf/tokio-named-pipes#1
  • Config integration
  • Client integration
    • Map the transaction flow: Map the transaction flow. #12
    • Engine
      • Add a Placeholder
      • Remove all functionality apart from verifying seals from the placeholder. (No fork choice, etc.; temporarily accept empty seal?)
      • Add verification of threshold signature seals.
    • Hbbft daemon
      • After each epoch: Create a new block from the epoch's transactions, remove all the block's transactions from the TransactionQueue, import the new block into the chain, (Temporarily with empty seal?), randomly select B / N out of the first B transactions from the TransactionQueue, and input them as an HB contribution for the next epoch.
      • Start HB at epoch latest_block_number + 1.
      • Keep block numbers in sync with epochs: Ignore batches where we already have the full block. (Keep batches in a cache if the parent block is still missing—but maybe that can't even happen?)
      • After creating the block (second point above) and before importing it into the chain:
        • Multicast a threshold signature share of the block.
        • With enough shares received, add the signature as a seal to the block.
      • Pay out block rewards using the contract.
      • To avoid empty block spam, only propose transactions if either hydrabadger.state().dhb().unwrap().should_propose() or we have enough pending transactions. Needs to be checked after each epoch, Hydrabadger message and incoming transaction.
      • ...
  • Network integration
    • Validators
      • Make the validator set follow the governance contract.
        • Detect completed ballots to add or remove validators.
        • On "add", connect to the new node and get its PublicKey (unless we can integrate that into the contracts themselves).
        • Vote for adding or removing the node in HB.
        • When a node was successfully removed, disconnect from it.
      • Sign (and encrypt?) all validator-to-validator messages.
      • Report fault logs? Disconnect from faulty nodes immediately?
  • Add a daemon to act as an intermediary between hydrabadger and the parity client.
  • Move the test code (tx/account creation) into a separate script that connects via IPC.
  • Make the network restartable: wait for sync to finish, until we all are at the same block number, then start Honey Badger with the appropriate epoch, validators and keys.
  • Allow switching consensus mechanism in a live network, at least from Aura to HB and back.
  • Possible optimization: With the current flow, actual block creation time (executing the transactions) is added to the HB epoch time. We could parallelize that and start the next epoch right away, tentatively filtering out the to-be-executed transactions.
  • ...
• Full non-breaking integration (merge fork).
  • ...

Other Misc.:

• Use failure (related issue: Errors and failure openethereum/parity-ethereum#9282)
• Fix home_dir deprecation warning: Replace std::env::home_dir() with home crate impl. openethereum/parity-ethereum#9293

[DemiMarie]

Is this still being worked on? @igorbarinov