Introduce MockPrimaryNode for the domain testing framework

[NingLin-P]

This PR is part of #1213.

This PR introduces MockPrimaryNode for the domain testing framework, each commit contains the changes of one functionality/API that MockPrimaryNode bring, please refer to the tests of domain-client-executor or subspace-fraud-proof in https://github.com/subspace/subspace/tree/domain-testing for the usage of these functionalities/APIs.

Code contributor checklist:

• I have read, understood and followed contributing guide

[liuchengxu]

It's pretty inconvenient to review with new codes completely in one PR but their usages are in another branch. Can you restructure this PR with a minimal new test primary node in the first commit, and then update the tests incrementally by adding new apis necessary for the tests, so that it's easier to see what problems the new apis are solving accordingly.

[NingLin-P]

For a test that is integrated with the MockPrimaryNode, both the slot production and block production are necessary, thus they can't be added incrementally.

I have restructured this PR to remove some block production APIs along with the last 2 commits and added a few more commits to integrate an existing test with `MockPrimaryNode. Hope this could be helpful.

[NingLin-P]

The CI failed on the test test_executor_full_node_catching_up due to:

2023-03-21 10:05:03 [//Bob (SystemDomain)] Failed to process primary block error=Application(Backend("Receipt for #0,0x6492725754a0071120116e973eb79683f5f09f11836ecf71df332abd69cf690d not found"))

This is so weird that:

• logically executor should not load for the receipt of its genesis block
• the log shows that the mock primary node has started the DSN, archiver, etc, also logically impossible
• the test logic starts before the executor bob finish its initialization

I going to push some temporary commits to add more logs in order to debug the CI.

[liuchengxu]

@NingLin-P I usually would push the debugging branch to my subspace fork and trigger the CI there in order to keep the monorepo clean and save some CI resources of monorepo.

[liuchengxu]

Make sense overall as a start, looking forward to more integrations of the new testing into the old tests.

One thing I see is that no network for the primary chain which might be inconvenient for the messenger tests as the relayer then needs to send the message to the target domain node directly by hand, but I guess we can always add it later if necessary.

[NingLin-P]

The CI failed on the test test_executor_full_node_catching_up due to:

2023-03-21 10:05:03 [//Bob (SystemDomain)] Failed to process primary block error=Application(Backend("Receipt for #0,0x6492725754a0071120116e973eb79683f5f09f11836ecf71df332abd69cf690d not found"))

This is probably due to #1283, I have patched a fix to this issue (the solution mentioned at the issue) to my subspace fork, and the CI pass.

[vedhavyas]

Great work! Some minor nits.

I have a wild thought. Since we are aiming to test as close to the production node as possible. Can't we just run a mock farmer node and provide mock solutions. We can even mock verify solution. With Nazar's changes to add this verify_solution as Host function, we can create a mock host function instead.

[NingLin-P]

Can't we just run a mock farmer node and provide mock solutions. We can even mock verify solution. With Nazar's changes to add this verify_solution as Host function, we can create a mock host function instead.

The solution for block production is generate and verify at the client side where we can't mock by utilizing the host function. Even if we can achieve that what it provide is the ability to trigger block production on demand, but we want further control on something like the new block's the parent block and which extrinsics is included, etc.

Since we are aiming to test as close to the production node as possible.

Right, that is our goal, with the new testing framework it is true for the domain (as that is want we essentially testing about), but it is not 100% true for the primary chain:

• The primary chain runtime is almost the same as the production node except for some config changes
• For the client side of the primary chain, MockPrimaryNode's the client, backend, task_manager and transaction_pool, etc, is the same as the the production node. What had changed is the consensus engine, namely the block production and verification. The archiver, plotter, block proposer/verifier etc, as port the consensus engine are removed, because they are no more needed and kind of useless as we are going to mock the consensus engine.

Another way to look at this is not matter how the primary node behave, either being slow, having a network partition or any other situations, its influence to the domain are reflect on its block import notification, onchain state, etc, we chose a way that is easier to control the block import notification, onchain state, such that we can accurately and efficiently construct the test case.

BTW, utilizing host function to mock verify_solution is a great idea! It could apply to the verification of the bundle solution as that is inside the runtime, such that we are able to construct/submit bundle directly to the primary chain's txn pool. Will explore the idea further.

[nazar-pc]

With Nazar's changes to add this verify_solution as Host function, we can create a mock host function instead.

It is only used by votes and sp-lightclient. If you don't have those you don't need to worry about it. The rest of verification is done client-side.