Compile contracts after state sync

[bowenwang1996]

In order to switch to the new costs shown in #4231, we need to make sure that when a contract is called, it is already compiled. Normally this is the case when we switch to compiling contracts at deployment time. However, after a node finishes state syncing to a shard, it does not have all the deployed contracts in the shard compiled and this may cause major issues when the contracts are called (the node would take much longer than expected to apply the chunk). As a result, we need to the following:

• Go through the state to compile all contracts after state sync.
• After catchup, we need to compile the contracts stored in the newly synced shard.

Note: we don't have to implement the second one for now since we only have one shard on mainnet.

[bowenwang1996]

During today's discussion, @abacabadabacaba proposed a different idea: we should compile the contract when it is written to the state. More concretely, it means that we should invoke compile_module when the a new wasm binary is written to the state. We can do this in TrieUpdate::finalize, for example. This way we don't have to differentiate between regular state sync and catchup.

[matklad]

Note that another case where we might need to compile many contracts is when we upgrade our wasmer version. The data in the CompiledContractsCache depends on the specific engine version and config we use, and those might change, necessitating re-compilation of all of the contracts.

I wonder if both catchup and wasmer upgrade use-cases should be served by the same code path?

[bowenwang1996]

@matklad I don't think so. When a wasmer upgrade happens we have two options:

• Recompile the contract when it is touched. This has the benefit of not requiring any migration. The downside is that when some old contract is invoked for the first time after the upgrade, it needs to be recompiled.
• Do a database migration that recompiles all contracts in state. Note that this does not change the state per se, since compiled modules are not part of the state. However, this could still take a nontrivial amount of time if we have a lot of unique contracts.

My suggestion is to decide based on when we plan to do this migration. If we are going to do it pretty soon I'd suggest a database migration since there are < 600 unique contracts on mainnet right now and the cost of recompiling them is not very high. Plus we can potentially recompile contracts in parallel (not sure whether it is easy to do though).