Investigating the bottleneck in compile_and_load()

[akhi3030]

Zulip discussion: https://near.zulipchat.com/#narrow/stream/295306-contract-runtime/topic/bottleneck.20on.20compile_and_load

In this example we see that from time to time, compile_and_load takes a long time (44.8ms). This issue is to investigate where the time is going.

Based on the spans that are available, it seems like the bulk of the time is being spent here. In production code, the cache is a handle to rocksdb so we suspect that the time is spent loading the serialized module from rocksdb.

[akhi3030]

Current summary:

• Based on this, we now believe that borsh deserialization is quite expensive and Simonas is going to work on swapping it out for rkyv.
• Alex is going to work on PRs to improve the caching of compiled artifacts in rocksdb
• We need to come up with ideas on how to reduce the function call base costs immediately. We will continue iterating on this.

[akhi3030]

Action items from the meeting:

• @Simonas (nagisa / ie0aua46): reintroduce the in memory cache. i.e. finish do not merge: cache for storing vm artifacts #10787.
• @Andrei Kashin (akashin): to work on adding support for compute costs for Actions Support Compute Costs for Actions #8806
• Reduce function call base cost compute cost to 700GGas
• Increase the contract loading per byte cost compute cost to 52MGas

[akhi3030]

Summary of the meeting:

@nagisa: continue working on #10787
@Ekleog-NEAR: work on #10801
@Akashin: It seems like #8806 in general is going to be too complex a implementation to land. So we are now considering reducing gas costs directly. Hence, he will investigate the impact of reducing the gas costs and prototyping introducing compute costs just for function calls.