Benchmark by fuel usage

[Kubuxu]

I would love to know if there is an existing harness to run a benchmark of a given piece of code/function in the context of fuel usage (possibly with wall time timing as well).

[Stebalien]

Not on a per-function basis, but it would be nice to get fuel usage in https://github.com/filecoin-project/ref-fvm/tree/master/testing/conformance/benches.

[Kubuxu]

There is a nice-ish way to do it by implementing an instrumentation gas profiler.
Within wasm-instrumentation introduce two code changes:

• at the beginning of all functions: introduce a call to a new syscall trace() u64 (syscall only available when ref-fvm is built with a specific feature or maybe runtime toggle).
• The trace() u64 syscall, saves the current gas used and a backtrace (using Rust side backtrace-rs, symbols don't have to be resolved immediately). The returned u64 is the index of the trace, this index is stored in a new local variable.
• at all exit points of functions, write current gas to a memory buffer in the form (trace_id, gas). If there are more than N entries in that buffer, the trace() syscall will read the buffer and copy it into Rust-land and clear it in WASM-land.
• alternative to it would be to call trace(trace_id) or another new syscall.

This would give us all information required to build very precise profiles of execution gas usage at the function level. These profiles can be integrated into existing tooling like flamegraphs or pprof.

At least one syscall per tracepoint is required due to the lack of facilities to gather backtrace info from within WASM.
This, of course, would skew execution timing so it isn't as relevant to setting correct gas values but it is more relevant to optimising actors for fuel usage.

[raulk]

@Kubuxu This is useful for the Filecoin Core Devs as maintainers of built-in actors, as well as for the dev community, so they can understand how their actors are performing in terms of gas, and can make informed decisions when optimising. Would you be able to write this up in a RFP format so we can consider a devgrant?