Add ability to listen to the vm step

[cgewecke]

@davidmurdoch Hi, continuing our discussion in #455 here...

You asked

Would #381 solve your issue? Or would you prefer the Ganache provider just emit low-level events like the vm's step itself (e.g., provider.on("vm:step", handler))?

After looking into this a bit I agree something like the latter would be really helpful. One low-cost approach would be to add a provider option which is a function ganache passes the step info to at createVMFromStateTrie. That method gets run for both transactions and calls and seems like a good place to manage this from.

If you are open to this solution I can submit a PR w/ tests next week. Or if you see a better way happy to help w/ that as well.

[davidmurdoch]

I think I prefer making the provider an EventEmitter (I'd probably want to use emittery instead of EventEmitter though), as this will make it easy to enable and disable the vm step handlers ad hoc, as well as provide multiple handlers, if needed.

Eventually we'll be making the provider an event emitter for all RPC calls, as well as some internal operations, as this will allow projects to access raw data directly without having to decode it from the JSON-RPC response.

[cgewecke]

@davidmurdoch

I'd probably want to use emittery instead of EventEmitter though

Yes, emittery is nice. Also agree an emitter interface is a good approach. It seems more complex to implement to me than the option, that's the only thing...

In this context if the logger was written even slightly differently, it would be possible to acquire the step info for both calls and sends with very few code changes.

cf: solidity-coverage 346

[hkalodner]

I just saw this issue and the link to your previous one. This is really a response to the original issue, but I thought I'd post this here since the other is closed. This jumps heavily into ganache internals, but there's a currently working easy solution to what you want which is to use:

provider.engine.manager.waitForInitialization(function(err, state) {
    // state.blockchain.vm is defined here
}

[cgewecke]

@hkalodner Thank you for looking into this.

In my case I need to hear the vm step when it runs for .call's as well as .send.

To me the code looks like creates a new, ephemeral instance of the VM for each .call and doesn't pass through that initialization step, possibly because the state trie is already defined. Do you know if that's correct?

I was using the vm attached to the StateManager before but it wasn't reporting all the data I'd like to see.

[hkalodner]

Happy to help. I use ganache in order to instrument the constructors of ethereum contracts https://github.com/OffchainLabs/arbitrum/blob/master/packages/arb-provider-truffle/lib/index.js which is how I came up with that solution.

You're totally right about call. For my purposes hooking into the general runtime was enough, but it's definitely incomplete.

This was a really interesting question so I decided to get hacky with it 😁, but I think I have a solution that could potentially work for you (obviously 100% unsupported and dangerous).

const BlockchainDouble = require("ganache-core/lib/blockchain_double");
const createVMFromStateTrie = BlockchainDouble.prototype.createVMFromStateTrie;
BlockchainDouble.prototype.createVMFromStateTrie = function(state, activatePrecompiles) {
    const vm = createVMFromStateTrie.apply(this, arguments);
    // vm is defined here
    return vm;
}

[cgewecke]

@hkalodner Oh that's really smart! Thanks, will try it out.

[cgewecke]

@davidmurdoch

I think I prefer making the provider an EventEmitter (I'd probably want to use emittery instead of EventEmitter though), as this will make it easy to enable and disable the vm step handlers ad hoc, as well as provide multiple handlers, if needed.

Wondering if you'd be open to accepting a PR for the smaller change I've proposed and tabling this more extensive work for later. This thread and others suggest there's an immediate community need to consume the step data and it can be provided in a stable, simple way by passing an option down to that layer in the code.

Adding a stepListener option for the provider doesn't seem like it would cost much to support into the future and it could be deprecated when a more sophisticated event architecture becomes available.

[davidmurdoch]

@cgewecke, finally getting around to implementing this and will release it this week in a new ganache alpha (we've re-written ganache: https://github.com/trufflesuite/ganache/releases/tag/ganache%407.0.0-alpha.0).

My current plan is to emit something like:

vm:tx:before, vm:tx:step, vm:tx:after

for eth_call, eth_estimateGas, eth_sendTransaction (and eth_sendRawTransaction, personal_sendTransaction, etc), debu_traceTransaction and debug_storageRangeAt (currently planning on emitting for only the transaction being traced).

I plan on having vm:tx:* messages extend from:

type VmTransactionContext = {
   transactionHash: string,
   rpc: string, // eth_call, eth_sendTransaction, etc
   context: {} // unique to each transaction lifecycle
}

If this works for you, then the best way to listen to step events would be something like:

const transactionContexts = new Map();
provider.on("vm:tx:before", (event) => {
  transactionContexts.add(event.context, []);
});
provider.on("vm:tx:step", (event) => {
  const context =  transactionContexts.get(event.context);
  context.push(event.data);
});
provider.on("vm:tx:after", (event) => {
  transactionContexts.remove(event.context);
});

You can listen to only vm:tx:step, as solidity-coverage does now, but if you need to be ensure multiple RPCs aren't conflicting the above may be a better way. Note: you can't use the transactionHash property because it is possible to call debug_traceTransaction and eth_call for the same transaction multiple times simultaneously; the events may be intermingled.

[davidmurdoch]

Aside:

I wonder if there is a way we can offset the gas costs of the coverage instrumentation effects internally? Perhaps we can make it so that vm:tx:step allows the listener to return a value which can instruct ganache internals to manipulate the run state of the transaction. Perhaps we can temporarily disable the VM's EEI gas accounting? Or maybe this is such a rare edge case it wouldn't even be worth it.

[davidmurdoch]

release in https://github.com/trufflesuite/ganache/releases/tag/ganache%407.0.0-alpha.1