cmd, core, eth: add support for Reth style ExEx plugins

[karalabe]

We've been working on live tracers for a while now, allowing hooking into the EVM lifecycle. Whilst that work is super useful in itself, sometimes it is useful to hook into higher level chain events too (head events, reorgs, etc). This is what Reth pioneered with ExEx, and it's also essentially what this PR introduces.

The PR is a work in progress, so the API might change. Even if it gets merged, the API will remain fluid for now.

Quickstart

From a user perspective, execution extensions are essentially plugins that implement a slew of (optional) callbacks so that they might react to chain and EVM events with 0-delay, immediately as they are happening within the node itself. Since Go does not have the capability to dynamically load/unload code, all such code needs to be compiled into Geth for now (and probably forseeable future).

To implement a plugin, you need 2 things:

• Create a Go method that returns an exex.Plugin struct.
• Register your plugin into Geth's ExEx plugin registry.

For an absolute minimal code, take a look at our minimal ExEx plugin, included in Geth in the plugins package.

package plugins

import (
	"github.com/ethereum/go-ethereum/core/exex"
	"github.com/ethereum/go-ethereum/core/types"
	"github.com/ethereum/go-ethereum/log"
)

// Register the minimal ExEx plugin into Geth.
func init() {
	exex.RegisterV1("minimal", newMinimalPlugin)
}

// newMinimalPlugin creates a minimal Execution Extension plugin to react to some
// chain events.
func newMinimalPlugin(config *exex.ConfigV1) (*exex.PluginV1, error) {
	return &exex.PluginV1{
		OnHead: func(head *types.Header) {
			config.Logger.Info("Chain head updated", "number", head.Number, "hash", head.Hash())
		},
	}, nil
}

This code is super tiny, yet achieves a lot:

• The plugin constructor (newMinimalPlugin) we've defined creates an *exex.PluginV1, which is essentially a set of optional callbacks. Out of all these callbacks, this plugin only implements the OnHead method, so will only receive events for that.
• The constructor needs to implement exex.NewPluginV1, which takes a config. The config contains a logger from Geth to allow hooking into Geth's logging subsystem with the plugin name already pre-set. It also contains a user configuration string that the plugin can interpret as it wants.
• The init call registers this plugin under the globally unique name of minimal. This will automatically create a new CLI flag for Geth --exex.minimal that will allow running Geth with or without the plugin enabled; and also --exex.minimal.config that the user can pass arbitrary opaque configurations to the plugin. You can run geth --help to check what plugins have been registered by us - or by you.

   EXECUTION EXTENSIONS

   
    --exex.minimal                      (default: false)                   ($GETH_EXEX_MINIMAL)
          Enables the 'minimal' execution extension plugin
   
    --exex.minimal.config value                                            ($GETH_EXEX_MINIMAL_CONFIG)
          Opaque config to pass to the 'minimal' execution extension plugin

You can place your custom plugins anywhere really within Geth, but they need to be imported, so for small plugins consider dropping them into the same plugins folder which is already seeded into Geth. Alternatively create a subdirectory under plugins, but then add a blank import statement to plugins so your code gets compiled into Geth.

Hooks

This section is fluid. I've tried to document as I go along, but don't expect API stability until I merge this thing. Even after that it might be a bit wobbly for a few releases until people start using it.

• Chain hooks:
  • OnInit(chain exex.Chain): Called when the chain gets initialized within Geth.
  • OnClose(): Called when the chain gets torn down within Geth.
  • OnHead(head *types.Header): Called when the chain head block is updated.
  • OnReorg(headers []*types.Header, revert bool): Called when the chain is reorging.
  • OnFinal(header *types.Header): Called when the chain finalizes a past block.

The plugins have access to the following chain constructs (via *exex.Chain):

• Head() *types.Header: Retrieves the current head block's header.
• Final() *types.Header: Retrieves the current finalized block's header.
• Header(number uint64) *types.Header: Retrieves a canonical block header with the given number.
• Block(number uint64) *types.Block: Retrieves an entire canonical block with the given number.
• State(root common.Hash) State: Retrieves a state accessor at a given root hash.
• Receipts(number uint64) []*types.Receipt: Retrieves a set of canonical receipts with the given number.

The plugins have access to the following state constructs (via exex.State):

• Balance(addr common.Address) *uint256.Int: Retrieves the balance of the given account.
• Nonce(addr common.Address) uint64: Retrieves the nonce of the given account.
• Code(addr common.Address) []byte: Retrieves the bytecode associated with the given account.
• Storage(addr common.Address, slot common.Hash) common.Hash: Retrieves the value associated with a specific storage slot.
• AccountIterator(seek common.Hash) snapshot.AccountIterator: Retrieves an iterator to walk across all the known accounts.
• StorageIterator(account common.Hash, seek common.Hash) snapshot.StorageIterator: Retrieves an iterator to walk across all the known storage slots

Internals

Package structure

Implementation wise, there are 2 exex packages that this PR introduces. This might seem a bit wonky, but it's done for an API separation of concerns reason:

• core/exex is the public API that will have versioning and stability guarantees.
• core/exex/exex is the internal API that Geth itself will call for operating the plugins.

You'll see that the actual functionality is fully implemented within core/exex (because doing otherwise would lead to dependency cycles), but it is hidden from public consumption and the core/exex/exex package just re-exposes those via an interface what core/exex took effort to hide. This might be a bit of a funky design, but it ensures that core/exex will contains a strongly versioned stable user API without risking that some plugin calls something it's not supposed.

Singleton namespace

Our live tracer is cheating a bit, because it was injected into the vmConfig struct and passed around Geth everywhere. We could do a similar integration for exex plugins too, but with a more-encompassing event surface, we'd potentially end up with passing it across everything.

We might want to do that in the future, but until we figure out what's needed and what not, the current code uses a singleton global instance and every event trigger directly calls into it via the internal exex package (e.g. exex.TriggerHeadHook. This in theory permits us to trigger hooks from anywhere within Geth. In practice, dependency loops will be the killer, so let's see :D

[maoueh]

One hook that I think would be a good addition to the current set you have is a OnFinalizeBlock hook that would be invoked as the consensus chain decides to finalize block.

Having such hook:

• Enables some efficient management of reversible segment of a chain
• Enables signalling for offloading processing when stuff are "final"

[karalabe]

Ah yeah, kept wondering what that event was I wanted :D Thx @maoueh

[karalabe]

@maoueh Done, pushed the commit

[MariusVanDerWijden]

Wdyt about renaming exex/exex to exex/internal? This way we can make sure that no one depends on the internal parts of exex

[karalabe]

Import fail from main packages because those can’t access internal /facepalm + not much use because plugins shipped inside Geth do have access :)

…

On Fri, 18 Oct 2024 at 09:16, Marius van der Wijden < ***@***.***> wrote: Wdyt about renaming exex/exex to exex/internal? This way we can make sure that no one depends on the internal parts of exex — Reply to this email directly, view it on GitHub <#30611 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA7UGIUWX6N5OXRGR37MBLZ4CRTZAVCNFSM6AAAAABQBINC7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRRGUYTMOBWGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[karalabe]

This work was decided against. I will be maintaining it in the future in https://github.com/karalabe/tinygeth.

[roninjin10]

I don't see the tinygeth repo anymore (404). This is something I could replicate though correct? It seems like this code is not invasive to geth itself