Forge Extensions

[brockelmore]

Component

Forge

Describe the feature you would like

Rational

vm.ffi is an unsafe security hole, bad UX, and slow. Differential testing is too slow to be reasonable. Huff compilation is annoying.

Scope

Initial scope should be EVM specific replacement/alternative to vm.ffi. Longer term, once the initial wasm runtime works well in an EVM context, there may be other places that make sense to allow plugins to tap into (compilation, post-test results, etc).

Design Choices

There are two paths this design could take shape (there are more but these are most realistic):

1. WASM based
2. RPC based

Each have their pros and cons.

WASM based extensions are the "easiest" for plugin/extension developers - they don't have to implement an RPC server. Users can just write a snippet of code in their language of choice and ship it as either a standalone, shareable repo or even ship it inside their foundry-based project. This path likely leads to fragmentation, experimentation, but ultimately more developers making better use of foundry.

RPC based extensions are more flexible than WASM based extensions because there is no compilation to WASM (neither at the script or interpreter level). There is more overhead on the developers here, to where there is a centralizing and stabilizing force of extensions (think VSCode extensions - most people don't write custom extensions).

WASM path

Use wasmer wasm runtime at the core. Ship with pyodide by default to allow python to be ran. If we want to support javascript (I am not inclined to include this), we could ship with javy to compile JS to WASM on the fly and allow javascript to be ran.

I lean towards only supporting python extensions natively, and all others must be compiled to WASM by the user. I imagine python & rust will be most common languages for extensions to be built in.

WASM EVM interface

To satisfy the replacement of vm.ffi for most use cases, I suggest treating each extensions similarly to how we treat vm, i.e. as a contract. A sketch of the full implementation:

1. In foundry.toml have an extensions configuration that points to wasm files. i.e. extensions = [ { name: "my_extension", path: "./my_local_extension.wasm", abi: "my_extension_abi.json"}]. Extensions must be uniquely named and the path must be provided*(in theory, we could allow for global extensions in ~/.foundry). The path could potentially be something like { style: "git", url: "https://github.com/...", wasm: "path/in/repo"} in the future, and could be managed by git submodules akin to libraries.
2. when starting a contract runner, we load up the wasm extensions. We introduce a new cheatcode: getExtension(string memory name) returns (address). Extensions are assigned an address by the contract runner, and upon calling getExtension, an address is returned allowing the user to interact with the extension via a normal contract call
3. Under the hood when a call is made, add a line similar to this, but check

if let Some(ext) = self.addr_to_extensions(addr) {
	self.call_extension(ext, data)
}

4. call_extension would determine the function in the extension to call and decode the input using the abi provided in the foundry.toml and call the function via the WASM runtime
5. Load the return data back into the EVM as return data

Python Example:

# my_extension.py
def addOne(x):
    return x + 1

# foundry.toml
extensions = [ {
	name: "my_extension",
	path: "./my_extension.py",
	abi: "my_extension_abi.json"
}]

interface MyExtension {
	function addOne(uint256 x) external returns (uint256);
}

contract T is Test {
	MyExtension myPyExt = vm.getExtension("my_extension");

	function testExt() public {
		uint256 ret = myPyExt.addOne(100);
		assertEq(ret, 101);
	}
}

Eventually, we could expand this and define specific interfaces for non-evm based calls for compilation & post-test extensions (although arguably, an RPC-based system likely makes more sense for this kind of extension).

RPC based

Technically, this is sort of possible today with the introduction of vm.rpc, so I wont go into much more detail. You can in theory define a contract like:

contract Extension is Test {
	function addOne(uint256 x) public returns (uint256) {
		uint currFork = vm.activeFork();
		vm.selectFork(...);
		uint256 ret = abi.decode(vm.rpc("addOne", abi.encode(x)), (uint256));
		vm.selectFork(currFork);
		return ret;
	}
}

A real version of this would likely clean up some of this, maybe automatically creating some of this based on an abi in foundry.toml.

RPC based is more interesting for pre and post-compilation, and pre and post-test as those interfaces are much more strictly defined.

My personal preference is towards the WASM path, but it is a not insignificant amount of complexity. The RPC version pushes more complexity on to developers and I think will result in much less usage.

Additional context

This is a request for feedback - I don't claim that either of these are the best path but just paths I have considered

[brockelmore]

related: #706

[onbjerg]

this would impact a lot of forge and would need more consideration. for example, fuzzing could randomly hit an extension address, potentially triggering the extension. there are 2 camps here, one that says we should automatically remove these from the fuzzer, and others who say we should still include them since any address is an attack vector

not sure which i buy into more, but something to think about.

generally, it makes sense to have pluggable precompiles as they fill a gap cli plugins couldn't

in terms of wasm for post-compilation etc etc, i think we're better off having a cargo-like plugin ecosystem for non-evm things (compilers, static analyzers etc) rather than having hooks that call out to wasm

[brockelmore]

this would impact a lot of forge and would need more consideration.

Fully agree - but this has been talked about literally for over a year and no progress has been made! Really this issue is just to get the ball rolling on what it could actually look like and actually begin the conversation around all the edge cases.

I do think this would be a major upgrade, if only for the differential fuzzing aspect (and have gotten multiple requests for extensions not just for differentials)

[roninjin10]

Jumping in here to advocate for a few things:

1. Strong strong strong advocation for "everybody makes extensions" model rather than vscode style. I believe what will happen when you do that is foundry users will find use cases none of us ever saw coming
2. Please please please support javascript

Lastly wanted to share that I implemented similar foundry style scripting for my EVMts project. For context EVMts is a JavaScript library that takes very strong inspiration from foundry and essentially tries to bring foundry to the browser and node. It includes a VM inspired by anvil and a programming model inspired by foundry scripting.

Here is how EVMts works for this. It's very similar to the python proposal:

1. You write an interface in solidity

2. You then implement it in JavaScript. Note here I am importing the solidity interface and that is proving me typesafety. TypeScript knows what methods I need to implement and what their params and return value are.

3. You can then use it in your contracts. I went with a dependency injection approach but I believe the proposed vm.addExtension api is nice and I'm strongly considering changing this to look more like the python proposal here:

4. Lastly you can import scripts and run them in javascript too in addition to cli. I hope to support foundry in this regard in future too:

[RPate97]

I'm super happy to see some activity on better support for extensions. We'd love an official extensions system to tap into for Sphinx. I agree with @roninjin10 that JS support would be very nice. We use JS internally, so support for that would help us adopt this system much faster. However, we would probably migrate to any official extension system even if it required the use of a new language.

Regarding the long-term scope for extensions, we would love to see hooks into the compilation process and especially into the transaction execution process. We would like to implement our custom execution strategy to replace the existing broadcast process and enable scripts to be executed natively within Foundry via our smart contract wallet. It would be awesome if this could be done in a generalized way, so anyone could inject extensions to modify the way scripts are executed. Some interesting examples might be:

• use a custom create2 factory
• use create3 instead of create2
• execute via a smart contract wallet
• execute via a relayer service like Gelato

[hiddentao]

I'd like create3 support to be built-in too. There are some highly optimized libraries now available for this.

[zerosnacks]

Supportive of adding extensions to Foundry, tracking extension related tickets here: A-extensions Area: extensions

It will require more discussion and specs before a start can be made. Personally I think having WASM based extensions is the preferable solution.

It would be great if all of you can share your latest thoughts on the topic in the extensions meta-ticket: #8266

[zerosnacks]

Closing this ticket (as duplicate) in favor of the meta-ticket: #8266, in an effort to have a more central place to discuss / spec ideas related to extensions.

I've done the same for #706 and #7760