-
Notifications
You must be signed in to change notification settings - Fork 23
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
protocol: forge scripts deploy in Go (#61)
* protocol: design forge scripts deploy in Go * Update protocol/forge-scripts-deploy-in-go.md Co-authored-by: Matt Solomon <matt@mattsolomon.dev> --------- Co-authored-by: Matt Solomon <matt@mattsolomon.dev>
- Loading branch information
1 parent
bad336b
commit efb33c6
Showing
1 changed file
with
222 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,222 @@ | ||
| # Forge Scripts Deploy in Go | ||
|
|
||
| # Purpose | ||
|
|
||
| The purpose of this doc is to improve the integration between the `forge script` based chain deployment/genesis flow | ||
| and the Go tooling and testing. | ||
|
|
||
| # Summary | ||
|
|
||
| By running the chain-deployment/genesis solidity scripts in an instrumented Go EVM we can reduce the roundtrip time, | ||
| and greatly improve integration with inputs (configs, dependencies) and outputs (state, artifacts). | ||
|
|
||
| This unlocks runtime-customization of deployments in Go tests, | ||
| to increase coverage and create new multi-L2 chain deployments for Interop tests. | ||
|
|
||
| # Problem Statement + Context | ||
|
|
||
| From [Design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52): | ||
|
|
||
| > The current L2 chain deployment approach originates from a time with Hardhat, | ||
| > single L1 target, and a single monolithic set of features. | ||
| > | ||
| > Since then the system has migrated to Foundry and extended for more features, | ||
| > but remains centered around a single monolithic deploy-config for all its features. | ||
| > | ||
| > **With Interop we need to configure a new multi-L2 deployment**: | ||
| > the number of ways to compose L2s in tests grows past what a single legacy config template can support. | ||
| > | ||
| > Outside of interop, deployment also seems increasingly complex and opaque, while it does not have to be, | ||
| > due to the same configuration and composability troubles. | ||
| See the design-doc for further in-depth context on the problem. | ||
|
|
||
| The integration between (1) Go testing/tooling and (2) Forge deployment/genesis needs to improve | ||
| to reduce the complexity of deploying and testing. | ||
|
|
||
| Specifically, Go tools need reliable and organized inputs for deployment/genesis, | ||
| and interface with the forge scripts artifacts or scripts themselves | ||
| in a way that is configurable but not error-prone. | ||
|
|
||
| # Proposed Solution | ||
|
|
||
| In [deployment-chains design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52) and | ||
| [OPStackManager design doc 60](https://github.com/ethereum-optimism/design-docs/pull/60) the importance | ||
| of Modular configs and incremental deploy steps is outlined. | ||
|
|
||
| To utilize those modular configs and deploy steps, we need a way to either | ||
| run and cache the individual Forge script functions (as outlined in 52), | ||
| or integrate it closer such that no caching complexity is required. | ||
|
|
||
| This document proposes the latter: run the Forge scripts within Go, | ||
| such that we do not need multiple steps of Forge script sub-process to build a state for testing / genesis tooling. | ||
|
|
||
| ## Running Forge scripts in Go | ||
|
|
||
| Forge scripts are really just solidity smart-contracts, with the addition of Forge cheat-codes. | ||
| Running a contract in Go is relatively trivial: e.g. Cannon tests run the Cannon contracts in an instrumented Go EVM. | ||
|
|
||
| To support the Forge cheatcodes, we need to emulate the cheatcode behavior, as the Geth EVM does not natively have support for it. | ||
| Cheatcodes are essentially responses to `STATICCALL`s to a special system address, acting like a precompile, | ||
| with functions that interact with the EVM environment. | ||
|
|
||
| We do not have to support all Forge cheatcodes; just the subset used by the OP-Stack scripts would be sufficient. | ||
|
|
||
| The most important cheat-codes are: | ||
| - `vm.chainId`: change chain ID | ||
| - `vm.load`, `vm.store`: storage getter/setter | ||
| - `vm.etch`: write contract code | ||
| - `vm.deal`: set balance | ||
| - `vm.prank`: change sender | ||
| - `vm.getNonce`/`vm.setNonce`: account nonce getter/setter | ||
| - `vm.broadcast`,`vm.startBroadcast`, `vm.stopBroadcast`: capture calls as transaction candidates. (Can be no-op, if we do not want to go through Go for production deployments) | ||
| - `vm.getCode`/`vm.getDeployedCode`: artifact inspection | ||
| - `vm.env{Bool/Uint/Int/etc.}`: env var getters | ||
| - `vm.keyExists{...}/parse{...}/writeJson/etc.`: encoding utils | ||
| - `vm.addr`: priv key to addr | ||
| - `vm.label`: name an address | ||
| - `vm.dumpState`: export EVM state | ||
| - `vm.loadAllocs`: import EVM state | ||
|
|
||
| Note that with many of these, Go instrumentation can really improve integration: | ||
| - `dumpState`/`loadAllocs`: no need to encode/decode state or read/write to disk -> faster Go tests | ||
| - `getCode`/`getDeployedCode`: attach directly to artifacts, and *track which artifacts were used for a deployment* | ||
| - `broadcast`: we can extend the deploy-tool functionality with transactions. Out of scope for now, but can be very useful. | ||
|
|
||
| ## Forge-artifacts as Go FS | ||
|
|
||
| A Go FS is a simple filesystem abstraction: it provides read-only access to some source of files. | ||
| A local directory can be wrapped into such FS, but also tarballs, or even data embedded in the Go binary, can be represented as Go FS. | ||
| By using this FS abstraction, we can make the access to artifacts very simple, and "mount" the relevant FS into it. | ||
|
|
||
| For Go tests, this would be the local FS. | ||
|
|
||
| For Go genesis tooling, this might be a bundled FS, or one from a versioned release tarball. | ||
|
|
||
| Using an FS helps simplify the way we interact with forge-artifacts. | ||
|
|
||
| ## Semver the scripts | ||
|
|
||
| We have an existing `Semver` pattern for production contracts, but don't apply the same to deploy scripts, yet. | ||
| If we introduce this, then the version of the scripts (part of the artifacts) can be inspected | ||
| by the Go genesis / test tooling, and usage of the script can then be adapted. | ||
| Or at the very least, a warning can be thrown when the Go tool does not support the script. | ||
|
|
||
| This improves on the current situation, since the Go tool cannot tell anything about the compatibility | ||
| of the deployment output of the forge scripts with the genesis-generation it does. | ||
|
|
||
| ## Fast input/output | ||
|
|
||
| In addition to the Go forge cheatcodes, we could also substitute known contracts in the Forge script setup. | ||
| In particular, deploy configs are registered at fixed global addresses. | ||
|
|
||
| By mocking these contracts, we can couple config-reads directly to the actual config, | ||
| rather than having to load a JSON into a long list of EVM MPT storage leafs, | ||
| only then to read it construct it back into a memory JSON string many times over right after. | ||
|
|
||
| Instrumentation of the config inputs can also provide a clear trace of when and how configuration affects a deployment. | ||
|
|
||
| Similar to config inputs, we can capture outputs more efficiently: | ||
| upon `vm.dumpState` we do not have to encode the state; we can simply copy it in-process. | ||
| This is great for the execution speed of the Go tests: we do not have to use intermediate state JSON files on disk. | ||
|
|
||
| ## Usage by op-chain-ops | ||
|
|
||
| The op-e2e tests and `op-node genesis` tool both rely on `op-chain-ops/genesis` to prepare a chain state, | ||
| given some deploy-configuration. | ||
|
|
||
| The `op-chain-ops` package is then responsible of applying the configuration, to generate the correct state. | ||
|
|
||
| With this solution it means that it: | ||
| - Opens the forge-artifacts FS | ||
| - Takes any configuration, and sets up the necessary ENV vars for cheat-code usage. | ||
| - Instantiates an instrumented EVM, with the initial script entry-point loaded into it. | ||
| - Including cheat-codes hooked up to the forge-artifacts for ENV data | ||
| - Includes cheat-codes hooked up to state loading / writing ability. | ||
| - Includes any mocked config contracts, where we basically map the bytes4 calldata back to a config attribute. | ||
| - Calls the entry-point script, with an ABI argument, to run the particular deploy function of interest. | ||
| - E.g. `deploySuperchain`, or smaller deploy functions like `deployProxyAdmin` | ||
| - Capture any output state | ||
| - Capture any deployed contract addresses (calls to `Artifacts.s.sol` interface), | ||
| or alternatively simplify the script to just use simple `vm.label` functionality. | ||
| - Capture any `vm.label` for later debugging. | ||
|
|
||
| ## Resource Usage | ||
|
|
||
| While this solution does not include caching like | ||
| [deployment-chains design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52), | ||
| it does improve the performance of genesis generation a lot by bringing the forge execution closer, into the Go process. | ||
| In this new solution there is less cost in disk-IO, as there is no writing of cache files, | ||
| and configuration and state data can all stay in-process and skip JSON encoding/decoding. | ||
|
|
||
| In the past we have generated L2 genesis state through manual artifact inspection and Go based state surgery, | ||
| which was moved away from due to complexity of the surgery code, but was sufficiently fast for Go scripts. | ||
|
|
||
| This keeps the test setup simple (we don't duplicate any special deploy function work into Go) | ||
| and fast (avoid lots of IO / encoding / sub-process overhead). | ||
|
|
||
| ## Close integration, but no contract logic leaks | ||
|
|
||
| By loading the forge scripts, the deployment logic all stays native to the smart-contracts, | ||
| to avoid manual surgery steps in Go. | ||
|
|
||
| When adding a deployment config variable, the only Go change needed is to add it to the Go config definition, | ||
| and write any tests to exercise that deployment, as should be the default for every protocol feature. | ||
|
|
||
| The deployment implementation details stay encapsulated in the Forge scripting, | ||
| which is unified with the Forge testing, and thus unifying the production and test code paths. | ||
|
|
||
| ## Potential future extension: deployment integration | ||
|
|
||
| The `vm.broadcast` cheat-code is an opportunity for future deployment improvements: | ||
| rather than running through Forge script when preparing the production deployment transactions, | ||
| we could run through the Go genesis tool. | ||
|
|
||
| This would allow us to script more advanced post-processing of the transactions: | ||
| - cross-validation against the superchain-registry | ||
| - simulation of the transaction with custom tracing | ||
|
|
||
| And all bundled in Go, so the end-user does not need to ensure a specific Forge version, | ||
| does not need to as many manual `forge script` invocations, | ||
| and all environment settings (that might otherwise be set with forge script flags) | ||
| can be controlled by defining the exact CLI interface. | ||
|
|
||
| This is out-of-scope for now, but may help unify the deploy process that production chains use, | ||
| and the deploy process that devnets / op-e2e use. | ||
|
|
||
| # Alternatives Considered | ||
|
|
||
| See proposed solution [in design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52), | ||
| where `forge script` is used as is, and performance concerns are mitigated with caching. | ||
|
|
||
| An [experimental draft of the caching](https://github.com/ethereum-optimism/optimism/pull/11297) was implemented, | ||
| but arguably the caching introduced too much complexity and fragility. | ||
|
|
||
| Other solutions / ideas are discussed in design-doc 52 as well, but were not viable. | ||
|
|
||
| # Risks & Uncertainties | ||
|
|
||
| ## Geth Go EVM | ||
|
|
||
| The Go EVM instrumentation might be difficult, as the geth EVM is not as widely used in tooling. | ||
| However, the tracing functionality is excellent, and Go is quite flexible. | ||
| If we need to we can make very minor tweaks to `op-geth`, | ||
| to expose any inaccessible EVM internals needed to implement the forge cheatcodes. | ||
|
|
||
| ## Eng time | ||
|
|
||
| This is a mini-project: the scope of implementing the cheat-codes is not that large (a few days at most), | ||
| but the integration into tooling and op-e2e may be more involved (can be a week, maybe two). | ||
|
|
||
| In the past the `L2Genesis.s.sol` and `allocs` work, that moved us away from the manual and error-prone op-chain-ops surgery, | ||
| was completed successfully in the form of an interop side-project to remove tech-debt. This project scope looks quite similar. | ||
|
|
||
| This functionality does block Interop op-e2e testing: without it, | ||
| we are not able to customize deployments sufficiently and cleanly (avoiding many more `allocs` special cases), | ||
| to get multi-L2 deployments into the op-e2e. | ||
|
|
||
| ## Devrel feedback | ||
|
|
||
| Historically devrel has not been included sufficiently in the deployment-flow design. | ||
| Known pain-points like inconsistency between the Go and forge genesis generation, | ||
| unclear allocs, and monolithic deploy-config are being addressed, but more feedback may still improve the design. |