non-blocking-da v1: da errgroup #7
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
samlaf
force-pushed
the
devnet-monitoring
branch
from
74bcbdb
to
bed447f
Compare
samlaf
changed the base branch from
devnet-monitoring
to
altda-devnet-monitoring
* cannon: Fix stack patching And add `memprofilerate=0` to envp * Update cannon/mipsevm/program/patch.go Co-authored-by: protolambda <proto@protolambda.com> * cleanup argv/envp string ptrs * nit * fix envar name * Update cannon/mipsevm/program/patch.go Co-authored-by: mbaxter <meredith@oplabs.co> * align op-program arg0 --------- Co-authored-by: protolambda <proto@protolambda.com> Co-authored-by: mbaxter <meredith@oplabs.co>
To demonstrate how we can make our deployments more modular, this PR proposes archiving smart contract artifacts as tarballs that get uploaded to GCS. This allows deployment tools to use precompiled artifacts rather than generating them on-the fly. The archives are named after the hash of all Solidity files in the contracts-bedrock folder, including those in `lib/`, plus some additional metadata files like `foundry.toml` and `semver-lock.json`. See `calculate-checksum.sh` for details on how the algorithm works. I'm open to feedback around what should make up the checksum. Since the atifacts are content-addressable, this PR also updates the CI pipeline to download the artifacts from GCS prior to running `pnpm monorepo`. When the Solidity codebase doesn't change, this allows the `pnpm monorepo` job to skip compiling Solidity altogether. While this won't work as well when we're actively modifying the Solidity codebase, since the hash will change, it does provide a modest speedup in CI.
…1605) * cannon: Implement multithreaded clone fuzz test * cannon: Add more clone evm tests * cannon: Add evm test for GetTID syscall * cannon: Add evm test for SysExit * cannon: Add evm test for popping exited threads from the stack * cannon: Fix futex wait handling, add evm test * cannon: Add evm test for handling waiting thread * cannon: Add test utils for defining / validating MTState expectations * cannon: Add tests for futex wake, wake traversal * cannon: Add test for SysYield * cannon: Add SysOpen test, todos * cannon: Add test for SchedQuantum preemption, fix inconsistency * cannon: Add tests for noop, unsupported syscalls * cannon: Remove duplicate constants * cannon: Add tests for unsupported futex ops * cannon: Group traversal tests, fix TestEVM_WakeupTraversalStep * cannon: Add tests for nanosleep * cannon: Add additional testcase for wakeup traversal * cannon: Tweak futex wake tests * cannon: Update mt fuzz test to use new test utils * cannon: Rename contructor method for consistency * cannon: Add some simple tests for ExpectedMTState util * cannon: Add another validation test * cannon: Move syscall lists to tests where they're used * cannon: Add comment * cannon: Extract some evm test helpers * cannon: Cleanup - use require.Equalf for formatting * cannon: Rename test util to AssertEVMReverts * cannon: Add GetThreadStacks helper * cannon: Add a few more traversal tests
* Shutdown sequencer before stopping p2p * Check p2p isn't also disabled Co-authored-by: Sebastian Stammler <seb@oplabs.co> * Remove missed time.Sleep * Fix up use of SetupP2P.Disabled * Revert error check after RPC boundary * Add comment about context for StopSequencer * Add Config.p2pEnabled * op-node: Make Config.P2PEnabled public --------- Co-authored-by: Sebastian Stammler <seb@oplabs.co>
* test: fix L2 standard bridge interop tests * test: mock factory implementation instead of proxy
…thereum-optimism#11633) - protect txpool state vars with a mutex so they can be automically updated to avoid potential race condition
…ethereum-optimism#11670) Bumps [github.com/hashicorp/raft](https://github.com/hashicorp/raft) from 1.7.0 to 1.7.1. - [Release notes](https://github.com/hashicorp/raft/releases) - [Changelog](https://github.com/hashicorp/raft/blob/main/CHANGELOG.md) - [Commits](hashicorp/raft@v1.7.0...v1.7.1) --- updated-dependencies: - dependency-name: github.com/hashicorp/raft dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
bxue-l2
reviewed
Aug 29, 2024
samlaf
force-pushed
the
non-blocking-da-implv1-simple-waitgroup
branch
from
c7f96cc
to
f4ca960
Compare
samlaf
force-pushed
the
non-blocking-da-implv1-simple-waitgroup
branch
from
f4ca960
to
d78ffa9
Compare
samlaf
changed the title
samlaf
changed the title
karlb
reviewed
Aug 30, 2024
This was referenced Aug 30, 2024
--------- Co-authored-by: 0xDiscotech <131301107+0xDiscotech@users.noreply.github.com>
…m#11618) * feat: add createX preinstall * feat: change name from CreateXDeployer to CreateX
…hereum-optimism#11687) Contexts are scoped to a specific GitHub user group, which doesn't work with the GitHub merge queue or OSS contributors. This PR updates the packaging job to use raw project-level env vars instead (which are not user-scoped), and to only run on commits to `develop`.
samlaf
force-pushed
the
non-blocking-da-implv1-simple-waitgroup
branch
from
81cf5db
to
c42fedd
Compare
test(batcher): add e2e test for concurrent altda requests doc: add explanation comment for FakeDAServer chore: fix if condition in altda sendTransaction path feat: add maxConcurrentDaRequests config flag + semaphore refactor: batcher to use errgroup for da instead of separate semaphore/waitgroup fix: nil pointer bug after using wrong function after rebase fix: defn of maxConcurrentDaRequests=0 fix: TestBatcherConcurrentAltDARequests chore: remove unneeded if statement around time.Sleep refactor: use TryGo instead of Go to make logic local and easier to read chore: clean up some comments in batcher chore: make batcher shutdown cancel pending altda requests by using shutdownCtx instead of killCtx
samlaf
force-pushed
the
non-blocking-da-implv1-simple-waitgroup
branch
from
c42fedd
to
6acfb99
Compare
|
Closing since this was merged upstream. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the simplest possible implementation to fix the non-blocking da requests. We build off of @karlb's celo-org#213. We use an errgroup for da requests, with its own limit to the number of concurrent goroutines. See https://hackmd.io/@samlaf/op-batcher-concurrent-altda-requests-design-doc for more details.
Testing
We added a basic e2e test by using a fakeDaServer that sleeps for 5 seconds before returning commitments, in order to simulate slow EigenDA responses.
We also tested this manually by running the devnet and submitting large txs and observing the behavior.