feat(proof-data-handler): exclude batches without object file in GCS #2980
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pbeza
force-pushed
the
tee/flag-old-batches-as-permanently-ignored-automatically
branch
3 times, most recently
from
f1b8ad3 to
65cc26e
Compare
|
@popzxc, I remember you mentioned not to ask for code reviews this wave, but you're probably the most familiar with this code (along with @slowli). So, if you could make an exception this time, I’d really appreciate it. If you're busy, no worries – feel free to ignore, and I’ll ask @RomanBrodetski to find someone else. Thanks! |
|
Kindly ping @slowli @RomanBrodetski. I need a reviewer. |
slowli
reviewed
Oct 2, 2024
There was a problem hiding this comment.
@pbeza to be honest I don't fully follow this solution. I understand what we are trying to do (mark older unresolved jobs as skipped), but I'm not sure I understand the Why here. We can discuss over a huddle or async
core/lib/dal/migrations/20240930110000_tee_add_permanently_ignored_state.down.sql
Outdated
Show resolved
Hide resolved
pbeza
force-pushed
the
tee/flag-old-batches-as-permanently-ignored-automatically
branch
17 times, most recently
from
4ee505b to
bfeddc9
Compare
haraldh
reviewed
Nov 8, 2024
haraldh
reviewed
Nov 8, 2024
pbeza
commented
Nov 8, 2024
slowli
reviewed
Nov 8, 2024
…as-permanently-ignored-automatically
…as-permanently-ignored-automatically
We already depend on zksync_types, which re-exports a substantial part of basic types.
…as-permanently-ignored-automatically
…as-permanently-ignored-automatically
…as-permanently-ignored-automatically
…as-permanently-ignored-automatically
…as-permanently-ignored-automatically
…as-permanently-ignored-automatically
|
@slowli @haraldh I just retested everything manually to ensure that transitioning to the |
slowli
approved these changes
Nov 20, 2024
haraldh
deleted the
tee/flag-old-batches-as-permanently-ignored-automatically
branch
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 11, 2024
🤖 I have created a release *beep* *boop* --- ## [25.3.0](core-v25.2.0...core-v25.3.0) (2024-12-11) ### Features * change seal criteria for gateway ([#3320](#3320)) ([a0a74aa](a0a74aa)) * **contract-verifier:** Download compilers from GH automatically ([#3291](#3291)) ([a10c4ba](a10c4ba)) * integrate gateway changes for some components ([#3274](#3274)) ([cbc91e3](cbc91e3)) * **proof-data-handler:** exclude batches without object file in GCS ([#2980](#2980)) ([3e309e0](3e309e0)) * **pruning:** Record L1 batch root hash in pruning logs ([#3266](#3266)) ([7b6e590](7b6e590)) * **state-keeper:** mempool io opens batch if there is protocol upgrade tx ([#3360](#3360)) ([f6422cd](f6422cd)) * **tee:** add error handling for unstable_getTeeProofs API endpoint ([#3321](#3321)) ([26f630c](26f630c)) * **zksync_cli:** Health checkpoint improvements ([#3193](#3193)) ([440fe8d](440fe8d)) ### Bug Fixes * **api:** batch fee input scaling for `debug_traceCall` ([#3344](#3344)) ([7ace594](7ace594)) * **tee:** correct previous fix for race condition in batch locking ([#3358](#3358)) ([b12da8d](b12da8d)) * **tee:** fix race condition in batch locking ([#3342](#3342)) ([a7dc0ed](a7dc0ed)) * **tracer:** adds vm error to flatCallTracer error field if exists ([#3374](#3374)) ([5d77727](5d77727)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
/tee/proof_inputsendpoint no longer returns batches that have no corresponding object file in Google Cloud Storage for an extended period.Why ❔
TEE's
proof-data-handleronmainnetwas flooded with warnings.Since the recent
mainnet's24.25.0redeployment, we've been flooded with warnings for theproof-data-handleronmainnet(the warnings are actually not fatal in this context):The issue is caused by the code behind the
/tee/proof_inputsendpoint (which is equivalent to the/proof_generation_dataendpoint) – it finds the next batch to send to the requestingtee-proverby looking for the first batch that has a corresponding object in the Google object store. As it skips over batches that don’t have the objects, it logsFailed request with a fatal errorfor each one (unless the skipped batch was successfully proven, in which case it doesn’t log the error). This happens with every request thetee-proversends, which is why we're getting so much noise in the logs.One possible solution is to flag the problematic batches as
permanently_ignored, like Thomas did before onmainnet.Checklist
zk fmtandzk lint.