Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Epic] Scaling the provers 📈 #8495

Open
13 of 21 tasks
alexghr opened this issue Sep 11, 2024 · 0 comments · Fixed by #8609
Open
13 of 21 tasks

[Epic] Scaling the provers 📈 #8495

alexghr opened this issue Sep 11, 2024 · 0 comments · Fixed by #8609
Assignees
@alexghr
Labels
T-epic
Milestone
TestNet

Comments

@alexghr
Copy link
Contributor

alexghr commented Sep 11, 2024 •
edited
Loading

We need a scalable and resilient model for handling the proof necessary to build large blocks at a high frequency.

Relevant links

Design
Preview Give feedback

Tasks
Preview Give feedback
  1. [Prover] New ProvingBroker #9529
    team-alpha
    alexghr
  2. [Prover] ProvingBroker restores persisted jobs and proofs #9530
    team-alpha
    alexghr
  3. [Prover] ProvingBroker resolves conflicts when same jobs is worked on by two agents #9531
    team-alpha
    alexghr
  4. [Prover] Persist proving jobs #9532
    team-alpha
    alexghr
  5. [Prover] Updates to ProvingAgent to work with new ProvingBroker #9533
    team-alpha
    alexghr
  6. [Prover] add k8s files for ProvingBroker and small/big ProvingAgent #10108
    alexghr
  7. [Prover][Docs] Update docs with info about broker/orchestrator storage requirements #10109
    A-documentation
  8. [Prover] use common DATA_DIRECTORY config #10267
    alexghr
  9. [Prover] Prover-Node follows pending chain and proves each block of an epoch off a historic block (world state snapshot) #10265
    spalladino
  10. [Prover] Prover node keeps track of how many blocks of an epoch were started prior to a crash #10266
    alexghr
  11. [Prover] Clean up configuration split across multiple packages #10269
    spypsy
  12. [Prover] Refactor IDs for proving jobs #10398
    alexghr

Tests
Preview Give feedback
  1. [Prover][Test] Ensure proper integration between orchestrator<->broker<->agent (no serialisation errors) #9537
    team-alpha
    alexghr
  2. [Prover][Test] Performance test one broker + 1000 agents #9538
    team-alpha
    alexghr
  3. [Provier][Test] Performance test: one broker + 10k agents #9539
    team-alpha
    alexghr
  4. [Prover][Test] Broker failure: system must recover if the ProvingBroker goes down #9540
    team-alpha
    alexghr
  5. [Prover][Test] Agent failure: a busy agent goes down, the broker must give their job to another agent #9541
    team-alpha
    alexghr
  6. [Prover][Test] Agent temporary failure: agent goes down temporarily #9542
    team-alpha
    alexghr
@github-project-automation github-project-automation bot moved this to Todo in A3 Sep 11, 2024
@alexghr alexghr self-assigned this Sep 12, 2024
@spalladino spalladino changed the title [Epic] Scaling the provers [Epic] Scaling the provers 📈 Sep 12, 2024
@spalladino spalladino mentioned this issue Sep 12, 2024
Closed
@aminsammara aminsammara added this to the TestNet milestone Oct 1, 2024
@alexghr alexghr mentioned this issue Oct 23, 2024
Merged
@alexghr alexghr linked a pull request Oct 23, 2024 that will close this issue
feat: new proving broker implementation #8609
Merged
alexghr added a commit that referenced this issue Oct 24, 2024
@alexghr
feat: new proving broker implementation (#8609)
5351c4b 
This PR adds `ProvingBroker` which implements a new interface for
distributing proving jobs to workers as specified in #8495
@alexghr alexghr mentioned this issue Oct 24, 2024
Merged
@PhilWindle PhilWindle moved this from Todo to In Progress in A3 Oct 27, 2024
alexghr added a commit that referenced this issue Nov 15, 2024
@alexghr
feat: new proving broker implementation (#9400)
da711bf 
Reopening of #8609, which was closed/merged by mistake. This PR is
stacked on top of #9391

This PR adds ProvingBroker which implements a new interface for
distributing proving jobs to workers as specified in
#8495
just-mitch pushed a commit that referenced this issue Nov 16, 2024
@alexghr @just-mitch
feat: new proving broker implementation (#9400)
428738b 
Reopening of #8609, which was closed/merged by mistake. This PR is
stacked on top of #9391

This PR adds ProvingBroker which implements a new interface for
distributing proving jobs to workers as specified in
#8495
@alexghr alexghr mentioned this issue Nov 27, 2024
Merged
PhilWindle pushed a commit that referenced this issue Nov 28, 2024
@alexghr @spalladino
feat: new proving broker (#10174)
6fd5fc1 
This PR integrates the new proving broker implementation added in #8495
into the orchestrator and adds a new caching layer between the
orchestrator and the job broker able to resolve jobs that have already
been finished (in order to provide crash recovery for the prover-node).

---------

Co-authored-by: Santiago Palladino <santiago@aztecprotocol.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexghr alexghr

Labels
T-epic
Projects
A3
Status: In Progress
Milestone
TestNet
Development

Successfully merging a pull request may close this issue.

feat: new proving broker implementation AztecProtocol/aztec-packages
3 participants
@alexghr @iAmMichaelConnor @aminsammara