Skip to content

Commit

Permalink
feat(job-processor): report attempts metrics (#448)
Browse files Browse the repository at this point in the history
## What ❔

Every component implementing JobProcessor now exposes two metrics:

`job_attempts` -- heatmaps in grafana will be added displaying this
metric.
`max_attempts_reached` -- alert will fire when this metric is increased
(alert rules not added yet).

Also, `attempts` columns in DB were increased in both `get_next*`
methods and `requeue_stuck_jobs`, so it was basically `attempts += 2` on
retry. I removed `attempts` update in `requeue_stuck_jobs`.

## Why ❔

Better monitoring and alerting

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
  • Loading branch information
perekopskiy authored Nov 10, 2023
1 parent cb2bc72 commit ab31f03
Show file tree
Hide file tree
Showing 28 changed files with 775 additions and 258 deletions.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions core/bin/contract-verifier/src/verifier.rs
Original file line number Diff line number Diff line change
Expand Up @@ -524,4 +524,12 @@ impl JobProcessor for ContractVerifier {
// Do nothing
Ok(())
}

fn max_attempts(&self) -> u32 {
u32::MAX
}

async fn get_job_attempts(&self, _job_id: &Self::JobId) -> anyhow::Result<u32> {
Ok(1)
}
}
Loading

0 comments on commit ab31f03

Please sign in to comment.