chore(sequencer)!: add metrics #1248
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
noot
approved these changes
Jul 9, 2024
noot
reviewed
Jul 9, 2024
Comment on lines
+332
to
+343
| pub(crate) fn record_actions_per_transaction_in_mempool(&self, count: usize) { | ||
| // allow: precision loss is unlikely (values too small) but also unimportant in histograms. | ||
| #[allow(clippy::cast_precision_loss)] | ||
| self.actions_per_transaction_in_mempool.record(count as f64); | ||
| } | ||
|
|
||
| pub(crate) fn record_transaction_in_mempool_size_bytes(&self, count: usize) { | ||
| // allow: precision loss is unlikely (values too small) but also unimportant in histograms. | ||
| #[allow(clippy::cast_precision_loss)] | ||
| self.transaction_in_mempool_size_bytes.record(count as f64); | ||
| } | ||
|
|
There was a problem hiding this comment.
these are set every time a tx is inserted (and is per-tx) right? would it be more/less useful to also have a mempool total?
There was a problem hiding this comment.
I was wondering about that too. Thing is, it's relatively cheap to get the number of txs in the mempool (call to lock the RwLock and then a call to len()), but it could be expensive to track either of these (e.g. lock then iterate all txs).
If we do want this, it's probably best to track this inside the mempool. We'd want the tx size to be passed in via insert to avoid having to convert the signed tx back into PB form to reserialize it.
I'll merge this PR as is, and if anyone thinks we should add these metrics, I can do a follow-up PR.
steezeburger
added a commit
that referenced
this pull request
Jul 15, 2024
* main: feat(cli): add cmd to collect withdrawal events and submit as actions (#1261) fix(core, bridge, sequencer)!: dismabiguate return addresses (#1266) fix(withdrawer): support withdrawer address that differs from bridge address (#1262) (core, sequencer)!: generate serde traits impls for all protocol protobufs (#1260) fix(charts): add resources for sequencer/cometbft (#1254) chore(sequencer)!: add metrics (#1248) fix(sequencer-utils): fixes issue in `parse_blob` tests (#1243) feat(core, proto)!: make bridge unlock memo string (#1244) fix(conductor): don't panic during panic (#1252) feat(core)!: lowerCamelCase for protobuf json mapping (#1250) refactor(bridge-withdrawer)!: refactor startup to a separate subtask and remove balance check from startup (#1190) fix: rollup archive node configurations (#1249) refactor(core, bridge-withdrawer)!: move bridge-unlock memo to core (#1245) fix(sequencer)!: store native asset ibc->trace mapping in init_chain (#1242)
steezeburger
added a commit
that referenced
this pull request
Jul 19, 2024
* main: (24 commits) chore: update `bytes` and `ics23` crates (#1279) fix(sequencer): improve and fix instrumentation (#1255) feature(charts): hermes chart fixes, bech32 updates, ibc bridge test (#1130) chore(cli): remove unused rollup cli code (#1275) chore(test): use a temporary file to not pollute the workspace (#1269) chore(sequencer): add mempool benchmarks (#1238) fix(bridge-withdrawer)!: fix nonce handling (#1215) feat(cli, bridge-withdrawer)!: share code between cli and service (#1270) feat(cli): add cmd to collect withdrawal events and submit as actions (#1261) fix(core, bridge, sequencer)!: dismabiguate return addresses (#1266) fix(withdrawer): support withdrawer address that differs from bridge address (#1262) (core, sequencer)!: generate serde traits impls for all protocol protobufs (#1260) fix(charts): add resources for sequencer/cometbft (#1254) chore(sequencer)!: add metrics (#1248) fix(sequencer-utils): fixes issue in `parse_blob` tests (#1243) feat(core, proto)!: make bridge unlock memo string (#1244) fix(conductor): don't panic during panic (#1252) feat(core)!: lowerCamelCase for protobuf json mapping (#1250) refactor(bridge-withdrawer)!: refactor startup to a separate subtask and remove balance check from startup (#1190) fix: rollup archive node configurations (#1249) ...
bharath-123
pushed a commit
that referenced
this pull request
Jul 25, 2024
## Summary This adds further metrics to the sequencer. ## Background This should help diagnose block production slowdown when the sequencer is stress-tested. ## Changes - Added metrics (see below for list). - Enabled `cnidarium` metrics. Note that all histograms are still rendered as Prometheus summaries for now. I have [an open PR](#1192) which will make it simple to provide buckets for histograms, after which they will be rendered as true histograms. ## Testing Testing will likewise be relatively simple once #1192 is merged. In the meantime, I ran the smoke test using the code in this PR and manually checked the new metrics are available and appear sane. ## Metrics - Added `astria_sequencer_check_tx_duration_seconds` histograms with the following labels: - `length check and parse raw tx` - `stateless check` - `nonce check` - `chain id check` - `balance check` - `check for removal` - `insert to app mempool` - Added `astria_sequencer_actions_per_transaction_in_mempool` histogram - Added `astria_sequencer_transaction_in_mempool_size_bytes` histogram - Added `astria_sequencer_transactions_in_mempool_total` gauge - Enabled `cnidarium_get_raw_duration_seconds` histogram - Enabled `cnidarium_nonverifiable_get_raw_duration_seconds` histogram ## Related Issues Closes #1247.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This adds further metrics to the sequencer.
Background
This should help diagnose block production slowdown when the sequencer is stress-tested.
Changes
cnidariummetrics.Note that all histograms are still rendered as Prometheus summaries for now. I have an open PR which will make it simple to provide buckets for histograms, after which they will be rendered as true histograms.
Testing
Testing will likewise be relatively simple once #1192 is merged. In the meantime, I ran the smoke test using the code in this PR and manually checked the new metrics are available and appear sane.
Metrics
astria_sequencer_check_tx_duration_secondshistograms with the following labels:length check and parse raw txstateless checknonce checkchain id checkbalance checkcheck for removalinsert to app mempoolastria_sequencer_actions_per_transaction_in_mempoolhistogramastria_sequencer_transaction_in_mempool_size_byteshistogramastria_sequencer_transactions_in_mempool_totalgaugecnidarium_get_raw_duration_secondshistogramcnidarium_nonverifiable_get_raw_duration_secondshistogramRelated Issues
Closes #1247.