Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CCIP-4447 Promwrapper for OCR3 plugins and factories #15521

Merged
merged 16 commits into from
Dec 6, 2024

Conversation

mateusz-sekara
Copy link
Collaborator

@mateusz-sekara mateusz-sekara commented Dec 5, 2024

Implementing a very basic monitoring layer for OCR3 Reporting Plugins to support similar behavior to what we have for OCR2 https://github.com/smartcontractkit/chainlink/blob/develop/core/services/ocr2/plugins/promwrapper/plugin.go

Compared to the OCR2 implementation some labels like chainType, configDigest and oracleID were removed. This is done to reduce the cardinality of the metrics published to Prometheus. Intentionally, starting with a fundamental set of metrics which can be extended in the future if needed.

These metrics will help us answer the following observability questions:

  • what is the error rate for different OCR3 phases
  • what is the throughput per plugin (per OCR3 phase)
  • what are the latencies per OCR3 phase
  • how many reports are generated by plugins
  • which oracles/plugins are accepting/transmitting reports

Copy link
Contributor

github-actions bot commented Dec 5, 2024

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

@mateusz-sekara mateusz-sekara requested a review from emate December 5, 2024 13:20
Copy link
Collaborator

@samsondav samsondav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Can you add it to the LLO plugin in core/services/llo/delegate.go?

@mateusz-sekara mateusz-sekara added this pull request to the merge queue Dec 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 6, 2024
@mateusz-sekara mateusz-sekara added this pull request to the merge queue Dec 6, 2024
Merged via the queue into develop with commit f6f2457 Dec 6, 2024
183 checks passed
@mateusz-sekara mateusz-sekara deleted the ocr3-promwrapper branch December 6, 2024 07:54
Copy link
Collaborator

@0xnogo 0xnogo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, i had few minors

result, err := withObservedExecution(p, reports, func() ([]ocr3types.ReportPlus[RI], error) {
return p.ReportingPlugin.Reports(ctx, seqNr, outcome)
})
p.trackReports(reports, len(result))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we are tracking report even when we have an error?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reports should be nil or empty in that case

)

var (
buckets = []float64{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these buckets applicable for all chains? Solana might be significantly faster than Ethereum. I wonder if we should have buckets per chain.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For CCIP it shouldn't be a big deal. Most of the work we do in OCR is reading from the database (ChainReader), doing some data transformation, reaching 3rd party APIs etc. There is still some communication with the chain involved but it shouldn't be that relevant. However, we might come back to that after seeing different values on non-evm chains

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To answer your question: yes, buckets are applicable for all chains

jhweintraub pushed a commit that referenced this pull request Dec 6, 2024
* PoC

* Basic implementation

* Basic implementation

* Basic implementation

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants