[CRE-1028] Optimize beholder validator in system tests: structured init and message handling (part 2)

[chudilka1]

CRE-1028: Optimize beholder validator in system tests: structured init and message handling (part 2)

In beholder.go:

• Added exponential backoff retry logic to protobuf registration on error (which usually takes place in CI, the first test runs)
• Improved error handling for schema registration failures

In beholder_provider.go:

• Code structure: modularized; extracted more helpers; improved naming.
• Configuration: Added ConsumerOptions; larger buffers (40→200 / 1→100); session timeout 10s→20s; async/sync commit modes; nanosecond group IDs.
• Reliability: Added Kafka + heartbeat preflight checks; heartbeat retries (3×, 5s); exponential backoff (2s→30s, 10% jitter); safe cleanup & error-checked closes; fixed timer leaks.
• Consumer Behavior: Manual commits; configurable isolation; client.id added; stronger metadata validation; uses SubscribeTopics.
• Message Processing: Blocking reads with timeout; removed old timestamp filter; only UserLogs reset timer; richer logs (offsets, partitions).
• Observability: Detailed structured logs (subscription, partitions, retries, backoff); improved error wrapping; clear success/failure reporting.
• API: Same signature; internal defaults; fail-fast validation; startup timeout (2 min).
• Performance: Larger buffers, efficient blocking reads, reduced retry churn.
• Removed: Panic recovery, channel-full retry, old timestamp filtering, ticker polling.
• New: Heartbeat validation (msg="heartbeat"), multi-topic support, full metadata validation, startup timeout guard.

[Tofel]

proto registration retry is and should be handled in the CTF, not here.

[Tofel]

why not use avast's retry?

[chudilka1]

Regarding this specific case, I have several concerns, as I am uncertain whether the retry library can handle the following:

• When using retry.Do(), we need to carefully determine when to close the output channel (e.g. defer close(out)) - only after all retries are exhausted or the context is canceled.
• A jitter feature to prevent thundering herd problems will be lost, along with some essential debug logging.
• The readyCh must be carefully managed - it should only signal once, at the first successful connection, not on every retry. This requires state tracking outside the retry function.

[Copilot]

Pull Request Overview

This PR implements part 2 of the Beholder validator optimization in system tests, focusing on enhanced reliability, error handling, and message processing. It significantly restructures the Kafka consumer implementation to address timer leaks, improve connection handling, and add comprehensive validation.

• Enhanced Kafka consumer with exponential backoff, heartbeat validation, and structured error handling
• Added fail-fast validation for Beholder subscription errors during initialization
• Updated dependencies and improved documentation/comments

Reviewed Changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
system-tests/tests/test-helpers/t_helpers.go	Added fail-fast error checking for Beholder subscription initialization
system-tests/tests/test-helpers/beholder_provider.go	Complete restructure with improved Kafka consumer, validation, and error handling
system-tests/tests/regression/cre/v2_evm_regression_test.go	Fixed typo and increased timeout duration
system-tests/tests/regression/cre/cre_regression_suite_test.go	Updated documentation comments
system-tests/tests/go.mod	Moved retry-go dependency from indirect to direct
go.md	Added dependency relationship for chainlink-evm/gethwrappers
core/scripts/go.mod	Moved retry-go dependency from indirect to direct
core/scripts/cre/environment/examples/workflows/v2/proof-of-reserve/cron-based/main.go	Replaced hardcoded ABI with generated bindings
core/scripts/cre/environment/examples/workflows/v2/proof-of-reserve/cron-based/go.mod	Added chainlink-evm/gethwrappers dependency
core/scripts/cre/environment/environment/beholder.go	Added retry logic for protobuf registration
.changeset/slick-drinks-like.md	Changeset for PoR workflow binding updates
.changeset/eight-tips-bathe.md	Changeset for Beholder optimization

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube