Misc improvements to cluster_controller/logs_controller #2570
Conversation
Test Results 7 files ±0 7 suites ±0 3m 25s ⏱️ - 1m 27s For more details on these errors, see this check. Results for commit 27e8e07. ± Comparison against base commit c61c551. ♻️ This comment has been updated with latest results. |
|
Still very similar experience to before - about 10 min after the end of the test, all nodes remain unresponsive to ingress requests. Test results at: #2565 (comment) |
AhmedSoliman
force-pushed
the
pr2570
branch
5 times, most recently
from
29c16a7 to
792d2c3
Compare
|
I'll revisit some of the logging again before the releasae. |
| tokio::time::sleep(delay).await; | ||
| } | ||
|
|
||
| match bifrost.admin().seal(log_id, segment_index).await { | ||
| Ok(sealed_segment) => { | ||
| if sealed_segment.tail.is_sealed() { |
There was a problem hiding this comment.
Is sealed_segment.tail.is_sealed() reliable or can it return stale information?
There was a problem hiding this comment.
if seal returns Ok, the segment must be sealed.
For futures where we want to observe if the are too slow or overdue, this extension trait adds the ability to log with automatic exponential backoff on a certain tracing level, then optionally upgrade the level and reports once it crosses the "overdue" threshold.
## Usage:
```rust
use restate_futures_util::overdue::OverdueLoggingExt;
let result = my_future
.log_slow_after(
Duration::from_secs(1),
tracing::Level::INFO,
"Doing something important.",
)
// optional
.with_overdue(
Duration::from_secs(30),
tracing::Level::WARN
).await;
// prints
// INFO "[slow] Doing something important. elapsed=1s");
// INFO "[slow] Doing something important. elapsed=2s");
// INFO "[slow] Doing something important. elapsed=4s");
// INFO "[slow] Doing something important. elapsed=8s");
// ...
// WARN "[overdue] Doing something important. elapsed=30s");
// WARN "[completed] Doing something important. elapsed=35.9s");
```
Some of the debug messages will likely be changed to trace before the release cut off date
Some of the debug messages will likely be changed to trace before the release cut off date
Stack created with Sapling. Best reviewed with ReviewStack.