Make metrics collection optional/faster #1147

QuerthDP · 2024-12-10T13:28:48Z

This patch contains:

Implementation of lock-free histogram with hot and cold bucket pools
Introduction of crate feature "metrics" to make using them optional
Functionality to gather latency statistics via histogram snapshots
Implementation of rates of queries per second based on cpp-driver implementation.
Initial implementation of connection metrics (changes required after review)

Fixes: #330

Pre-review checklist

I have split my patch into logically separate commits.
All commit messages clearly explain what they change and why.
I added relevant tests for new features and bug fixes.
All commits compile, pass static checks and pass test.
PR description sums up the changes and reasons why they should be introduced.
I have provided docstrings for the public items that I want to introduce.
I have adjusted the documentation in ./docs/source/.
I added appropriate Fixes: annotations to PR description.

github-actions · 2024-12-10T13:35:03Z

cargo semver-checks detected some API incompatibilities in this PR.
Checked commit: 27e4e86

See the following report for details:

cargo semver-checks output

./scripts/semver-checks.sh --baseline-rev bb824b61cefb8426cc00aef563ce27882cf0aa7c
+ cargo semver-checks -p scylla -p scylla-cql --baseline-rev bb824b61cefb8426cc00aef563ce27882cf0aa7c
     Cloning bb824b61cefb8426cc00aef563ce27882cf0aa7c
    Building scylla v0.15.0 (current)
       Built [  23.408s] (current)
     Parsing scylla v0.15.0 (current)
      Parsed [   0.054s] (current)
    Building scylla v0.15.0 (baseline)
       Built [  22.781s] (baseline)
     Parsing scylla v0.15.0 (baseline)
      Parsed [   0.051s] (baseline)
    Checking scylla v0.15.0 -> v0.15.0 (no change)
     Checked [   0.108s] 107 checks: 105 pass, 2 fail, 0 warn, 0 skip

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.38.0/src/lints/inherent_method_missing.ron

Failed in:
  Metrics::new, previously in file /home/runner/work/scylla-rust-driver/scylla-rust-driver/target/semver-checks/git-bb824b61cefb8426cc00aef563ce27882cf0aa7c/8e88ff90ea17547a103649e6571db48be24ce469/scylla/src/observability/metrics.rs:35

--- failure struct_with_no_pub_fields_changed_type: public API struct with no public fields is no longer a struct ---

Description:
A struct without pub fields became an enum or union, breaking pattern matching.
        ref: https://internals.rust-lang.org/t/rest-patterns-foo-should-match-non-struct-types/21607
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.38.0/src/lints/struct_with_no_pub_fields_changed_type.ron

Failed in:
  struct scylla::observability::metrics::MetricsError became enum in file /home/runner/work/scylla-rust-driver/scylla-rust-driver/scylla/src/observability/metrics.rs:11

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  47.854s] scylla
    Building scylla-cql v0.4.0 (current)
       Built [  11.326s] (current)
     Parsing scylla-cql v0.4.0 (current)
      Parsed [   0.034s] (current)
    Building scylla-cql v0.4.0 (baseline)
       Built [  11.290s] (baseline)
     Parsing scylla-cql v0.4.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking scylla-cql v0.4.0 -> v0.4.0 (no change)
     Checked [   0.111s] 107 checks: 107 pass, 0 skip
     Summary no semver update required
    Finished [  23.569s] scylla-cql
make: *** [Makefile:61: semver-rev] Error 1

QuerthDP · 2025-01-13T12:43:27Z

Changelog:

rebased on main

muzarski

I still need to review commits that introduce Meter and connection metrics. Posting this review early, because there are some matters to discuss.

muzarski · 2025-01-15T12:46:06Z

scylla/src/observability/lock_free_histogram.rs

+#[derive(Error, Debug, PartialEq)]
+pub enum LFError {
+    #[error("invalid use of histogram")]
+    HistogramErr(#[from] histogram::Error),
+    #[error("could not lock the snapshot mutex")]
+    Mutex,
+}


Please add a docstring. Even the simplest one - mentioning that it indicates some failure during LockFreeHistogram's operation

I don't like the name. Since this is a public type, the error name should be more descriptive (current version would be fine for driver's internal error type). I suggest a verbose name such as LockFreeHistogramError (WDYT @wprzytula ?)

The additional context is not displayed for the variants. You can display it via {0}, such as #[error("Invalid use of histogram: {0}")]. Also, please begin the error messages with capital letter (this is a convention we currently use in the driver).

HistogramErr -> HistogramError (variant name)

HistogramErr variant currently has a type from a pre-1.0.0 crate. We need to think what to do with this. Corresponding issue: Remove types from pre-1.0 crates from the public API, or hide them behind features #771. I see that later in this PR, you introuce a metrics feature, so I assume that this error type will be hidden behind it as well. Is it OK if feature name does not directly correspond to the unstable crate's name? In this case, the unstable crate is histogram, and feature name is metrics. I'm not sure how this interacts with API stability. cc: @piodul @Lorak-mmk

scylla/src/observability/lock_free_histogram.rs

muzarski · 2025-01-15T12:49:50Z

scylla/src/observability/lock_free_histogram.rs

+    Mutex,
+}
+
+pub struct LockFreeHistogram {


This needs a docstring as well. Not only this is a public type (does it need to be public?), but its methods contain some very complex logic, which could be briefly explained here.

Also, it would be nice to mention the motivation behind this struct. I understand the logic (after reviewing the methods), but I still don't quite get WHY we need it. Why can't we just use the AtomicHistogram from histogram crate? Are there any races that can occur without this additional layer of synchronization?

Okay, so there are a couple of things to mention here, but I'll start from the beginning.

In the issue description here it is noted that no suitable solution for the problem was found on crates.io (I believe AtomicHistogram already existed by the time of writing of that issue), so I assumed it must have had a flaw. And, as it turned out, it did.

I highly recommend reading through the issue I opened up on the crate histogram repo, where I explain all of the motivation in detail, but in short: the .load() method has no synchronisation with increments, which causes a logical race (the state of the loaded histogram is dependent on the speed at which it is loaded).

The idea of some sort of a lock-free algorithm was also proposed in the "Make metrics optional" issue, along with sharding, which I considered potentially harder to implement, thus I went with a lock-free algorithm.

However (!), the LockFreeHistogram's implementation comes with potential drawbacks in terms of performance (in comparison to AtomicHistogram, not a global mutex) due to global atomic counters accessed upon each bucket increment.

I haven't managed to run any benchmarks in this regard, nor do I have concrete examples of cases when AtomicHistogram's implementation yields a very significant error in results (though I did come up with some ideas and calculations; they can be found on my linked issue on crate histogram's repo). Therefore, the decision of which histogram implementation to incorporate into this driver is up to you. I've just provided a safe alternative and done some research.

Also, should you choose to go with AtomicHistogram, the change will be rather effortless, as I maintained the API schema used in crate histogram for my implementation.

Thank you for the explanation! I think part of this deserves to be put in the docstring.

Also, should you choose to go with AtomicHistogram, the change will be rather effortless, as I maintained the API schema used in crate histogram for my implementation.

This is great. Also, please unpub (pub(crate)) the LockFreeHistogram and its methods. Only then will we be 100% sure that if we ever decide to drop/modify LockFreeHistogram, such change will not be API breaking.

Which histogram implementation we want to incorporate to the driver? Well, this needs to be discussed. I'd wait until @Lorak-mmk and @wprzytula review the code.

I unpubbed LockFreeHistogram and its methods, but now I can see it might be difficult to make such a change completely non-API-breaking. That's because LockFreeHistogramError is propagated as MetricsError's cause and thus needs to be pub. Upon change to AtomicHistogram this error struct would be removed entirely.

I'm not yet sure how we could go around this issue.

Ah, good catch. There are some workarounds for this, however. For example, we could always hide underlying LockFreeHistogramError under Arc<dyn Error>. I wouldn't worry about this now. Let's wait for others to join the discussion.

For what it's worth, while .load() is not atomic wrt concurrent increments into the histogram. I'd still consider using the AtomicHistogram which is used in metrics for both rpc-perf and Pelikan.

As I said in the issue opened in rustcommon, I'm not sure that the potential skew here would be meaningful. But I do welcome some concrete details if such skew does prove to be meaningful.

The TLDR is I'm not sure how much it matters whether it's on one side or the other of loading a histogram. If you envision periodically snapshotting the histogram, it seems you have to accept that the latency is already being recorded at the tail end of the event. Imagine a request that takes a long time, that latency gets incremented after the fact, when the service might be back to responding quickly.

My feeling is that ultimately this is all an approximation and I've found the AtomicHistogram as in the histogram crate to be satisfactory for projects I work on.

scylla/src/observability/lock_free_histogram.rs

muzarski · 2025-01-15T12:56:43Z

scylla/src/observability/lock_free_histogram.rs

+impl Default for LockFreeHistogram {
+    fn default() -> Self {
+        // Config: 64ms error, values in range [0ms, ~262_000ms].
+        // Size: (2^13 + 5 * 2^12) * 8B * 2 ~= 450kB.
+        let grouping_power = 12;
+        let max_value_power = 18;
+        LockFreeHistogram::new(grouping_power, max_value_power)
+    }
+}


Where are these defaults taken from?

Since the histogram crate no longer provides defaults, I had to come up with some choice here. It was my best guess at what might be needed, though that is obviously to be discussed and modified if needed.

You may find our calculator useful while evaluating what parameters are appropriate: https://observablehq.com/@iopsystems/h2-histogram

❓ @NikodemGapski Have you consulted the above calculator about the defaults?

I mean, I had made my guess according to the mentioned calculator, but I can't know if it meets the needs of the users of this driver (the range and absolute error I noted in the comment for reference). It is your decision whether or not to change it.

scylla/src/observability/metrics.rs

examples/Cargo.toml

scylla/src/observability/lock_free_histogram.rs

scylla/src/observability/metrics.rs

wprzytula · 2025-01-17T13:20:50Z

scylla/src/observability/metrics.rs

+/// Snapshot is a structure that contains histogram statistics such as
+/// min, max, mean, standard deviation, median, and most common percentiles
+/// collected in a certain moment.
+#[derive(Debug)]
+pub struct Snapshot {
+    pub min: u64,
+    pub max: u64,
+    pub mean: u64,
+    pub stddev: u64,
+    pub median: u64,
+    pub percentile_75: u64,
+    pub percentile_95: u64,
+    pub percentile_98: u64,
+    pub percentile_99: u64,
+    pub percentile_99_9: u64,
+}


🔧 For future compatibility, let's either:

make this struct #[non_exhaustive] to allow adding more fields in the future without breaking the API;

or make all those fields private and expose a getter for each field.

Which one do you find a more suitable solution? @Lorak-mmk @muzarski

So far I've used the #[non_exhaustive] macro, but this case is still open for discussion.

scylla/src/observability/metrics.rs

QuerthDP · 2025-01-26T16:14:25Z

Changelog:

rebased on main

docs/source/metrics/metrics.md

examples/basic.rs

scylla/src/client/session.rs

scylla/src/network/connection.rs

wprzytula · 2025-01-27T09:00:23Z

scylla/src/policies/speculative_execution.rs

+    #[cfg(not(feature = "metrics"))]
+    fn retry_interval(&self, _: &Context) -> Duration {
+        warn!("PercentileSpeculativeExecutionPolicy requires the 'metrics' feature to work as intended, defaulting to 100 ms");
+        Duration::from_millis(100)
+    }


💭 Perhaps we should hide the PercentileSpeculativeExecutionPolicy behind the metrics feature, too? @Lorak-mmk @muzarski

scylla/src/observability/metrics.rs

scylla/src/network/connection_pool.rs

QuerthDP · 2025-01-27T18:34:32Z

Changelog:

rebased @wprzytula's refactor on the branch and squashed his contribution into b3ae501
adjusted EWMA docstrings
fixed invalid unwraps in examplary code
added more clarity to the request timeout
resolved nitpicks

Implement a lock-free histogram with hot and cold bucket pools. Implementation inspired by Prometheus library in Go and the histogram crate. This commit also updates crate histogram dependency from 0.6.9 to 0.11.1 for atomic functionalities and cleaner error handling (since the update introduced breaking changes to crate histogram's API, it was merged in one commit with the new, lock-free histogram implementation).

Implement metrics making use of Lock-Free Histogram to measure query latencies. Added metrics are provided by the Snapshot structure containing metrics such as: min, max, mean, median, standard deviation and various percentiles. Co-authored-by: NikodemGapski <n36vayjb@duck.com>

Implement rates similar to those available in cpp-driver. [CPP-Driver implementation](https://github.com/scylladb/cpp-driver/blob/9d6b05c9d4ebd0a6d7006af4df3e33fcdf956eeb/src/metrics.hpp#L39C1-L252C5)

Implement gathering of connection metrics like total number of active connections, connection timeouts and request timeouts. Co-authored-by: Wojciech Przytuła <wojciech.przytula@scylladb.com>

Add metrics crate feature which enables usage and gathering of metrics. Therefore everyone willing to use metrics in their code is required to add metrics feature in their Cargo.toml file or compile otherwise with --features metrics flag. Additionally, add a CI step with cargo checks for this feature.

Inform that metrics may now only be used under crate feature 'metrics'. Mention new metrics in documentation and show an example how to collect them. Adjust examples to include new metrics.

As the user should not be able to create metrics instance otherwise than by `get_metrics()` function, the `Metrics::new()` method shall be set to pub(crate) visibility to support only internal usage.

QuerthDP · 2025-01-27T19:43:14Z

Changelog:

rebased on main
bumped Cargo.lock.msrv as it was changed by other commits

I think this PR should pass all the CI now.

wprzytula

💯 Apart from the open discussion over PercentileSpeculativeExecutionPolicy, LGTM!

🎉 You've done a truly great job!

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from 74b7fa3 to d989a59 Compare December 11, 2024 10:50

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from d7b32d0 to f9a7153 Compare January 7, 2025 22:28

wprzytula added this to the 0.16.0 milestone Jan 9, 2025

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from bff846c to 88de7ff Compare January 12, 2025 18:46

QuerthDP marked this pull request as ready for review January 12, 2025 19:01

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from 88de7ff to 185ff57 Compare January 13, 2025 12:42

wprzytula requested review from wprzytula, Lorak-mmk and muzarski January 13, 2025 12:45

github-actions bot added the semver-checks-breaking cargo-semver-checks reports that this PR introduces breaking API changes label Jan 13, 2025

NikodemGapski force-pushed the 330-make-metrics-collection-optional/faster branch from 185ff57 to 7be0e38 Compare January 14, 2025 19:03

muzarski reviewed Jan 15, 2025

View reviewed changes

NikodemGapski force-pushed the 330-make-metrics-collection-optional/faster branch from 7be0e38 to 2f6ae32 Compare January 15, 2025 18:31

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from 2f6ae32 to b36ef1f Compare January 16, 2025 11:46

NikodemGapski force-pushed the 330-make-metrics-collection-optional/faster branch from 0bcde8e to cf43ed2 Compare January 16, 2025 17:20

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from cf43ed2 to 4392cb8 Compare January 16, 2025 19:57

wprzytula requested changes Jan 17, 2025

View reviewed changes

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from 4392cb8 to 6dab731 Compare January 22, 2025 11:52

QuerthDP requested a review from wprzytula January 22, 2025 12:14

wprzytula assigned QuerthDP Jan 22, 2025

wprzytula requested a review from muzarski January 22, 2025 13:36

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from 6dab731 to 32537f9 Compare January 26, 2025 16:13

wprzytula requested changes Jan 27, 2025

View reviewed changes

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from bcb3b77 to f297b75 Compare January 27, 2025 18:33

QuerthDP requested a review from wprzytula January 27, 2025 18:35

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from f297b75 to b31d836 Compare January 27, 2025 19:13

NikodemGapski and others added 8 commits January 27, 2025 20:38

metrics: add rates of queries per second

d72e9f3

Implement rates similar to those available in cpp-driver. [CPP-Driver implementation](https://github.com/scylladb/cpp-driver/blob/9d6b05c9d4ebd0a6d7006af4df3e33fcdf956eeb/src/metrics.hpp#L39C1-L252C5)

metrics: add connection metrics

cbc2bb9

Implement gathering of connection metrics like total number of active connections, connection timeouts and request timeouts. Co-authored-by: Wojciech Przytuła <wojciech.przytula@scylladb.com>

docs: adjust metrics documentation

8241c70

Inform that metrics may now only be used under crate feature 'metrics'. Mention new metrics in documentation and show an example how to collect them. Adjust examples to include new metrics.

metrics: unpub Metrics::new()

41b2f5a

As the user should not be able to create metrics instance otherwise than by `get_metrics()` function, the `Metrics::new()` method shall be set to pub(crate) visibility to support only internal usage.

node: fix heavily outdated docstring

27e4e86

QuerthDP force-pushed the 330-make-metrics-collection-optional/faster branch from b31d836 to 27e4e86 Compare January 27, 2025 19:41

wprzytula approved these changes Jan 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make metrics collection optional/faster #1147

Make metrics collection optional/faster #1147

QuerthDP commented Dec 10, 2024 •

edited

Loading

github-actions bot commented Dec 10, 2024 •

edited

Loading

QuerthDP commented Jan 13, 2025

muzarski left a comment

muzarski Jan 15, 2025

muzarski Jan 15, 2025

NikodemGapski Jan 15, 2025

muzarski Jan 15, 2025 •

edited

Loading

NikodemGapski Jan 15, 2025

muzarski Jan 15, 2025

brayniac Jan 17, 2025

muzarski Jan 15, 2025

NikodemGapski Jan 15, 2025

brayniac Jan 17, 2025

wprzytula Jan 27, 2025

NikodemGapski Jan 27, 2025

wprzytula Jan 17, 2025

QuerthDP Jan 22, 2025

QuerthDP commented Jan 26, 2025

wprzytula Jan 27, 2025

QuerthDP commented Jan 27, 2025

QuerthDP commented Jan 27, 2025 •

edited

Loading

wprzytula left a comment

Make metrics collection optional/faster #1147

Are you sure you want to change the base?

Make metrics collection optional/faster #1147

Conversation

QuerthDP commented Dec 10, 2024 • edited Loading

Pre-review checklist

github-actions bot commented Dec 10, 2024 • edited Loading

QuerthDP commented Jan 13, 2025

muzarski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muzarski Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuerthDP commented Jan 26, 2025

Choose a reason for hiding this comment

QuerthDP commented Jan 27, 2025

QuerthDP commented Jan 27, 2025 • edited Loading

wprzytula left a comment

Choose a reason for hiding this comment

QuerthDP commented Dec 10, 2024 •

edited

Loading

github-actions bot commented Dec 10, 2024 •

edited

Loading

muzarski Jan 15, 2025 •

edited

Loading

QuerthDP commented Jan 27, 2025 •

edited

Loading