Add DogStatsD Package

[duncanpharvey]

What does this PR do?

Adds the package dogstatsd, a DogStatsD server which uses Saluki for distribution metrics.

Moved upstream from https://github.com/DataDog/datadog-lambda-extension/tree/main/bottlecap/src/metrics

Motivation

Add a DogStatsD server that will allow the Serverless Mini Agent to send runtime and custom metrics from Azure Spring Apps and Azure Functions (to be done in an upcoming PR).

https://datadoghq.atlassian.net/browse/SVLS-5177

Additional Notes

• Updated Rust version from 1.71.1 to 1.76.0 -> changes moved to Upgrade to Rust 1.76.0 #612
  • error[E0445]: crate-private trait MapFieldAccessor in public interface on 1.71.1, 1.72.0, 1.73.0
    • Resolved by upgrading Rust version
  • error[E0658]: use of unstable library feature 'round_ties_even' on 1.74.0, 1.75.0, 1.76.0
    • Resolved by forking Datadog/saluki and replacing round_ties_even() with round()
      • https://github.com/DataDog/saluki-backport
• Use reqwest 0.12.4
  • Error introduced in several other libdatadog crates when dependencies are pulled in for reqwest 0.12.5 => no process-level CryptoProvider available -- call CryptoProvider::install_default() before this point
• Updates to get this passing in CI
  • Add copyright headers
  • Apply formatting
  • Update license
  • Install Protoc Binary for lint, test, and miri workflows
    • https://github.com/protocolbuffers/protobuf/releases
    • Use bash shell when running test commands so PATH is picked up properly in Windows
  • dogstatsd added to Alpine Dockerfile
  • Alpine docker image version updated to 3.19.3 to work with Rust 1.76.0
  • Update benchmark image
    • Update to Rust 1.76.0: https://github.com/DataDog/benchmarking-platform/pull/96
    • Install unzip: https://github.com/DataDog/benchmarking-platform-tools/pull/16
  • Use clippy argument invalid_reference_casting instead of clippy::cast_ref_to_mut
  • Disable miri for integration test due to error

error: unsupported operation: can't call foreign function `socket` on OS `linux`
  --> /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/sys/unix/net.rs:28:18
   |
28 |     let socket = syscall!(socket(domain, socket_type, 0))?;

How to test the change?

• Already present unit tests
• Integration test to confirm happy path works

[bwoebi]

Currently the sidecar crate is using cadence for its dogstatsd needs. Maybe, as a first use case, and to show the crate working properly, the sidecar might be switched over to this crate?

[duncanpharvey]

@bwoebi The original motivation for this is to use this dogstatsd crate in the serverless so I've done some initial testing to confirm that it works there. But the sidecar can definitely be switched over to use this crate as well!

I should have this PR ready to review soon once I get CI passing.

[bwoebi]

Ah, I'm seeing you're using 1.77.2. We're currently locked in php to use at most rust 1.76.0 until the next alpine release.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 90.65126% with 89 lines in your changes missing coverage. Please review.

Project coverage is 73.18%. Comparing base (00cf6dc) to head (4e414d6).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #606      +/-   ##
==========================================
+ Coverage   72.69%   73.18%   +0.48%     
==========================================
  Files         246      252       +6     
  Lines       34990    35959     +969     
==========================================
+ Hits        25437    26317     +880     
- Misses       9553     9642      +89     

Components	Coverage Δ
crashtracker	20.53% <ø> (ø)
datadog-alloc	98.73% <ø> (ø)
data-pipeline	90.12% <ø> (ø)
data-pipeline-ffi	0.00% <ø> (ø)
ddcommon	83.08% <ø> (ø)
ddcommon-ffi	69.52% <ø> (ø)
ddtelemetry	59.10% <ø> (ø)
ipc	83.63% <ø> (ø)
profiling	84.26% <ø> (ø)
profiling-ffi	77.42% <ø> (ø)
serverless	0.00% <ø> (ø)
sidecar	40.12% <ø> (ø)
sidecar-ffi	0.00% <ø> (ø)
spawn-worker	50.36% <ø> (ø)
tinybytes	91.66% <ø> (ø)
trace-mini-agent	70.88% <ø> (ø)
trace-normalization	98.25% <ø> (ø)
trace-obfuscation	95.73% <ø> (ø)
trace-protobuf	77.67% <ø> (ø)
trace-utils	92.86% <ø> (-0.42%)	⬇️

[duncanpharvey]

I'd like to avoid bumping the rust version but I'm running into these errors on versions below 1.77.x. Any ideas for how to resolve these errors without upgrading rust?

• error[E0445]: crate-private trait MapFieldAccessor in public interface on 1.71.1, 1.72.0, 1.73.0
• error[E0658]: use of unstable library feature 'round_ties_even' on 1.74.0, 1.75.0, 1.76.0

Update: I forked saluki into https://github.com/DataDog/saluki-backport and replaced round_ties_even() with round().

[VianneyRuhlmann]

We already have a ddsketch implementation in libdatadog, there is one in mp-rs and there is one on crates.io that is not owned by DD. We should probably all agree on a single implementation.

[duncanpharvey]

Interesting, this package uses the same implementation used by the Datadog Lambda Extension (owned by the Serverless team). Since I intend to use this DogStatsD package for a Serverless solution in Azure I'm inclined to stay consistent with the implementations.

https://github.com/DataDog/datadog-lambda-extension/tree/main/bottlecap/src/metrics

[alexgallotta]

I tried to remove the duplication but I read that this one is slightly different and optimized for the agent. So it might be not possible
https://github.com/DataDog/saluki/blob/main/lib/ddsketch-agent/src/lib.rs#L149

/// This implementation is subtly different from the open-source implementations of DDSketch, as Datadog made some
/// slight tweaks to configuration values and in-memory layout to optimize it for insertion performance within the
/// agent.

[morrisonlevi]

slight tweaks to configuration values and in-memory layout to optimize it for insertion performance within the agent.

I'm not familiar with sketches, but optimizing in-memory layout and optimizing for insertion performance also sound relevant to a library, if it is feasible. Any insight as to whether it's appropriate or not?

[alexgallotta]

I am afraid I did not understand your question, can you elaborate more?

[tobz]

👋🏻

I'm the original author of the ported DDSketch implementation being used here. As the doc comments allude to, the main goal of the port was to maintain fidelity with the Agent's own DDSketch implementation, which at the time (and to this day) is not based on the open-source Go implementation of DDSketch.

I didn't write the original Agent-specific implementation, so I can't speak to all of the decisions made there, but I also tried to port over some of the code comments to, again, maintain fidelity with the Agent implementation.

While it probably behooves us, in a general sense, to have One True Implementation in Rust... maintaining compatibility with the Datadog Agent in the most important factor in this case.

[morrisonlevi]

Does this spawn any threads on the client side of things? I see a test server which probably does, and that's okay. I scanned through it, but it's a big PR so I may have missed something.

Most languages need to have control over threads so they can mask off appropriate signals, register them with the runtime, etc, so the libraries generally need to design APIs that don't spawn threads.

[duncanpharvey]

Does this spawn any threads on the client side of things? I see a test server which probably does, and that's okay. I scanned through it, but it's a big PR so I may have missed something.

Most languages need to have control over threads so they can mask off appropriate signals, register them with the runtime, etc, so the libraries generally need to design APIs that don't spawn threads.

@morrisonlevi Correct, the only thread spawned is in the test.

libdatadog/dogstatsd/tests/integration_test.rs

Lines 83 to 85 in 5d181a0

	tokio::spawn(async move {
	dogstatsd_client.spin().await;
	});