add Prometheus to remaining sidecar types

[ghirsch-reddit]

💸 TL;DR

Reproducing #742. This makes all sidecar types expose Prometheus metrics.

📜 Details

JIRA

• https://reddit.atlassian.net/browse/DEVX-753
• https://reddit.atlassian.net/browse/BASE-122

🧪 Testing Steps / Validation

• tested with a service in snoodev - it worked insofar as it ran, though I'm not 100% sure that all of the behavior is unchanged since the service was not using the sidecars in dev before
• a test in snoodev for services with each of the sidecar types
• deploy a service for real with an override to use the new versions, and closely monitor it

and only then update infrared to use these by default

✅ Checks

• CI tests (if present) are passing
• Adheres to code style for repo
• Contributor License Agreement (CLA) completed if not a Reddit employee

[nsheaps]

While I recognize the prom metrics are basically in parity with the statsd metrics, I think we should be aiming for the proper metrics, rather than replacement metrics. Additionally, these need spec changes (even though the same sidecar are used for go and python), and we should ensure any clients used in the sidecars are the baseplate clients so we get that instrumentation for free

[ghirsch-reddit]

Maybe I'm missing something, but it seems like this was getting overwritten below anyway (and same in the other file)

[ghirsch-reddit]

Added the metrics to the spec here https://github.snooguts.net/reddit/baseplate.spec/pull/74

[nsheaps]

Nice job! Do you have some example metrics from testing?

[ghirsch-reddit]

Not good ones 😅 My plan now is to ask a team who actually uses these sidecars in a not-very-critical service if I can do better tests using their service

[ghirsch-reddit]

Okey doke, finally tested it:

• events_publisher: https://github.snooguts.net/reddit/reddit-service-http-request-collector/pull/98
• trace_publisher: https://github.snooguts.net/reddit/reddit-service-realtime/pull/342

Example metrics from the trace_publisher test (forgot to record ones from the other test, but it was pretty similar)

# HELP zipkin_trace_publishes_total Multiprocess metric
# TYPE zipkin_trace_publishes_total counter
zipkin_trace_publishes_total 36.0
# HELP http_client_requests_total Multiprocess metric
# TYPE http_client_requests_total counter
http_client_requests_total{http_client_name="trace_collector",http_method="post",http_response_code="202",http_success="true"} 3.0
# HELP http_client_latency_seconds Multiprocess metric
# TYPE http_client_latency_seconds histogram
http_client_latency_seconds_sum{http_client_name="trace_collector",http_method="post",http_success="true"} 0.031773647060617805
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.0001"} 0.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.0005"} 0.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.001"} 0.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.0025"} 0.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.005"} 0.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.01"} 2.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.025"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.05"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.1"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.25"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="0.5"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="1.0"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="5.0"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="15.0"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="30.0"} 3.0
http_client_latency_seconds_bucket{http_client_name="trace_collector",http_method="post",http_success="true",le="+Inf"} 3.0
http_client_latency_seconds_count{http_client_name="trace_collector",http_method="post",http_success="true"} 3.0
# HELP http_client_active_requests Multiprocess metric
# TYPE http_client_active_requests gauge
http_client_active_requests{http_client_name="trace_collector",http_method="post",pid="7"} 0.0