Add Graphite metrics provider

[mdb]

This adds a Graphite metric provider, closes #911.

Co-authored-by: Mike Ball mike.ball@warnermedia.com
Co-authored-by: Nathan Mische Nathan.Mische@warnermedia.com
Signed-off-by: Mike Ball mike.ball@warnermedia.com
Signed-off-by: Nathan Mische Nathan.Mische@warnermedia.com

[stefanprodan]

Please create a section for Graphite in the metrics docs https://github.com/fluxcd/flagger/blob/main/docs/gitbook/usage/metrics.md

[codecov-commenter]

Codecov Report

Merging #915 (cd6f363) into main (39a3898) will increase coverage by 0.10%.
The diff coverage is 62.50%.

@@            Coverage Diff             @@
##             main     #915      +/-   ##
==========================================
+ Coverage   56.78%   56.89%   +0.10%     
==========================================
  Files          69       70       +1     
  Lines        5727     5825      +98     
==========================================
+ Hits         3252     3314      +62     
- Misses       1990     2020      +30     
- Partials      485      491       +6     

Impacted Files	Coverage Δ
pkg/metrics/providers/factory.go	0.00% <0.00%> (ø)
pkg/metrics/providers/graphite.go	63.82% <63.82%> (ø)
pkg/canary/config_tracker.go	83.40% <0.00%> (+0.15%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39a3898...cd6f363. Read the comment docs.

[stefanprodan]

Was this tested? I haven't used Graphite in ages but this query seem wrong to me, also it doesn't filter the canary pods.

[nmische]

@stefanprodan: We have updated the query to include the {{target}} parameter. One question we do have, is the query intended to consider metrics for both the primary and canary pods, or just the canary pods?

As for the query's correctness, we went ahead and set up a demo Spring Boot app that ships metrics to graphite to test this out. You can find the example here: https://github.com/nmische/spring-boot-graphite. Based on some testing with that application we believe this query correctly calculates the request success rate using the default metrics generated by Spring Boot (with an additional tag app configured to identify the application).

[stefanprodan]

The builtin checks are for the canary pods only, my assumption is that users don't want a canary analysis to fail if the older version (primary) is running into errors, imagine you have some bug in production, if you look at both versions you'll never manage to deliver a fix.

Assuming that your app uses the Kubernetes Deployment name when it reports to Graphite, then the current query is ok, as the primary app name will be {{target}}-primary thus its metrics are not included in the query.

[stefanprodan]

LGTM

Thanks Mike & Nathan 🏅

[stefanprodan]

One last thing missing: in here https://github.com/fluxcd/flagger/blob/main/artifacts/flagger/crd.yaml#L1104 there should be graphite otherwise Kubernetes will reject the canary. After you add it there please run make crd and push all changes.