feat: Add ServiceMonitor for Prometheus operator

[nikolaik]

Description of the change

Backstage instances with the Prometheus /metrics endpoint configured can set metrics.serviceMonitor.enabled to allow prometheus to scrape it's metrics

Disabled by default.

Example diff:

backstage, backstage, ServiceMonitor (monitoring.coreos.com) has been added:
+ # Source: backstage/templates/servicemonitor.yaml
+ apiVersion: monitoring.coreos.com/v1
+ kind: ServiceMonitor
+ metadata:
+   name: backstage
+   namespace: "backstage"
+   labels:
+     app.kubernetes.io/name: backstage
+     helm.sh/chart: backstage-0.16.0
+     app.kubernetes.io/instance: backstage
+     app.kubernetes.io/managed-by: Helm
+     app.kubernetes.io/component: backstage
+ spec:
+   namespaceSelector:
+     matchNames:
+       - "backstage"
+   selector:
+     matchLabels:
+       app.kubernetes.io/name: backstage
+       helm.sh/chart: backstage-0.16.0
+       app.kubernetes.io/instance: backstage
+       app.kubernetes.io/managed-by: Helm
+       app.kubernetes.io/component: backstage
+   endpoints:
+   - port: http-backend
+     path: /metrics

Checklist

• Chart version bumped in Chart.yaml according to semver.
• Variables are documented in the values.yaml and added to the README.md. The helm-docs utility can be used to generate the necessary content. Use helm-docs --dry-run to preview the content.
• JSON Schema generated.
• List tests pass for Chart using the Chart Testing tool and the ct lint command.

[tumido]

Thank you @nikolaik! 🙂 While I like your PR in general, I have conflicting feelings, namely because:

1. ServiceMonitor is a CR, it's not part of vanilla kubernetes and you need to have Prometheus operator installed. This introduces undisclosed dependency.
2. Default, vanilla Backstage instance has no metrics endpoint. This introduces configuration requirement on the Backstage image side.

While I agree with the purpose, I'm not entirely sure this should be part of a canonical chart but rather an opinionated extension of it.

What do others think? cc @backstage/helm-charts

[ChrisJBurns]

I'm not that against this feature if I'm being honest. Mainly because by default it won't be enabled which will allow folks to be able to turn it on if they ever want it. The second point touches on a more crucial point for me though which is that for this functionality to work, the underlying backstage image would have to have the metrics code.

On the whole, I do agree that we do need to be careful on what we allow as non vanilla Kubernetes features, although if I'm being honest, we can probably make those judgements on an Issue/PR basis. This for example is something that I believe would be good to offer in the Chart because monitoring is quite a big deal in Kubernetes - the only caveat to this is that it assumes that the Prometheus operator is installed. Although by default it isn't enabled so for me its not the end of the world.

Interested to hear others thoughts on this

[vinzscam]

Looks good to me. Nice addition @nikolaik! 🙏

[nikolaik]

Thanks for the review @tumido <3 I'll update the PR with your feedback shortly.

And thank you for bringing up the pros and cons of including ServiceMonitor in the chart. I went with disabled by default because using the option requires non default functionality in backstage. I'll point users to the metrics endpoint tutorial and some other relevant docs where it makes sense in the values.yaml file.

One point that is not mentioned is that ServiceMonitor ties into the CNCF ecosystem of packages, which both backstage and prometheus is part of, although I see that the prometheus-operator is not part of the official prometheus project.

[tumido]

One last thing I've noticed... We should maintain consistency with all other resources in label/annotation templating and rendering.

[nikolaik]

One last thing I've noticed... We should maintain consistency with all other resources in label/annotation templating and rendering.

Sorry for being unresponsive, fixing the last bits now...

[tumido]

No worries at all! 🙂

/lgtm
/approve

Looks great now 👍

[tumido]

@nikolaik CI is complaining, can you please fix this?

https://probot.github.io/apps/dco/

[nikolaik]

@nikolaik CI is complaining, can you please fix this?

https://probot.github.io/apps/dco/

Sorry about that, fixed ✅