Grafana image renderer CrashLoopBackoff due to NetPol

[clementnuss]

The issue is the following:
if you enable imageRenderer: true in your Grafana Helm Chart, a default Network Policy will be created, which will only allow the traffic coming from namespace labeled as follows: (here is the Helm chart)

apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: my-ns
  name: my-ns

In my case though, the namespace didn't have this label but only a kubernetes.io/metadata.name: my-ns label.
The liveness probe was therefore failing and my container was in the CrashLoopBackoff state.

Adding the label name: my-ns solves the issue for the grafana pod, but the liveness probe still fails.

[clementnuss]

I really think this should be checked upon. by default, k8s seems to be assigning only the label kubernetes.io/metadata.name: namespace-name to namespaces, but this helm chart is checking on an non-existent name: ... label.

[drmalex07]

I have also stumbled on this. @clementnuss is correct: the label (required to match) does not exist (on the namespace) unless someone manually assigns it (something not always possible). As mentioned here in the official Kubernetes docs, in order to target a namespace by name, the kubernetes.io/metadata.name label should be used (which is generated by Kubernetes if the relevant feature-gate NamespaceDefaultLabelName is enabled).

So, practically, the image renderer service is not accessible from Grafana (unless you entirely disable the network policy by setting imageRenderer.networkPolicy.limitIngress=false in values.yml).

My workaround is to let the network policy enabled, and patch it after a helm install/upgrade command. Since we want to allow traffic only originating from same namespace (as netpol lives in), a better approach is to set namespaceSelector to null (as, by default, it does exactly that (kubectl explain netpol.spec.ingress.from.namespaceSelector)).

A patch would be something like:

kubectl patch --type json netpol \
   -n ${MONITORING_NAMESPACE} \
   ${RELEASE_NAME}-grafana-image-renderer-ingress \
   -p '[{"op": "remove", "path": "/spec/ingress/0/from/0/namespaceSelector"}]'

[tim-felix-lange]

I ran into the same issue and needed to use the workaround of disabling the policy entirely. I'd really love to see this fixed.

[zanhsieh]

@tim-felix-lange @drmalex07 @clementnuss
Does commenting this line would help?

helm-charts/charts/grafana/values.yaml

Line 885 in a1f486d

explicitNamespacesSelector: {}

...
  ## Example:
  ## explicitNamespacesSelector:
  ##   matchLabels:
  ##     role: frontend
  ##   matchExpressions:
  ##    - {key: role, operator: In, values: [frontend]}
  ##
  ## explicitNamespacesSelector: {}                          <=== this line

[clementnuss]

@zanhsieh no, I just created a PR to fix this issue instead