[loki] /tmp read-only file system on backend pods (3 targets mode)

[Tazmaniac]

Bug #609 again:

A fix similar to gafana#7695 should be applied to production/helm/templates/backend/statefulset-backend.yaml to fix read-only tmp in 3 targets mode (read,write,backend).

[Tazmaniac]

actually I get :

level=warn ts=2023-02-09T09:12:02.023072398Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675924744929872614 err="open /tmp/marker-view-3389782386: read-only file system"
level=warn ts=2023-02-09T09:12:02.023086198Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675925344449163411 err="open /tmp/marker-view-833108166: read-only file system"
level=warn ts=2023-02-09T09:12:02.023100098Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675925344733780518 err="open /tmp/marker-view-4267517007: read-only file system"
level=warn ts=2023-02-09T09:12:02.023115698Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675925345190749995 err="open /tmp/marker-view-538712842: read-only file system"
level=warn ts=2023-02-09T09:12:02.023129397Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675925943778624780 err="open /tmp/marker-view-1817898873: read-only file system"
level=warn ts=2023-02-09T09:12:02.023142397Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675925944057887045 err="open /tmp/marker-view-96155975: read-only file system"
level=warn ts=2023-02-09T09:12:02.023156297Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675925944496443406 err="open /tmp/marker-view-284468473: read-only file system"
level=warn ts=2023-02-09T09:12:02.023169097Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675926544174855298 err="open /tmp/marker-view-3113922886: read-only file system"
level=warn ts=2023-02-09T09:12:02.023181997Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675926544542977731 err="open /tmp/marker-view-3487964693: read-only file system"
level=warn ts=2023-02-09T09:12:02.023195097Z caller=marker.go:213 msg="failed to process marks" path=/var/loki/compactor-retention/retention/markers/1675926545100041031 err="open /tmp/marker-view-2200564611: read-only file system"

[MaxDiOrio]

I randomly had this same issue in my setup. 3 Loki-backend pods, 2 were normal, 1 was using 45GB of RAM and repeating the failed to process marks error.

I killed the "bad" pod and it came back up fine so far. Nothing would have prompted this to happen that I know of - it was a slow burn.