Compaction job is failing: no space left on device

[rosh0450]

ts=2024-11-21T03:59:27.340973454Z caller=bucket_compactor.go:271 level=error component=compactor user=tenant-constellation groupKey=0@17241709254077376921-merge--1731628800000-1731715200000 job_type=merge minTime="2024-11-15 00:00:00.03 +0000 UTC" maxTime="2024-11-16 00:00:00 +0000 UTC" msg="compaction job failed" duration=4m40.939156002s duration_ms=280939 err="compact blocks 01JD55EB93ATWRYQ23ZG8GY026,01JD5B4F5F7FRRAMY968MP9P4H: 2 errors: preallocate: no space left on device; sync /data/compact/0@17241709254077376921-merge--1731628800000-1731715200000/01JD6DZ7M0N5KXH9YZPCGJ98XJ.tmp-for-creation/chunks/000002: file already closed" block_count=2 ts=2024-11-21T04:04:33.389292912Z caller=bucket_compactor.go:271 level=error component=compactor user=tenant-constellation groupKey=0@17241709254077376921-merge--1731628800000-1731715200000 job_type=merge minTime="2024-11-15 00:00:00.03 +0000 UTC" maxTime="2024-11-16 00:00:00 +0000 UTC" msg="compaction job failed" duration=4m47.420991746s duration_ms=287420 err="compact blocks 01JD55EB93ATWRYQ23ZG8GY026,01JD5B4F5F7FRRAMY968MP9P4H: 2 errors: preallocate: no space left on device; sync /data/compact/0@17241709254077376921-merge--1731628800000-1731715200000/01JD6E8DVF374Z4MFZX1YNGEN1.tmp-for-creation/chunks/000002: file already closed" block_count=2 ts=2024-11-21T04:09:59.829872629Z caller=bucket_compactor.go:271 level=error component=compactor user=tenant-constellation groupKey=0@17241709254077376921-merge--1731628800000-1731715200000 job_type=merge minTime="2024-11-15 00:00:00.03 +0000 UTC" maxTime="2024-11-16 00:00:00 +0000 UTC" msg="compaction job failed" duration=5m4.07618179s duration_ms=304076 err="compact blocks 01JD55EB93ATWRYQ23ZG8GY026,01JD5B4F5F7FRRAMY968MP9P4H: 2 errors: preallocate: no space left on device; sync /data/compact/0@17241709254077376921-merge--1731628800000-1731715200000/01JD6EJJXA7FVY5CQEEBJBYFD9.tmp-for-creation/chunks/000002: file already closed" block_count=2 ts=2024-11-21T04:10:00.273151396Z caller=compactor.go:690 level=error component=compactor msg="failed to compact user blocks" user=tenant-constellation err="compaction: group 0@17241709254077376921-merge--1731628800000-1731715200000: compact blocks 01JD55EB93ATWRYQ23ZG8GY026,01JD5B4F5F7FRRAMY968MP9P4H: 2 errors: preallocate: no space left on device; sync /data/compact/0@17241709254077376921-merge--1731628800000-1731715200000/01JD6EJJXA7FVY5CQEEBJBYFD9.tmp-for-creation/chunks/000002: file already closed"

For the last few hours, i have been getting an error from the compactor saying: "no space left on device"
So I increased the PVC size and restarted the compactor pods but this error still exists.
Not sure what is happening.

The runbook says to increase the size so I did https://grafana.com/docs/mimir/latest/manage/mimir-runbooks/#mimircompactorhasrunoutofdiskspace

This is the compactor dashboard

[rosh0450]

Once I increased the PVC size for the compactor, there was no error in the compactor pod's log.
the compaction job has also been successful

But the alerts are still fired

The instances mentioned along with the alerts are the IPs for the old compactor pods, I guess they should go away after 24h.

[rosh0450]

Once I increased the PVC size for the compactor, there was no error in the compactor pod's log.
the compaction job has also been successful

But the alerts are still fired

The instances mentioned along with the alerts are the IPs for the old compactor pods, I guess they should go away after 24h.