thanos-sidecar : hard link block : operation not permitted

[khjean]

Thanos, Prometheus and Golang version used:
prometheus : v2.43.0
thanos : v0.32.4

Object Storage Provider:
minio

What happened:
I am running Prometheus and Thanos sidecar on a single VM in a docker environment.

Prometheus and sidecar are looking at the same tsdb path using volume monut.

When I proceed with upgrading the version of thanos-sidecar, the following error occurs:

thanos-sidecar version upgrade: v0.30.2 -> v0.32.4

Error details:
ts=2023-10-16T05:03:54.387051594Z caller=sidecar.go:354 level=warn err="upload 01HCVDP0JJ2SJVSY1GKY7EK54K: hard link block: hard link file chunks/000001: link /prometheus/01HCVDP0JJ2SJVSY1GKY7EK54K/chunks/000001 /prometheus/thanos/upload/01HCVDP0JJ2SJVSY1GKY7EK54K/chunks/000001: operation not permitted" uploaded=0

It seems to be caused by a breaking change in which the default user changed between version upgrades. How should sidecar be operated in the above situation?

breaking change :
#6107 breaking ⚠️ Change default user id in container image from 0(root) to 1001 in v0.32.0

What you expected to happen:
thanos sidecar ship the metrics

How to reproduce it (as minimally and precisely as possible):
create the prometheus thanos v0.32.4 setup with same tsdb path

Full logs to relevant components:
ts=2023-10-16T05:03:54.387051594Z caller=sidecar.go:354 level=warn err="upload 01HCVDP0JJ2SJVSY1GKY7EK54K: hard link block: hard link file chunks/000001: link /prometheus/01HCVDP0JJ2SJVSY1GKY7EK54K/chunks/000001 /prometheus/thanos/upload/01HCVDP0JJ2SJVSY1GKY7EK54K/chunks/000001: operation not permitted" uploaded=0

Anything else we need to know:
prometheus is running root user
I use Docker Compose to up the container, but changing the default user to root using config is blocked within the company policy.

If I want to use Prometheus and Thanos with docker on the same VM, i will need to match each other's users because of owner permission. Is that correct?

1. Change the user of Prometheus to 1001.
2. Change the user of Thanos to nobody or root.

[GiedriusS]

Yes or the Thanos user needs to have +x rights in the same dir where Prometheus writes. +x permission grants the ability to create hardlinks. Hope this helps. Closing this as we won't be able to change the user - this change was made on purpose so that users wouldn't run containers as root.

[khjean]

@GiedriusS Thank you for the answer. Additionally, i can create a hard link right now by adding +x permission, but the chunkfile where the actual hard link is created does not have x permission when created. This is not created by Thanos, but Prometheus. Therefore, +x permission It doesn't seem like it can be a long-term solution. Maybe I'm misunderstanding it?

[shaman007]

This happens to me too. Prometheus works from user 65534 and Thanos works from user 1001. This makes it impossible to create hardlinks since the directory Prometheus creates is 750. It would all be easily fixable, if not in containers. Can we have an option for the Thanos UID?