Bug: Old Data Unreadable After Migrating Mimir to Kubernetes with Helm and Upgrading to v2.14

[ZqsdYann]

What is the bug?

Issue Description
We encountered an issue after migrating two Mimir instances from ubuntu virtual machines to a distributed Mimir setup using Helm charts on a Kubernetes cluster.

Details

We kept the same S3 bucket to store all logs and metrics during the migration.
The Kubernetes-based Mimir setup can correctly write and read data it creates in the bucket.
However, our problem is that old data (from before the migration) appears to be inaccessible. The data still exists in the S3 bucket but cannot be read by Mimir.

Additional Context

We are aware that sometimes it takes time for old data to be indexed or processed, but the migration was completed three days ago, and we still cannot see any of the old logs.
We upgraded Mimir from version 2.12 to version 2.14 (latest) during the migration.
There are no error logs on any of the Mimir pods.

How to reproduce it?

Set up a Mimir instance (version 2.12) on a virtual machine.
Configure the instance to store logs and metrics in an S3 bucket.
Migrate the setup to a distributed Mimir using Helm charts on a Kubernetes cluster.
Upgrade Mimir from version 2.12 to version 2.14 during the migration.
Ensure that the same S3 bucket is used to store logs and metrics after the migration.
Observe that new data is being written and read correctly from the S3 bucket, but old data (prior to the migration) is not accessible.

What did you think would happen?

After migrating Mimir from version 2.12 to 2.14 and moving to a Kubernetes-based distributed setup, we expected the following:
The S3 bucket would continue to be used for storing logs and metrics
Mimir would be able to both read and write data correctly from the S3 bucket, including old data (prior to the migration)
Both new and old logs and metrics would be accessible without issues

What was your environment?

Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy

mimir installed from : Dec 22 2023 mimir-2.11.0_amd64.deb

Any additional context to share?

No response

[56quarters]

There was no change to how data is stored in object storage in 2.14. This is likely something specific to your environment.

Can you try:

• Make sure the compactor is running. It is responsible for updating bucket indexes which is how queriers and store-gateways discover blocks
• Make sure that bucket-index.json.gz files have both old and new blocks in them
• Look at the logs for store-gateways, they will log when they discover blocks that they "own"

[ZqsdYann]

Hello,
Thank you for your quick response it resolved the issue!

It turned out that the virtual machine installation created a /blocks folder containing tenant-specific data, each with its own bucket-index.json. However, our Kubernetes deployment was using the root directory for the same purpose. This mismatch caused the old data to be in the wrong location.

A s5cmd mv file to the right location and a reload of the compactor fixed everything on our end.

Thanks again for your help!