Unhealthy ingester is not being removed from the Ring

[alexinfoblox]

After upgrading Loki to version 3.1.1, we began experiencing an issue where, when the ingester is scaled down by the HPA, the pod is terminated but ingester does not leave the ring. Instead, it remains stuck in an "Unhealthy" state until manually removed.

This issue was not present in the previous version we used (2.9.6).

We didn’t change anything in the Loki configuration when upgrading to the new version.

I could find only this suspicious log:

level=error ts=2024-11-07T18:48:57.602540307Z caller=recalculate_owned_streams.go:55 msg="failed to get token ranges for ingester" err="zone not set"

We haven't seen this error message before in the previous release.