store: Thanos store does not support reloading of configuration

[pjastrzabek]

I'm rendering s3 bucket config using vault injector and after refreshing AWS credentials (after expiration) I'd like to 'notify' / trigger reload of thanos store.
I tried to send some signals to store like -HUP, I see in logs that they are being noticed, but credentials used for accessing bucket (in AWS in my case) are still old ones.

I found similar issue for ruler component
#4432

I ended up restarting thanos with -9

pkill -9 thanos

which is causing small downtime, but works

I tried to use /-/reload endpoint but store component does not seem to expose it

Object Storage Provider: AWS

What happened:

What you expected to happen:
Thanos store should have a way to trigger reload of bucket config without downtime.

How to reproduce it (as minimally and precisely as possible):

1. Start thanos store with reference to config file with short living credentials inside
2. When AWS credentials expire component will start complaining
3. execute pkill -1 thanos
4. Component should reload config with new credentials

[yeya24]

Reloading object store credentials is not supported right now. But it would be good to make it hot reloadable.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[pjastrzabek]

I was hoping that maybe that problem will be fixed as a result of adding aws-sdk ( #4667 ) but unfortunately it isn't (tested on 0.26.0)
Issue is still valid. There is no way of using thanos store with temporary credentials without downtime, because once credentials are expired component needs to be restarted to use newly rendered credentials.

[pjastrzabek-roche]

Still valid.

After updating thanos (to version that contains AWS SDK ) usage of temporary credentials is possible, we render credentials with vault injector inside /home/.aws/credentials

But after credentials are refreshed thanos component still has to be restarted, cause it does not reload them.

[pjastrzabek-roche]

I'd like to propose to reopen this bucket.

In certain places company policies force us to use only temporary credentials.
When AWS S3 bucket is used as a thanos store it means that every 12h (that is the longest validity of AWS STS token) we need to restart every thanos component that talks to AWS S3 bucket.

It's potentially a cause for brief unavailability and is very heavy operation.
I believe it has to trigger issues not only for us.

[edgrz]

+1 to it.

In our case it's also causing constant unwanted restarts.

[spoofedpacket]

Another +1

Would be great to see a graceful reload endpoint implemented. Or, even better, a mechanism to watch the credentials file for changes and reload as required.

[AronllStone]

+1

It would be great to have a soft reload when mTLS certificates have been updated (e.g., by using a sidecar container or when the secret has been modified, etc).