Azure MSI Rate Limit

[phillebaba]

Thanos, Prometheus and Golang version used:
v0.22.0

Object Storage Provider:
Azure Storage Account

What happened:
After running for a couple of minutes the compactor will crash due to API errors from the Azure API. This seems to be due to the fact that a new Service Principal token is created for every request, causing a rate limit to be hit. Running the same configuration but with static keys will not invoke the rate limit.

What you expected to happen:
Compactor should run as normal without any rate limits.

How to reproduce it (as minimally and precisely as possible):
Run compactor against Azure Storage Account with MSI authentication.

Full logs to relevant components:

level=error ts=2021-08-26T16:33:51.380291819Z caller=compact.go:488 msg="retriable error" err="compaction: sync: filter metas: filter blocks marked for deletion: get file: 01FE1EJAE19DA6199RV98EXSHJ/deletion-mark.json: cannot get blob reader: 01FE1EJAE19DA6199RV98EXSHJ/deletion-mark.json: cannot get Azure blob URL, address: 01FE1EJAE19DA6199RV98EXSHJ/deletion-mark.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net"
level=error ts=2021-08-26T16:33:51.761429073Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="incomplete view: 11 errors: meta.json file exists: 01FDRS4M15H8WATMJJ5KYEE8EC/meta.json: cannot get Azure blob URL, address: 01FDRS4M15H8WATMJJ5KYEE8EC/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request
failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDT81FNZ8AZQWDES0C53KAN3/meta.json: cannot get Azure blob URL, address: 01FDT81FNZ8AZQWDES0C53KAN3/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDKM640W3X832QH11B8G1AFS/meta.json: cannot get Azure blob URL, address: 01FDKM640W3X832QH11B8G1AFS/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDKCQDYHZ5T3NWYBYA6Q70CN/meta.json: cannot get Azure blob URL, address: 01FDKCQDYHZ5T3NWYBYA6Q70CN/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDW3Y0PP8C5DA203B10WD13A/meta.json: cannot get Azure blob URL, address: 01FDW3Y0PP8C5DA203B10WD13A/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDT85966XFZWAMVA94BV4DF2/meta.json: cannot get Azure blob URL, address: 01FDT85966XFZWAMVA94BV4DF2/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDEFDDATDH7RR3DSNAW5MCQG/meta.json: cannot get Azure blob URL, address: 01FDEFDDATDH7RR3DSNAW5MCQG/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDKMX900EP9V3K5SXXF9MC4Z/meta.json: cannot get Azure blob URL, address: 01FDKMX900EP9V3K5SXXF9MC4Z/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDRVCYRJ0RTV4SM830PQM55A/meta.json: cannot get Azure blob URL, address: 01FDRVCYRJ0RTV4SM830PQM55A/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDSKAJBSFAXQG50RZJK8A7ZK/meta.json: cannot get Azure blob URL, address: 01FDSKAJBSFAXQG50RZJK8A7ZK/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net; meta.json file exists: 01FDT7XMH6W9DFR6SSY2F0XNMS/meta.json: cannot get Azure blob URL, address: 01FDT7XMH6W9DFR6SSY2F0XNMS/meta.json: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Refresh request failed. Status Code = '429'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Temporarily throttled, too many requests\"}\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2F<redacted>.blob.core.windows.net"

Anything else we need to know:

Relevant code which may be the cause of the bug. The solution could be to create a single credential object which is used through the lifetime of the application.

thanos/pkg/objstore/azure/helpers.go

Lines 37 to 62 in 9355a82

	func getAzureStorageCredentials(conf Config) (blob.Credential, error) {
	if conf.MSIResource != "" {
	msiConfig := auth.NewMSIConfig()
	msiConfig.Resource = conf.MSIResource
	azureServicePrincipalToken, err := msiConfig.ServicePrincipalToken()
	if err != nil {
	return nil, err
	}
	// Get a new token.
	err = azureServicePrincipalToken.Refresh()
	if err != nil {
	return nil, err
	}
	token := azureServicePrincipalToken.Token()
	return blob.NewTokenCredential(token.AccessToken, nil), nil
	}
	credential, err := blob.NewSharedKeyCredential(conf.StorageAccountName, conf.StorageAccountKey)
	if err != nil {
	return nil, err
	}
	return credential, nil
	}

[wiardvanrij]

Thanks, let's try to smash this bug!