Too many requests

[EvertonSA]

When running airflow DatabricksRunNowOperator, if using managed identity, and if there is multiple tasks running in parallel, the code below can quickly exhaust the allowed metadata server as explained here https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=windows#rate-limiting

In general, requests to IMDS are limited to 5 requests per second (on a per VM basis). Requests exceeding this threshold will be rejected with 429 responses.

airflow/providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py

Line 568 in 4b83391

self._check_azure_metadata_service()

I don't understand why this check was put here 3 years ago, but I feel this is unnecessary.

I propose we remove this check, or make it optional. Can I open a PR for this?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[DjVinnii]

It looks like this check is done to validate if the metadata service is available before actually issuing a token. In my opinion this a good thing to validate, however there might be a better way to check this and not get rate limited. Otherwise a retry mechanism (with exponential backoff) could be useful, especially when kicking of a higher number of tasks.

[EvertonSA]

there is no need to check for the metadata to be available, if its not available it will fail anyway

[EvertonSA]

Out of all integrations available for airflow and Microsoft products this is the only one that checks the metadata availability before issuing the token. The code right now hammers the metadata api even if the token is valid.

[rxxj25]

HI,can I work on this?

[hakunamatata-sb]

@rxxj25 are you working on this ?, would also like to contribute..
@EvertonSA if noone is working on this can you assign this to me ?

[EvertonSA]

hi @hakunamatata-sb , i cant assign issues, please feel free to work on it. the draft is here #47956

[xchwan]

None work on it, so I make a PR to handle it. I think remove the check function is not a good way. So, I add exponential back-off. Once it get 429 it will delay and retry