High Latency from AKS to Azure Storage

[dewanymca]

HI Team

We have been using AKS workload to connect to Azure storage, we are experiencing latency in call to Azure Storage, It goes into 5-7 seconds, we checked with storage team and they confirmed there is no latency on storage side and calls are getting completed in msec.

Also, we are running same code from Azure App service and there we are not experiencing any latency to same storage. can you please help us in identifying from where this latency is coming.

Attaching latency graph from storage:

Thanks
Deepak

[andyzhangx]

@dewanymca how did you use azure storage? pod talk to azure storage directly? is it related to DNS resolution?

[dewanymca]

Pod directly talks to storage using storage sdk with Auth token which is being provided by pod identity. We are not sure whether DNs resolution is the issue or not. One thing we can say for sure we have tested same application in aks in non production environment under load but never ran into this high latency.

[GuyPaddock]

This sounds similar to what we are seeing in #1593. In that case, just unzipping a large ZIP file on a share with no other traffic leads to latency that's so bad that the Azure Files mount becomes unresponsive.

[dewanymca]

@GuyPaddock thanks for pointer, however we are not using volume, app is making a connection to storage directly via SDK so i think #1593 is not applicable in this case.

[andyzhangx]

it's caused by DNS resolution latency as discussed offline.

[github-actions]

Action required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[palma21]

Marking with answer provided. Please reach out if not the case.

[ghost]

Thanks for reaching out. I'm closing this issue as it was marked with "Answer Provided" and it hasn't had activity for 2 days.