Blob Storage SDK hits timeout issue periodically, no indication as to why

[Petermarcu]

Reported by @Garfinkel. @Garfinkel if you could share more details of the script you used, that would be helpful.

I ran a Python script which uploads a ~30-50mb GZIP file to Azure Blob Storage through the Python SDK (v 12.3.1) using Python 3.7.5 and 3.7.7. The error I receive indicates a timeout issue with no indication why other than "server response error".

We should better indicate when issues are transient / potentially an Azure issue as opposed to a user issue.

Region = us east

... 1_CallAPI.venv\lib\site-packages\azure\core\pipeline\transport_requests_basic.py", line 284, in send

raise error

azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[xiafu-msft]

Hi @Garfinkel

It could because of the max_single_put_size, sometime the first put blob request cannot finish within the specified/default timeout value, set max_single_put_size to a smaller value like 20*1024*1024 could be a workaround to fix the problem. We will investigate offline and see how we should modify the SDK.

[Petermarcu]

Note I filed this on behalf of @Garfinkel

[xiafu-msft]

Hi @Garfinkel

Is you problem resolved?

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[erfannariman]

We have this same issue, weird part is that it only happens on certain occasions and on certain laptops.

[mgopez]

Getting the same issue when we have max_concurrency set within the upload_blob() method.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[PeterZechel]

Hi, I have the same bug.

Versions:
Python 3.6.9
azure-storage-blob 12.6.0

File:
Type: json
Size: 50508312

Upload parameters:
max_single_put_size is set to 20*1024*1024
timeout is set to 600

Error:
azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out',))

Within this function multiple files are uploaded. If I change the order of the upload, it works until it comes to the large file.

[amirbnnsr]

I am also getting a similar issue. I did notice that this issue seem to occur for certain file sizes and upload speeds, though I am not saying that they are definitely correlated.

For example, when uploading a 48MB mkv video file with around 8400Kbps upload speed leads me to this issue's error:
azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))

As @xiafu-msft has suggested, changing max_single_put_size to a lower number, in my case being 10*1024*1024, did help for this particular speed and file size, but not for slower speeds. For example, uploading a 37MB mkv video with 1000Kbps upload speed. Decreasing the value of the max_single_put_size parameter further, even down to 500KB, did not help in this case.

Directly uploading via an http request (requests package) using a SAS blob url with similar header parameters as the one used within BlobClient's request {"x-ms-blob-type": "BlockBlob","x-ms-version": "2020-02-10","Content-Type": "application/octet-stream"} works without an issue for all uploads.

Versions:
Python 3.8.0
azure-storage-blob 12.6.0

Error:
azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out',))

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[eladg]

Same issue here: azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))

I'm dealing with big video files. For small files, upload works well, but I'm getting this timeout for bigger files (anything above ~15MB).

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[xiafu-msft]

Hi @amirbnnsr @hellothisisMatt @PeterZechel @eladg

We tried hard to get a write operation timeout error (we tried account in a far region, and make the max_single_put_size=60010241024, also tried to set max_single_put_size to a small value and use max_concurrency=60 to upload a big blob) while we were not able to get the error...
Can any of you try to turn on logging by copy paste this block of code https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/storage/azure-storage-blob/README.md#logging
After you turn on logging, you will get x-ms-request-id and we can search service log based on that. Sorry about the inconvenience and really appreciated for your help!

[eladg]

In my case, creating BlobServiceClient with this setting helped:

{
      "max_single_put_size": 4*1024*1024, # split to 4MB chunks`
      "max_single_get_size": 4*1024*1024, # split to 4MB chunks
 }

@xiafu-msft , will try to provide logs if/when I see this issue again

[amirbnnsr]

@xiafu-msft Attached are two log files with the request header information. I have omitted Request URL and Request body from the logs.

upload_bug_logs_1000Kbps_15MB.txt are logs from uploading a 15MB video file with 1000Kbps upload speed.
upload_bug_logs_8000Kbps_52MB.txtare logs from uploading a 52MB video file with 8200Kbps upload speed.

The default value for the max_single_put_size parameter was used.

Thank you for looking into this.

upload_bug_logs_1000Kbps_15MB.txt
upload_bug_logs_8200Kbps_52MB.txt

[xiafu-msft]

Hi @amirbnnsr
Thanks so much for the log! Could you also paste the response there, we need the response info to look up on our end

[amirbnnsr]

@xiafu-msft Unfortunately, there are no response because the timeout error happens when sending the request. The logs that I posted are actually everything that came out of azure.storage.blob

[xiafu-msft]

Hi @amirbnnsr

can you please send me your account name? Thanks!

Also have you ever tried to set read_timeout on BlobServiceClient? Though I think that’s not the problem.

[amirbnnsr]

@xiafu-msft The account name is: aiirtest

I have not tried the read_timeout, I will give that a go.

[gitjit]

I am also getting the timeout error while uploading files > 20 MB. It succeeds for smaller files. I am using Python3.6 and using the Azure Storage SDK (azure-storage-blob-12.7.1. Here is my stack overflow link. Just following the MSDN sample is showing this problem.

https://stackoverflow.com/questions/66394270/how-to-upload-large-file-100mb-to-azure-blob-storage-using-python-sdk

[xiafu-msft]

In the most recent patch release (azure-storage-blob 12.8.1) we had a fix about upload from a generator (#17418), hopefully that can mitigate this problem.
close this issue for now, if anyone meet this problem again feel free to comment on this issue.

[setinam]

Hi, I have a ticket (TrackingID#2111100050002885) open with Storage support for about half a year and we just came to some sort of data that my help this.
We have a solution that uses parallel processes to upload files to Blob storage. randomly when we push a lot of data to the blob we get a similar write timeouts. We set timeout value on upload_blob to 600, but it did not help 100%, it probably mitigated it.
On the last exchange we came to a conclusion that E2E latency was 20s on time of timeout. This is a lot less then 600 s.
We did not yet change max_single_put_size to try to find a sweet spot for it, but we use different sizes of files so adjusting it could not be that easy.
On the Storage side there were internal server errors Network timeout error for our client azsdk-python-storage-blob/12.9.0 Python.
If server tries to return Internal server error and can not get it to the client I suppose client throws this write timeout errors.