Storage insufficient data written #1018
Comments
|
Hi @fonzy2013, thanks for the report. Could you please share the full stacktrace of the exception? Also, sharing one sample input streams that causes the exception would really help us figuring out what's going wrong. |
|
Hi @mziccard , |
|
@fonzy2013 I doubt this problem is related to concurrent uploads. I have been doing some research and it seems that this is a recurrent issue, not necessarily related to
However, I have been uploading different size files since this morning (using your code) and I couldn't manage to reproduce the error. Are these errors recurring on the same file/file size or completely spurious? Information on the size (or size range) of files you are trying to upload might help as well. |
|
I get this exception every few hours over the past 24 hours, while I have not seen it before. Hundreds of files load fine, and then all of a sudden this exception happens. When I restart my program, the same file that threw this exception loads with no problems. Here is the trace: |
|
I am trying to get to the bottom of this but I haven't been able to reproduce the error yet. Where are your applications running? Do you have any change of trying out the changes in this branch and let me know if you still get those errors? |
|
I process ~500,000 files using a mix of Mac OS and Windows 10 PCs. I saw error this error on all of them. The error appears to be random - sometimes hundreds, sometimes thousands of files upload without problems before this exception occurs. This may explain why you cannot reproduce this error when trying with just a few files. I attached a couple of files that failed to upload. Both files succeeded on a second attempt, though. I have switched from |
That might work (it's in line with the workarounds of this issue I found around the internet) but it uses direct upload. Writer should be favored for large files. Thanks for sharing the files, I'll create a more "stressing" test. Any change you can build an artifact from the code in this repository and run your application with that? |
|
Any chance someone looked further into this? I have large files that needs to be uploaded so uploading them using storage.create(BlobInfo, byte[]) doesn't work. |
|
Hi @fonzy2013, it would be really helpful if you could build gcloud-java from the code at https://github.com/mziccard/gcloud-java/tree/storage-insufficient-data and give it a try. If you need me to I can rebase that code on top of master (to catch latest changes). |
|
@mziccard, I gave it a try and I can't see that exception now. However I will report 2 other ones in another thread that I have seen lately (SocketException: Operation timed out and SocketException: Socket is closed). I saw this fix was to implement a retry, do you know how the underlying problem can be adressed? |
I already reported here all information that I have on this issue. This is not an issue with
Out of curiosity, where is you application running? You should be able to mitigate these exceptions by setting more permissive read and connect timeouts (by using |
|
Our application is running on our own servers. No specific jvm options |
|
In #1187 we added this exception to our retry logic. This should mitigate the issue. I am closing this but feel free to reopen if you still experience the exception. |
…orting to v0.126.0-beta (#1018) * chore(deps): update dependency com.google.cloud:google-cloud-errorreporting to v0.126.0-beta * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Could someone help with pointing me to the right direction wrt to an exception I get (sometimes): com.google.cloud.storage.StorageException: insufficient data written.
The code that uploads and fails sometimes is:
The text was updated successfully, but these errors were encountered: