Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error "No Network Connection" on android 11 while uploading multiple files #370

Open
2 tasks done
pawanrai9999 opened this issue Oct 29, 2021 · 10 comments
Open
2 tasks done
Labels
storage:s3 type:bug Something isn't working
Milestone

Comments

@pawanrai9999
Copy link

Please agree to the following

Summary

I was trying to upload multiple .jpg image to my S3 Vault. Then it gave me error "No Network Connection" but when I try to upload a single file it uploads successfully. I am confirmed there is not issue with network.

System Setup

- Android: 11
- Cryptomator: 1.6.1

Cloud Type

S3

Steps to Reproduce

  1. Unlock a vault
  2. Click on upload file option
  3. Select multiple files and wait

Expected Behavior

Cryptomator should upload file and after uploading the file it should update the available file in folder.

Actual Behavior

Cryptomator will show a toast "No Network Connection".

Reproducibility

Intermittent

Relevant Log Output

D	20211028221002.916	ExceptionHandler	Unexpected error
org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
	at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
	at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
	at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.Flowable.subscribe(Flowable.java:14882)
	at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
	... 22 more
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
	at okio.OutputStreamSink.write(JvmOkio.kt:53)
	at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
	at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
	at okio.ForwardingSink.write(ForwardingSink.kt:29)
	at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
	at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.minio.S3Base.execute(S3Base.java:532)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
	... 24 more

ErrorCode: 10UH:OG5O
D	20211028221002.919	ExceptionHandler	org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
	at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
	at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
	at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.Flowable.subscribe(Flowable.java:14882)
	at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
	... 22 more
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
	at okio.OutputStreamSink.write(JvmOkio.kt:53)
	at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
	at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
	at okio.ForwardingSink.write(ForwardingSink.kt:29)
	at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
	at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.minio.S3Base.execute(S3Base.java:532)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
	... 24 more

ErrorCode: 10UH:OG5O
W	20211028221002.926	Message	No network connection

Anything else?

I am using Backblaze.
Automatic photo upload is active.
Cache size is 100 MB
Keep vaults unlocked while editing files : ON

@pawanrai9999 pawanrai9999 added the type:bug Something isn't working label Oct 29, 2021
@pawanrai9999
Copy link
Author

pawanrai9999 commented Oct 29, 2021

D	20211029091725.286	UploadFilesUseCase	failed 4a158cd9
D	20211029091725.313	ExceptionHandler	Unexpected error
org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
	at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
	at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
	at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.Flowable.subscribe(Flowable.java:14882)
	at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
	... 22 more
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
	at okio.OutputStreamSink.write(JvmOkio.kt:53)
	at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
	at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
	at okio.ForwardingSink.write(ForwardingSink.kt:29)
	at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
	at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.provideOfflineCacheInterceptor$lambda-1(S3ClientFactory.kt:79)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.$r8$lambda$ZlZNFkypEyH2ozaExGLRHlQReU8(Unknown Source:0)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion$$ExternalSyntheticLambda0.intercept(Unknown Source:2)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.minio.S3Base.execute(S3Base.java:532)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
	... 24 more

ErrorCode: 10UH:16N9
D	20211029091725.315	ExceptionHandler	org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
	at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
	at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
	at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.Flowable.subscribe(Flowable.java:14882)
	at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
	... 22 more
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
	at okio.OutputStreamSink.write(JvmOkio.kt:53)
	at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
	at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
	at okio.ForwardingSink.write(ForwardingSink.kt:29)
	at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
	at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
	at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.provideOfflineCacheInterceptor$lambda-1(S3ClientFactory.kt:79)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.$r8$lambda$ZlZNFkypEyH2ozaExGLRHlQReU8(Unknown Source:0)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion$$ExternalSyntheticLambda0.intercept(Unknown Source:2)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.minio.S3Base.execute(S3Base.java:532)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
	... 24 more

ErrorCode: 10UH:16N9
W	20211029091725.328	Message	No network connection
D	20211029091725.363	AssertUnlockedUseCase	started 4a158cf4
D	20211029091725.364	AssertUnlockedUseCase	finished 4a158cf4

@pawanrai9999
Copy link
Author

After digging some more I found this is happening with file sizes larger than 200 MB.

@SailReal
Copy link
Member

SailReal commented Oct 29, 2021

Can you please enable debug mode and reproduce the problem? Then we also see each request and response, this should bring some light into this.

Is it possible that you set e.g. an upload limit to files?

Can you please try to upload a file of this size using a different S3 client. Does the problem also occur with it?

I just tested it myself and unfortunately can not reproduce it.

@SailReal SailReal added state:awaiting-response We need further input from the issue author storage:s3 labels Oct 29, 2021
@no-response no-response bot removed the state:awaiting-response We need further input from the issue author label Oct 29, 2021
@SailReal SailReal added the state:awaiting-response We need further input from the issue author label Oct 29, 2021
@pawanrai9999
Copy link
Author

pawanrai9999 commented Oct 29, 2021

1.txt
0.txt

  1. Above are the links to my log file.
  2. There is no upload limit set on files. I tried smaller files and they are uploaded anything less than 200 MB.
  3. Yes I was successfully able to upload files of much larger size from different clients (BucketAnywhere for S3)

P.S. Error is in 1.txt at timestamp 20211029192152.373

@no-response no-response bot removed the state:awaiting-response We need further input from the issue author label Oct 29, 2021
@SailReal
Copy link
Member

SailReal commented Oct 29, 2021

Hmm there are two pretty strange response types in the log file. There is a possibility that they may or may not be the same or similar problems.

One problem is that it looks like the server is under heavy load and responds then with a 503 (ServiceUnavailable) HTTP status code while uploading a chunk. While recovery we delete the file again which works but we could also handle those 503 status codes using a exponential backoff algorithm and retry those requests and only give up when reaching a retry counter limit.

D	20211029100110.982	OkHttp	--> POST https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploads= http/1.1 (0-byte body)
D	20211029100110.982	OkHttp	Content-Type: application/octet-stream
V	20211029100110.982	Progress	UPLOAD 11%
D	20211029100110.982	OkHttp	Content-Length: 0
D	20211029100110.982	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029100110.982	OkHttp	Accept-Encoding: identity
D	20211029100110.983	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029100110.983	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029100110.983	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029100110.983	OkHttp	x-amz-date: 20211029T043110Z
D	20211029100110.983	OkHttp	--> END POST
D	20211029100111.326	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploads= (343ms)
D	20211029100111.327	OkHttp	x-amz-request-id: 884b41516d2721ac
D	20211029100111.328	OkHttp	x-amz-id-2: aOJJibTcgMEg3jWELZak5eWSfNHg3jzmY
D	20211029100111.328	OkHttp	x-amz-version-id: 4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029
D	20211029100111.328	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029100111.328	OkHttp	Date: Fri, 29 Oct 2021 04:31:11 GMT
D	20211029100111.328	OkHttp	Keep-Alive: timeout=5
D	20211029100111.328	OkHttp	Connection: keep-alive
D	20211029100111.328	OkHttp	<-- END HTTP
D	20211029100111.341	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1 http/1.1 (5242880-byte body)
D	20211029100111.342	OkHttp	Content-Type: application/octet-stream
D	20211029100111.342	OkHttp	Content-Length: 5242880
D	20211029100111.342	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029100111.342	OkHttp	Accept-Encoding: identity
D	20211029100111.342	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029100111.342	OkHttp	Content-MD5: 8YloTdnfiR5kF45tHoUKLQ==
D	20211029100111.342	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029100111.342	OkHttp	x-amz-date: 20211029T043111Z
D	20211029100111.343	OkHttp	--> END PUT
D	20211029100114.436	OkHttp	<-- 503  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1 (3093ms)
D	20211029100114.438	OkHttp	x-amz-request-id: fd487dd29df7b560
D	20211029100114.439	OkHttp	x-amz-id-2: aOHFiRjdzMNM3nGGqZck5I2S+NNc3oTkf
D	20211029100114.439	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029100114.439	OkHttp	Date: Fri, 29 Oct 2021 04:31:12 GMT
D	20211029100114.439	OkHttp	Connection: close
D	20211029100114.439	OkHttp	<-- END HTTP
D	20211029100114.463	OkHttp	--> DELETE https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029 http/1.1 (unknown length)
D	20211029100114.465	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029100114.465	OkHttp	Accept-Encoding: identity
D	20211029100114.465	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029100114.465	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029100114.465	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029100114.466	OkHttp	x-amz-date: 20211029T043114Z
D	20211029100114.466	OkHttp	--> END DELETE
D	20211029100116.296	OkHttp	<-- 204  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029 (1830ms)
D	20211029100116.298	OkHttp	x-amz-request-id: 365d72af6a5c723b
D	20211029100116.299	OkHttp	x-amz-id-2: aOOtiLzfzMGM34WGbZeg5ZmQUNB83kjnZ
D	20211029100116.299	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029100116.299	OkHttp	Date: Fri, 29 Oct 2021 04:31:16 GMT
D	20211029100116.299	OkHttp	Keep-Alive: timeout=5
D	20211029100116.299	OkHttp	Connection: keep-alive
D	20211029100116.300	OkHttp	<-- END HTTP
D	20211029100116.335	UploadFilesUseCase	failed 4a505e27
D	20211029100116.363	ExceptionHandler	Unexpected error
org.cryptomator.domain.exception.FatalBackendException: error occurred
ErrorResponse(code = ServiceUnavailable, message = no tomes available, bucketName = null, objectName = null, resource = null, requestId = null, hostId = null)
request={method=PUT, url=https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1, headers=Host: s3.us-west-000.backblazeb2.com
Accept-Encoding: identity
User-Agent: MinIO (Linux; aarch64) minio-java/dev
Content-MD5: 8YloTdnfiR5kF45tHoUKLQ==
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20211029T043111Z
Authorization: ██
Cache-Control: max-age=0
}
response={code=503, headers=x-amz-request-id: fd487dd29df7b560
x-amz-id-2: aOHFiRjdzMNM3nGGqZck5I2S+NNc3oTkf
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Content-Length: 151
Date: Fri, 29 Oct 2021 04:31:12 GMT
Connection: close
}

	at io.minio.S3Base.execute(S3Base.java:670)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	... 25 more

ErrorCode: 2B6R:5GJ3

The other problem is that sometimes the server completely terminates the connection, I might be wrong but probably for the same reason (heavy load). Perhaps the exponential backoff for the 503 status codes is already enough for the server to no longer disconnect.

D	20211029192138.138	OkHttp	--> HEAD https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r http/1.1 (unknown length)
D	20211029192138.139	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192138.139	OkHttp	Accept-Encoding: identity
D	20211029192138.139	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192138.140	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029192138.141	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192138.141	OkHttp	x-amz-date: 20211029T135138Z
D	20211029192138.141	OkHttp	--> END HEAD
D	20211029192138.501	OkHttp	<-- 404  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r (359ms)
D	20211029192138.502	OkHttp	x-amz-request-id: c436ebaa73234416
D	20211029192138.503	OkHttp	x-amz-id-2: aODJiajcaMCg36mHsZcs5iWQ1NAw32znD
D	20211029192138.504	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192138.505	OkHttp	Date: Fri, 29 Oct 2021 13:51:38 GMT
D	20211029192138.505	OkHttp	Keep-Alive: timeout=5
D	20211029192138.505	OkHttp	Connection: keep-alive
D	20211029192138.505	OkHttp	<-- END HTTP
D	20211029192138.690	OkHttp	--> POST https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploads= http/1.1 (0-byte body)
D	20211029192138.690	OkHttp	Content-Type: application/octet-stream
D	20211029192138.690	OkHttp	Content-Length: 0
D	20211029192138.690	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192138.690	OkHttp	Accept-Encoding: identity
D	20211029192138.690	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192138.690	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029192138.690	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192138.690	OkHttp	x-amz-date: 20211029T135138Z
D	20211029192138.691	OkHttp	--> END POST
D	20211029192139.057	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploads= (366ms)
D	20211029192139.058	OkHttp	x-amz-request-id: ca8ea16fe380bca1
D	20211029192139.059	OkHttp	x-amz-id-2: aOD5ijDckMEc3x2FMZSE5Q2SUNGA3MzkO
D	20211029192139.059	OkHttp	x-amz-version-id: 4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001
D	20211029192139.059	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192139.059	OkHttp	Date: Fri, 29 Oct 2021 13:51:38 GMT
D	20211029192139.059	OkHttp	Keep-Alive: timeout=5
D	20211029192139.059	OkHttp	Connection: keep-alive
D	20211029192139.059	OkHttp	<-- END HTTP
D	20211029192139.073	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=1 http/1.1 (5242880-byte body)
D	20211029192139.074	OkHttp	Content-Type: application/octet-stream
D	20211029192139.074	OkHttp	Content-Length: 5242880
D	20211029192139.074	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192139.074	OkHttp	Accept-Encoding: identity
D	20211029192139.074	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192139.075	OkHttp	Content-MD5: fq1azAcy9MADGOuVPeQ28g==
D	20211029192139.075	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192139.075	OkHttp	x-amz-date: 20211029T135139Z
D	20211029192139.075	OkHttp	--> END PUT
D	20211029192142.404	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=1 (3328ms)
D	20211029192142.405	OkHttp	x-amz-request-id: d3313547033b755a
D	20211029192142.406	OkHttp	x-amz-id-2: aOMhiUzfyMJA3OWHwZf05I2QRNEs3czkj
D	20211029192142.406	OkHttp	ETag: "7ead5acc0732f4c00318eb953de436f2"
D	20211029192142.407	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192142.407	OkHttp	Date: Fri, 29 Oct 2021 13:51:42 GMT
D	20211029192142.407	OkHttp	Keep-Alive: timeout=5
D	20211029192142.408	OkHttp	Connection: keep-alive
D	20211029192142.408	OkHttp	<-- END HTTP
D	20211029192142.551	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=2 http/1.1 (5242880-byte body)
D	20211029192142.551	OkHttp	Content-Type: application/octet-stream
D	20211029192142.551	OkHttp	Content-Length: 5242880
D	20211029192142.551	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192142.551	OkHttp	Accept-Encoding: identity
D	20211029192142.551	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192142.552	OkHttp	Content-MD5: 75SEsyqJUSyZAPkO+laAsQ==
D	20211029192142.552	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192142.552	OkHttp	x-amz-date: 20211029T135142Z
D	20211029192142.552	OkHttp	--> END PUT
D	20211029192145.270	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=2 (2717ms)
D	20211029192145.272	OkHttp	x-amz-request-id: 7eae625a095d2537
D	20211029192145.272	OkHttp	x-amz-id-2: aOFZiAzdNMFA3dGHwZWk5aWQvNFM3wjnm
D	20211029192145.272	OkHttp	ETag: "ef9484b32a89512c9900f90efa5680b1"
D	20211029192145.273	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192145.273	OkHttp	Date: Fri, 29 Oct 2021 13:51:44 GMT
D	20211029192145.273	OkHttp	Keep-Alive: timeout=5
D	20211029192145.273	OkHttp	Connection: keep-alive
D	20211029192145.274	OkHttp	<-- END HTTP
D	20211029192145.413	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=3 http/1.1 (5242880-byte body)
D	20211029192145.413	OkHttp	Content-Type: application/octet-stream
D	20211029192145.413	OkHttp	Content-Length: 5242880
D	20211029192145.413	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192145.413	OkHttp	Accept-Encoding: identity
D	20211029192145.413	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192145.414	OkHttp	Content-MD5: bmkA7uZvXTiyUMYLqtuEuA==
D	20211029192145.414	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192145.414	OkHttp	x-amz-date: 20211029T135145Z
D	20211029192145.414	OkHttp	--> END PUT
D	20211029192148.650	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=3 (3235ms)
D	20211029192148.651	OkHttp	x-amz-request-id: 2268e2fd86dddbd7
D	20211029192148.652	OkHttp	x-amz-id-2: aOOZiDjcyMNg3FWFoZeQ5eWRTNGg3QTmE
D	20211029192148.653	OkHttp	ETag: "6e6900eee66f5d38b250c60baadb84b8"
D	20211029192148.653	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192148.654	OkHttp	Date: Fri, 29 Oct 2021 13:51:47 GMT
D	20211029192148.654	OkHttp	Keep-Alive: timeout=5
D	20211029192148.654	OkHttp	Connection: keep-alive
D	20211029192148.654	OkHttp	<-- END HTTP
D	20211029192148.760	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=4 http/1.1 (5242880-byte body)
D	20211029192148.760	OkHttp	Content-Type: application/octet-stream
D	20211029192148.761	OkHttp	Content-Length: 5242880
D	20211029192148.761	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192148.761	OkHttp	Accept-Encoding: identity
D	20211029192148.761	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192148.761	OkHttp	Content-MD5: BgOtBD9qdhqzsy91HhksyA==
D	20211029192148.761	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192148.761	OkHttp	x-amz-date: 20211029T135148Z
D	20211029192148.761	OkHttp	--> END PUT
D	20211029192151.109	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=4 (2348ms)
D	20211029192151.111	OkHttp	x-amz-request-id: ad2d614a1c2b8ce2
D	20211029192151.112	OkHttp	x-amz-id-2: aOG1i7TeNMEs3jWHRZUM5wmT1NJs3WTmc
D	20211029192151.112	OkHttp	ETag: "0603ad043f6a761ab3b32f751e192cc8"
D	20211029192151.112	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192151.112	OkHttp	Date: Fri, 29 Oct 2021 13:51:50 GMT
D	20211029192151.113	OkHttp	Keep-Alive: timeout=5
D	20211029192151.113	OkHttp	Connection: keep-alive
D	20211029192151.113	OkHttp	<-- END HTTP
D	20211029192151.258	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=5 http/1.1 (5242880-byte body)
D	20211029192151.259	OkHttp	Content-Type: application/octet-stream
D	20211029192151.259	OkHttp	Content-Length: 5242880
D	20211029192151.259	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192151.259	OkHttp	Accept-Encoding: identity
D	20211029192151.259	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192151.259	OkHttp	Content-MD5: 92J8ozZqevkBmadcXQ1YwA==
D	20211029192151.259	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192151.259	OkHttp	x-amz-date: 20211029T135151Z
D	20211029192151.259	OkHttp	--> END PUT
D	20211029192152.373	OkHttp	<-- HTTP FAILED: java.net.SocketException: Connection reset
D	20211029192152.386	OkHttp	--> DELETE https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001 http/1.1 (unknown length)
D	20211029192152.387	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192152.388	OkHttp	Accept-Encoding: identity
D	20211029192152.389	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192152.389	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029192152.389	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192152.389	OkHttp	x-amz-date: 20211029T135152Z
D	20211029192152.389	OkHttp	--> END DELETE
D	20211029192154.795	OkHttp	<-- 204  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001 (2405ms)
D	20211029192154.797	OkHttp	x-amz-request-id: dfd418d42dc54fa5
D	20211029192154.798	OkHttp	x-amz-id-2: aOJFiIjfcMCY3f2HwZRU5jGQ+NB83BDlA
D	20211029192154.798	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192154.798	OkHttp	Date: Fri, 29 Oct 2021 13:51:55 GMT
D	20211029192154.799	OkHttp	Keep-Alive: timeout=5
D	20211029192154.799	OkHttp	Connection: keep-alive
D	20211029192154.799	OkHttp	<-- END HTTP
D	20211029192154.919	UploadFilesUseCase	failed 4c519f94
D	20211029192154.950	ExceptionHandler	Unexpected error
org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
	at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
	at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
	at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.Flowable.subscribe(Flowable.java:14882)
	at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
	... 22 more
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:215)
	at java.net.SocketInputStream.read(SocketInputStream.java:144)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readFromSocket(ConscryptEngineSocket.java:936)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:900)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readUntilDataAvailable(ConscryptEngineSocket.java:815)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.read(ConscryptEngineSocket.java:788)
	at okio.InputStreamSource.read(JvmOkio.kt:90)
	at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:427)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:320)
	at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
	at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178)
	at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.provideOfflineCacheInterceptor$lambda-1(S3ClientFactory.kt:79)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.$r8$lambda$ZlZNFkypEyH2ozaExGLRHlQReU8(Unknown Source:0)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion$$ExternalSyntheticLambda0.intercept(Unknown Source:2)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.minio.S3Base.execute(S3Base.java:532)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
	... 24 more

ErrorCode: 10UH:3SG0
W	20211029192154.965	Message	No network connection

@pawanrai9999
Copy link
Author

pawanrai9999 commented Oct 30, 2021

While I was reading the documentation of Backblaze I found this:-

Some errors returned from b2 upload file mean that you should get a new upload URL and try again, while others mean that there is a problem with your request and trying again will not help.

These indicate that you should get a new upload URL and try again:

  • Unable to make an HTTP connection, including connection timeout.
  • Status of 401 Unauthorized, and an error code of expired_auth_token
  • Status of 408 Request Timeout
  • Any HTTP status in the 5xx range, including 503 Service Unavailable
  • "Broken pipe" sending the contents of the file.
  • A timeout waiting for a response (socket timeout).

The last one, "broken pipe", happens when you are sending a file big enough that the buffers in the HTTP connection won't hold it. HTTP client libraries send the entire request before looking for a response, and if the B2 server has already replied with an error, you'll be unable to send the entire file and will get a "broken pipe" error.

@SailReal
Copy link
Member

SailReal commented Nov 3, 2021

Thanks for your amazing research regarding this topic.
Can you please also post the source for this because we need to create an issue in the minio-java repository as we use this as the S3 client and it would be great if this error handling would be implemented directly in it.

@pawanrai9999
Copy link
Author

@SailReal
Copy link
Member

SailReal commented Nov 9, 2021

Thanks for the source, just created this issue minio/minio-java#1255

@stale

This comment was marked as outdated.

@stale stale bot added state:stale Issues without any activity that will be closed automatically and removed state:stale Issues without any activity that will be closed automatically labels Apr 18, 2022
@SailReal SailReal added this to the Backlog milestone Apr 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
storage:s3 type:bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants