Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try to recover from HTTP status code 503 and connection reset by not deleting the file but using a exponential backoff retry strategy #1255

Closed
SailReal opened this issue Nov 9, 2021 · 5 comments
Closed

Try to recover from HTTP status code 503 and connection reset by not deleting the file but using a exponential backoff retry strategy #1255

SailReal opened this issue Nov 9, 2021 · 5 comments

Comments

@SailReal
Copy link

SailReal commented Nov 9, 2021

Summary

While uploading some files using minio-java to Backblaze, we encounter 503 (ServiceUnavailable) HTTP status codes as well as connection resets (SocketException: Connection reset) which always results in the file didn't upload successfully.

System Setup

- minio-java version 8.3.3
- backblaze cloud

Steps to Reproduce

Upload some files to Backblaze

Expected Behavior

File is uploaded

Actual Behavior

Sometimes a 503 status code is returned or a connection reset is initiated by the cloud which both aborts the upload.

Reproducibility

Intermittent

Relevant Log Output

HTTP status code 503

D	20211029100110.982	OkHttp	--> POST https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploads= http/1.1 (0-byte body)
D	20211029100110.982	OkHttp	Content-Type: application/octet-stream
V	20211029100110.982	Progress	UPLOAD 11%
D	20211029100110.982	OkHttp	Content-Length: 0
D	20211029100110.982	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029100110.982	OkHttp	Accept-Encoding: identity
D	20211029100110.983	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029100110.983	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029100110.983	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029100110.983	OkHttp	x-amz-date: 20211029T043110Z
D	20211029100110.983	OkHttp	--> END POST
D	20211029100111.326	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploads= (343ms)
D	20211029100111.327	OkHttp	x-amz-request-id: 884b41516d2721ac
D	20211029100111.328	OkHttp	x-amz-id-2: aOJJibTcgMEg3jWELZak5eWSfNHg3jzmY
D	20211029100111.328	OkHttp	x-amz-version-id: 4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029
D	20211029100111.328	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029100111.328	OkHttp	Date: Fri, 29 Oct 2021 04:31:11 GMT
D	20211029100111.328	OkHttp	Keep-Alive: timeout=5
D	20211029100111.328	OkHttp	Connection: keep-alive
D	20211029100111.328	OkHttp	<-- END HTTP
D	20211029100111.341	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1 http/1.1 (5242880-byte body)
D	20211029100111.342	OkHttp	Content-Type: application/octet-stream
D	20211029100111.342	OkHttp	Content-Length: 5242880
D	20211029100111.342	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029100111.342	OkHttp	Accept-Encoding: identity
D	20211029100111.342	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029100111.342	OkHttp	Content-MD5: 8YloTdnfiR5kF45tHoUKLQ==
D	20211029100111.342	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029100111.342	OkHttp	x-amz-date: 20211029T043111Z
D	20211029100111.343	OkHttp	--> END PUT
D	20211029100114.436	OkHttp	<-- 503  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1 (3093ms)
D	20211029100114.438	OkHttp	x-amz-request-id: fd487dd29df7b560
D	20211029100114.439	OkHttp	x-amz-id-2: aOHFiRjdzMNM3nGGqZck5I2S+NNc3oTkf
D	20211029100114.439	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029100114.439	OkHttp	Date: Fri, 29 Oct 2021 04:31:12 GMT
D	20211029100114.439	OkHttp	Connection: close
D	20211029100114.439	OkHttp	<-- END HTTP
D	20211029100114.463	OkHttp	--> DELETE https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029 http/1.1 (unknown length)
D	20211029100114.465	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029100114.465	OkHttp	Accept-Encoding: identity
D	20211029100114.465	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029100114.465	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029100114.465	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029100114.466	OkHttp	x-amz-date: 20211029T043114Z
D	20211029100114.466	OkHttp	--> END DELETE
D	20211029100116.296	OkHttp	<-- 204  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029 (1830ms)
D	20211029100116.298	OkHttp	x-amz-request-id: 365d72af6a5c723b
D	20211029100116.299	OkHttp	x-amz-id-2: aOOtiLzfzMGM34WGbZeg5ZmQUNB83kjnZ
D	20211029100116.299	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029100116.299	OkHttp	Date: Fri, 29 Oct 2021 04:31:16 GMT
D	20211029100116.299	OkHttp	Keep-Alive: timeout=5
D	20211029100116.299	OkHttp	Connection: keep-alive
D	20211029100116.300	OkHttp	<-- END HTTP
D	20211029100116.335	UploadFilesUseCase	failed 4a505e27
D	20211029100116.363	ExceptionHandler	Unexpected error
org.cryptomator.domain.exception.FatalBackendException: error occurred
ErrorResponse(code = ServiceUnavailable, message = no tomes available, bucketName = null, objectName = null, resource = null, requestId = null, hostId = null)
request={method=PUT, url=https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1, headers=Host: s3.us-west-000.backblazeb2.com
Accept-Encoding: identity
User-Agent: MinIO (Linux; aarch64) minio-java/dev
Content-MD5: 8YloTdnfiR5kF45tHoUKLQ==
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20211029T043111Z
Authorization: ██
Cache-Control: max-age=0
}
response={code=503, headers=x-amz-request-id: fd487dd29df7b560
x-amz-id-2: aOHFiRjdzMNM3nGGqZck5I2S+NNc3oTkf
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Content-Length: 151
Date: Fri, 29 Oct 2021 04:31:12 GMT
Connection: close
}

	at io.minio.S3Base.execute(S3Base.java:670)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	... 25 more

Connection reset

D	20211029192138.138	OkHttp	--> HEAD https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r http/1.1 (unknown length)
D	20211029192138.139	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192138.139	OkHttp	Accept-Encoding: identity
D	20211029192138.139	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192138.140	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029192138.141	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192138.141	OkHttp	x-amz-date: 20211029T135138Z
D	20211029192138.141	OkHttp	--> END HEAD
D	20211029192138.501	OkHttp	<-- 404  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r (359ms)
D	20211029192138.502	OkHttp	x-amz-request-id: c436ebaa73234416
D	20211029192138.503	OkHttp	x-amz-id-2: aODJiajcaMCg36mHsZcs5iWQ1NAw32znD
D	20211029192138.504	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192138.505	OkHttp	Date: Fri, 29 Oct 2021 13:51:38 GMT
D	20211029192138.505	OkHttp	Keep-Alive: timeout=5
D	20211029192138.505	OkHttp	Connection: keep-alive
D	20211029192138.505	OkHttp	<-- END HTTP
D	20211029192138.690	OkHttp	--> POST https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploads= http/1.1 (0-byte body)
D	20211029192138.690	OkHttp	Content-Type: application/octet-stream
D	20211029192138.690	OkHttp	Content-Length: 0
D	20211029192138.690	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192138.690	OkHttp	Accept-Encoding: identity
D	20211029192138.690	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192138.690	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029192138.690	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192138.690	OkHttp	x-amz-date: 20211029T135138Z
D	20211029192138.691	OkHttp	--> END POST
D	20211029192139.057	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploads= (366ms)
D	20211029192139.058	OkHttp	x-amz-request-id: ca8ea16fe380bca1
D	20211029192139.059	OkHttp	x-amz-id-2: aOD5ijDckMEc3x2FMZSE5Q2SUNGA3MzkO
D	20211029192139.059	OkHttp	x-amz-version-id: 4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001
D	20211029192139.059	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192139.059	OkHttp	Date: Fri, 29 Oct 2021 13:51:38 GMT
D	20211029192139.059	OkHttp	Keep-Alive: timeout=5
D	20211029192139.059	OkHttp	Connection: keep-alive
D	20211029192139.059	OkHttp	<-- END HTTP
D	20211029192139.073	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=1 http/1.1 (5242880-byte body)
D	20211029192139.074	OkHttp	Content-Type: application/octet-stream
D	20211029192139.074	OkHttp	Content-Length: 5242880
D	20211029192139.074	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192139.074	OkHttp	Accept-Encoding: identity
D	20211029192139.074	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192139.075	OkHttp	Content-MD5: fq1azAcy9MADGOuVPeQ28g==
D	20211029192139.075	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192139.075	OkHttp	x-amz-date: 20211029T135139Z
D	20211029192139.075	OkHttp	--> END PUT
D	20211029192142.404	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=1 (3328ms)
D	20211029192142.405	OkHttp	x-amz-request-id: d3313547033b755a
D	20211029192142.406	OkHttp	x-amz-id-2: aOMhiUzfyMJA3OWHwZf05I2QRNEs3czkj
D	20211029192142.406	OkHttp	ETag: "7ead5acc0732f4c00318eb953de436f2"
D	20211029192142.407	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192142.407	OkHttp	Date: Fri, 29 Oct 2021 13:51:42 GMT
D	20211029192142.407	OkHttp	Keep-Alive: timeout=5
D	20211029192142.408	OkHttp	Connection: keep-alive
D	20211029192142.408	OkHttp	<-- END HTTP
D	20211029192142.551	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=2 http/1.1 (5242880-byte body)
D	20211029192142.551	OkHttp	Content-Type: application/octet-stream
D	20211029192142.551	OkHttp	Content-Length: 5242880
D	20211029192142.551	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192142.551	OkHttp	Accept-Encoding: identity
D	20211029192142.551	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192142.552	OkHttp	Content-MD5: 75SEsyqJUSyZAPkO+laAsQ==
D	20211029192142.552	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192142.552	OkHttp	x-amz-date: 20211029T135142Z
D	20211029192142.552	OkHttp	--> END PUT
D	20211029192145.270	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=2 (2717ms)
D	20211029192145.272	OkHttp	x-amz-request-id: 7eae625a095d2537
D	20211029192145.272	OkHttp	x-amz-id-2: aOFZiAzdNMFA3dGHwZWk5aWQvNFM3wjnm
D	20211029192145.272	OkHttp	ETag: "ef9484b32a89512c9900f90efa5680b1"
D	20211029192145.273	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192145.273	OkHttp	Date: Fri, 29 Oct 2021 13:51:44 GMT
D	20211029192145.273	OkHttp	Keep-Alive: timeout=5
D	20211029192145.273	OkHttp	Connection: keep-alive
D	20211029192145.274	OkHttp	<-- END HTTP
D	20211029192145.413	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=3 http/1.1 (5242880-byte body)
D	20211029192145.413	OkHttp	Content-Type: application/octet-stream
D	20211029192145.413	OkHttp	Content-Length: 5242880
D	20211029192145.413	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192145.413	OkHttp	Accept-Encoding: identity
D	20211029192145.413	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192145.414	OkHttp	Content-MD5: bmkA7uZvXTiyUMYLqtuEuA==
D	20211029192145.414	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192145.414	OkHttp	x-amz-date: 20211029T135145Z
D	20211029192145.414	OkHttp	--> END PUT
D	20211029192148.650	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=3 (3235ms)
D	20211029192148.651	OkHttp	x-amz-request-id: 2268e2fd86dddbd7
D	20211029192148.652	OkHttp	x-amz-id-2: aOOZiDjcyMNg3FWFoZeQ5eWRTNGg3QTmE
D	20211029192148.653	OkHttp	ETag: "6e6900eee66f5d38b250c60baadb84b8"
D	20211029192148.653	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192148.654	OkHttp	Date: Fri, 29 Oct 2021 13:51:47 GMT
D	20211029192148.654	OkHttp	Keep-Alive: timeout=5
D	20211029192148.654	OkHttp	Connection: keep-alive
D	20211029192148.654	OkHttp	<-- END HTTP
D	20211029192148.760	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=4 http/1.1 (5242880-byte body)
D	20211029192148.760	OkHttp	Content-Type: application/octet-stream
D	20211029192148.761	OkHttp	Content-Length: 5242880
D	20211029192148.761	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192148.761	OkHttp	Accept-Encoding: identity
D	20211029192148.761	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192148.761	OkHttp	Content-MD5: BgOtBD9qdhqzsy91HhksyA==
D	20211029192148.761	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192148.761	OkHttp	x-amz-date: 20211029T135148Z
D	20211029192148.761	OkHttp	--> END PUT
D	20211029192151.109	OkHttp	<-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=4 (2348ms)
D	20211029192151.111	OkHttp	x-amz-request-id: ad2d614a1c2b8ce2
D	20211029192151.112	OkHttp	x-amz-id-2: aOG1i7TeNMEs3jWHRZUM5wmT1NJs3WTmc
D	20211029192151.112	OkHttp	ETag: "0603ad043f6a761ab3b32f751e192cc8"
D	20211029192151.112	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192151.112	OkHttp	Date: Fri, 29 Oct 2021 13:51:50 GMT
D	20211029192151.113	OkHttp	Keep-Alive: timeout=5
D	20211029192151.113	OkHttp	Connection: keep-alive
D	20211029192151.113	OkHttp	<-- END HTTP
D	20211029192151.258	OkHttp	--> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=5 http/1.1 (5242880-byte body)
D	20211029192151.259	OkHttp	Content-Type: application/octet-stream
D	20211029192151.259	OkHttp	Content-Length: 5242880
D	20211029192151.259	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192151.259	OkHttp	Accept-Encoding: identity
D	20211029192151.259	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192151.259	OkHttp	Content-MD5: 92J8ozZqevkBmadcXQ1YwA==
D	20211029192151.259	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192151.259	OkHttp	x-amz-date: 20211029T135151Z
D	20211029192151.259	OkHttp	--> END PUT
D	20211029192152.373	OkHttp	<-- HTTP FAILED: java.net.SocketException: Connection reset
D	20211029192152.386	OkHttp	--> DELETE https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001 http/1.1 (unknown length)
D	20211029192152.387	OkHttp	Host: s3.us-west-000.backblazeb2.com
D	20211029192152.388	OkHttp	Accept-Encoding: identity
D	20211029192152.389	OkHttp	User-Agent: MinIO (Linux; aarch64) minio-java/dev
D	20211029192152.389	OkHttp	Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D	20211029192152.389	OkHttp	x-amz-content-sha256: UNSIGNED-PAYLOAD
D	20211029192152.389	OkHttp	x-amz-date: 20211029T135152Z
D	20211029192152.389	OkHttp	--> END DELETE
D	20211029192154.795	OkHttp	<-- 204  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001 (2405ms)
D	20211029192154.797	OkHttp	x-amz-request-id: dfd418d42dc54fa5
D	20211029192154.798	OkHttp	x-amz-id-2: aOJFiIjfcMCY3f2HwZRU5jGQ+NB83BDlA
D	20211029192154.798	OkHttp	Cache-Control: max-age=0, no-cache, no-store
D	20211029192154.798	OkHttp	Date: Fri, 29 Oct 2021 13:51:55 GMT
D	20211029192154.799	OkHttp	Keep-Alive: timeout=5
D	20211029192154.799	OkHttp	Connection: keep-alive
D	20211029192154.799	OkHttp	<-- END HTTP
D	20211029192154.919	UploadFilesUseCase	failed 4c519f94
D	20211029192154.950	ExceptionHandler	Unexpected error
org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
	at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
	at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
	at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
	at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
	at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
	at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
	at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.Flowable.subscribe(Flowable.java:14882)
	at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
	at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
	at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
	... 22 more
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:215)
	at java.net.SocketInputStream.read(SocketInputStream.java:144)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readFromSocket(ConscryptEngineSocket.java:936)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:900)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readUntilDataAvailable(ConscryptEngineSocket.java:815)
	at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.read(ConscryptEngineSocket.java:788)
	at okio.InputStreamSource.read(JvmOkio.kt:90)
	at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:427)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:320)
	at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
	at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178)
	at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.provideOfflineCacheInterceptor$lambda-1(S3ClientFactory.kt:79)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.$r8$lambda$ZlZNFkypEyH2ozaExGLRHlQReU8(Unknown Source:0)
	at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion$$ExternalSyntheticLambda0.intercept(Unknown Source:2)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
	at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.minio.S3Base.execute(S3Base.java:532)
	at io.minio.S3Base.uploadPart(S3Base.java:2159)
	at io.minio.S3Base.putObject(S3Base.java:1175)
	at io.minio.MinioClient.putObject(MinioClient.java:1666)
	at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
	at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)

Relevant Log Output

@pawanrai9999 did some research and has found out the following:

Uploading files
During an upload using the b2_upload_file or b2_upload_part API, if one of the following conditions occur, the developer should call b2_get_upload_url or b2_get_upload_part_url and re-attempt the upload using the new URL and auth key pair:

  • If an HTTPS connection cannot be made. (e.g., connection timeout).
  • The HTTPS connection is broken. (e.g., broken pipe or connection reset)
  • The HTTPS connection times out waiting for a response (socket timeout).
  • The HTTP response status is 408 (Request Timeout).
  • The HTTP response status is between 500-599, including 503 Service Unavailable.

From https://www.backblaze.com/b2/docs/integration_checklist.html

We could implement such an exponential backoff retry strategy in our application logic but IMO it would be better kept here and would especially help other users who encounter this problem as well.
@SailReal SailReal mentioned this issue Nov 9, 2021
Open
2 tasks
@harshavardhana
Copy link
Member

@pawanrai9999 did some research and has found out the following:

Uploading files
During an upload using the b2_upload_file or b2_upload_part API, if one of the following conditions occur, the developer should call b2_get_upload_url or b2_get_upload_part_url and re-attempt the upload using the new URL and auth key pair:

  • If an HTTPS connection cannot be made. (e.g., connection timeout).
  • The HTTPS connection is broken. (e.g., broken pipe or connection reset)
  • The HTTPS connection times out waiting for a response (socket timeout).
  • The HTTP response status is 408 (Request Timeout).
  • The HTTP response status is between 500-599, including 503 Service Unavailable.

From https://www.backblaze.com/b2/docs/integration_checklist.html

We could implement such an exponential backoff retry strategy in our application logic but IMO it would be better kept here and would especially help other users who encounter this problem as well.

This is a non-S3 API behavior minio-java is an S3 API compatible SDK what are the alternatives for S3 API from B2 ?

@SailReal
Copy link
Author

SailReal commented Nov 9, 2021

Regarding HTTP status code 503 Amazon AWS states this too:

Amazon S3 supports a request rate of 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix in a bucket. The resources for this request rate aren't automatically assigned when a prefix is created. Instead, as the request rate for a prefix increases gradually, Amazon S3 automatically scales to handle the increased request rate.

If there is a fast spike in the request rate for objects in a prefix, Amazon S3 might return 503 Slow Down errors while it scales in the background to handle the increased request rate. To avoid these errors, you can configure your application to gradually increase the request rate and retry failed requests using an exponential backoff algorithm.

From https://aws.amazon.com/premiumsupport/knowledge-center/s3-503-within-request-rate-prefix/

@harshavardhana
Copy link
Member

503 is automatically retried by Okhttp @SailReal you need to provide a retriable HTTP client and okhttp provides such provisions.

We do not do anything special on our end instead it's up to you to write a retriable

https://github.com/square/okhttp/blob/07309c1c7d9e296014268ebd155ebf7ef8679f6c/okhttp/src/main/java/okhttp3/internal/http/RetryAndFollowUpInterceptor.java

https://square.github.io/okhttp/3.x/okhttp/okhttp3/OkHttpClient.Builder.html#retryOnConnectionFailure-boolean-

So you can provide a custom httpClient that does what you want we do not initialize this ourselves, so you should do what you want.

  public static OkHttpClient newDefaultHttpClient(
      long connectTimeout, long writeTimeout, long readTimeout) {
    OkHttpClient httpClient =
        new OkHttpClient()
            .newBuilder()
            .connectTimeout(connectTimeout, TimeUnit.MILLISECONDS)
            .writeTimeout(writeTimeout, TimeUnit.MILLISECONDS)
            .readTimeout(readTimeout, TimeUnit.MILLISECONDS)
            .protocols(Arrays.asList(Protocol.HTTP_1_1))
            .build();
    String filename = System.getenv("SSL_CERT_FILE");
    if (filename != null && !filename.isEmpty()) {
      try {
        httpClient = enableExternalCertificates(httpClient, filename);
      } catch (GeneralSecurityException | IOException e) {
        throw new RuntimeException(e);
      }
    }
    return httpClient;
  }

@balamurugana
Copy link
Member

Please pass custom OkHttpClient for any specific requirements.

@SailReal
Copy link
Author

SailReal commented Jul 4, 2022

FYI: I just looked again into this issue and found out that our application needs to provide a custom interceptor because retryOnConnectionFailure is set to false since the data cannot be provided as a byte array especially in a multi-part upload because of its size...

minio-java/api/src/main/java/io/minio/S3Base.java

Lines 515 to 519 in f8de495

OkHttpClient httpClient = this.httpClient;
if (!(body instanceof byte[]) && (method == Method.PUT || method == Method.POST)) {
// Issue #924: disable connection retry for PUT and POST methods for other than byte array.
httpClient = this.httpClient.newBuilder().retryOnConnectionFailure(false).build();
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@harshavardhana @balamurugana @SailReal