Unable to use Java AWS SDK v2 to get object using SHA256 #4694

gorell · 2023-11-13T19:50:19Z

Describe the bug

hi team,
after putting the object using the checksum algorithm SHA256 as following,

        PutObjectRequest putObjectRequest = PutObjectRequest.builder()
            .bucket(bucketName)
            .key(key)
            .contentType(contentType)
            .checksumAlgorithm(ChecksumAlgorithm.SHA256)
            .build();
        awsClient.putObject(putObjectRequest, RequestBody.fromString(content));

we're trying to get this object:

          GetObjectResponse getObjectResponse = awsClient.getObject(
                  b -> b
                      .bucket(bucketName)
                      .key(key)
                      .checksumMode(ChecksumMode.ENABLED))
              .response();

but MD5 checksum (and not SHA256) is applied instead, via this hard-coded reference this
and we're getting the exception: java.lang.IllegalStateException: Unexpected error creating MD5 checksum
note: headObject works fine
what are we missing?
thanks in advance

Expected Behavior

GetObject should work

Current Behavior

Getting an exception (see above)

Reproduction Steps

Please see the code in the bug description

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.20.42

JDK version used

openjdk 11.0.14.1

Operating System and version

Linux service-5b7c8c4788-c62fp 5.4.238-148.347.amzn2.x86_64 #1 SMP Thu Apr 6 19:42:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

gorell · 2023-11-13T19:51:01Z

debora-ito · 2023-11-13T22:57:26Z

@gorell can you share the full stacktrace of the error?

gorell · 2023-11-14T05:42:05Z

@debora-ito here it is:

Caused by: java.lang.IllegalStateException: Unexpected error creating MD5 checksum
	at software.amazon.awssdk.core.checksums.Md5Checksum.getDigest(Md5Checksum.java:63)
	at software.amazon.awssdk.core.checksums.Md5Checksum.<init>(Md5Checksum.java:32)
	at software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor.modifyHttpResponseContent(SyncChecksumValidationInterceptor.java:74)
	at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpResponse(ExecutionInterceptorChain.java:148)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:46)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:28)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:68)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:63)
	at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:4535)
	at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:8465)
	at .......doGetObject(....java:322)
	... 65 more
Caused by: java.security.NoSuchAlgorithmException: MD5 MessageDigest not available
	at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
	at java.base/java.security.MessageDigest.getInstance(MessageDigest.java:185)
	at software.amazon.awssdk.core.checksums.Md5Checksum.getDigest(Md5Checksum.java:61)

gorell · 2023-11-14T05:44:22Z

@debora-ito just to clarify - we've customized our JDK to use SHA256 (and not MD5 which we don't support) for checksum

debora-ito · 2023-11-15T02:57:16Z

The Java SDK will validate the MD5 checksum by default on GetObject. This is a client-side configuration, separate from the GetObject checksumMode option, and this is what the code in SyncChecksumValidationInterceptor is doing.

You can disable this default MD5 checksum validation in the S3 Client by setting a custom S3Configuration:

S3Client s3Client = S3Client.builder()
        .serviceConfiguration(S3Configuration.builder().checksumValidationEnabled(Boolean.FALSE).build())
        .region(Region.US_WEST_2)
        .build();

gorell · 2023-11-15T04:04:09Z

Got it @debora-ito , thank you. So is there a way to configure the GetObject validation to be SHA256? Given how the PutObject is configured.

debora-ito · 2023-11-16T02:02:03Z

In this use-case, the SDK will automatically validate the SHA256 checksum on GetObject, after finishing reading the object. The SDK validates any of the 4 checksum types supported by the ChecksumAlgorithm in PutObject.

gorell · 2023-11-16T04:46:07Z

But it looks like it does it with MD5, that's the reason I've opened this issue.

debora-ito · 2023-11-16T20:37:25Z

Ok, I took a step back and went through the question again. I also talked to the rest of the team about this issue.

Firstly, there is a known bug that I think is directly impacting you - we are performing both the default md5 checksum validation AND the checksumAlgorithm validation (SHA256 in your case) upon calling GetObject. We are going to change this to: if checksumMode is enabled on GetObject, we'll only perform the checksumAlgorithm validation, and not perform the md5 validation - the double validation is not necessary in this case.
While the fix for (1) is not released, as a workaround you can disable the md5 checksum in the s3 client by using the S3Configuration I showed in my previous comment -

S3Client s3Client = S3Client.builder()
        .serviceConfiguration(S3Configuration.builder().checksumValidationEnabled(Boolean.FALSE).build())
        .region(Region.US_WEST_2)
        .build();

To clarify: setting checksumValidationEnabled() to false will only disable the md5 checksum validation, not the SHA256 validation. The SHA256 checksum validation will still be performed if checksumMode is enabled in GetObject.

@gorell Will the fix of (1) address the issue you are reporting here? Is there anything else I'm missing?

gorell · 2023-11-16T20:49:29Z

@debora-ito The answer to you question is 'yes', the fix of (1) will do the job, thank you very much

debora-ito · 2023-11-16T21:02:36Z

Great, thank you for confirming.

I'll comment here once we have more updates on the fix.

debora-ito · 2023-12-04T21:25:26Z

The fix is available in version 2.21.38.

@gorell

kishd · 2024-08-13T12:39:55Z

@debora-ito - This appears to be an issue with the delete api. "There is no way to disable MD5 for S3Client.deleteObjects because the service API requires the MD5 checksum". https://github.com/aws/aws-sdk-java-v2/blob/master/core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/http/pipeline/stages/HttpChecksumStage.java

gorell added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 13, 2023

debora-ito self-assigned this Nov 13, 2023

debora-ito added closing-soon This issue will close in 4 days unless further comments are made. guidance Question that needs advice or information. and removed bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 15, 2023

github-actions bot removed the closing-soon This issue will close in 4 days unless further comments are made. label Nov 15, 2023

debora-ito added the closing-soon This issue will close in 4 days unless further comments are made. label Nov 16, 2023

github-actions bot removed the closing-soon This issue will close in 4 days unless further comments are made. label Nov 16, 2023

debora-ito added bug This issue is a bug. p2 This is a standard priority issue and removed guidance Question that needs advice or information. labels Nov 16, 2023

gorell mentioned this issue Nov 22, 2023

MD5 checksum crash in AWS SQS receive messages #4717

Closed

davidh44 mentioned this issue Dec 1, 2023

Do not send Md5 checksum header if ChecksumMode is enabled #4731

Merged

12 tasks

debora-ito added the closing-soon This issue will close in 4 days unless further comments are made. label Dec 5, 2023

github-actions bot added closed-for-staleness and removed closing-soon This issue will close in 4 days unless further comments are made. labels Dec 7, 2023

github-actions bot closed this as completed Dec 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to use Java AWS SDK v2 to get object using SHA256 #4694

Unable to use Java AWS SDK v2 to get object using SHA256 #4694

gorell commented Nov 13, 2023 •

edited

Loading

gorell commented Nov 13, 2023

debora-ito commented Nov 13, 2023

gorell commented Nov 14, 2023 •

edited

Loading

gorell commented Nov 14, 2023

debora-ito commented Nov 15, 2023

gorell commented Nov 15, 2023

debora-ito commented Nov 16, 2023

gorell commented Nov 16, 2023 •

edited

Loading

debora-ito commented Nov 16, 2023

gorell commented Nov 16, 2023

debora-ito commented Nov 16, 2023

debora-ito commented Dec 4, 2023

kishd commented Aug 13, 2024

Unable to use Java AWS SDK v2 to get object using SHA256 #4694

Unable to use Java AWS SDK v2 to get object using SHA256 #4694

Comments

gorell commented Nov 13, 2023 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Java SDK version used

JDK version used

Operating System and version

gorell commented Nov 13, 2023

debora-ito commented Nov 13, 2023

gorell commented Nov 14, 2023 • edited Loading

gorell commented Nov 14, 2023

debora-ito commented Nov 15, 2023

gorell commented Nov 15, 2023

debora-ito commented Nov 16, 2023

gorell commented Nov 16, 2023 • edited Loading

debora-ito commented Nov 16, 2023

gorell commented Nov 16, 2023

debora-ito commented Nov 16, 2023

debora-ito commented Dec 4, 2023

kishd commented Aug 13, 2024

gorell commented Nov 13, 2023 •

edited

Loading

gorell commented Nov 14, 2023 •

edited

Loading

gorell commented Nov 16, 2023 •

edited

Loading