Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not send Md5 checksum header if ChecksumMode is enabled #4731

Merged
merged 5 commits into from
Dec 4, 2023

Conversation

davidh44
Copy link
Contributor

@davidh44 davidh44 commented Dec 1, 2023

Motivation and Context

Currently, if checksum validation is enabled (default) and regular S3 is used, EnableTrailingChecksumInterceptor adds x-amz-te:append-md5 to a GetObjectRequest. The server adds the Md5 checksum to the response and the SDK validates it, wrapping with S3ChecksumValidatingInputStream (sync) or S3ChecksumValidatingPublisher (aync).

This is done even if ChecksumMode is enabled on the GetObjectRequest. If a ChecksumAlgorithm was specified on the PutObjectRequest, the HttpChecksumValidationInterceptor will validate the flexible checksum, wrapping with ChecksumValidatingInputStream or ChecksumValidatingPublisher. Validation is performed twice, both for Md5 and flexible checksums.

#4694

Modifications

If ChecksumMode is enabled, do not add x-amz-te:append-md5 to a GetObjectRequest.

If ChecksumMode is enabled and an algorithm was specified when the object was put, only the flexible checksum will be validated.

If ChecksumMode is enabled and no algorithm was specified when the object was put, no validation will be performed. (previously it would fallback to Md5 validation)

Testing

Added unit and integ tests

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

@davidh44 davidh44 requested a review from a team as a code owner December 1, 2023 03:18
Copy link
Contributor

@zoewangg zoewangg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make sure integ tests are passing before merging.

@davidh44 davidh44 enabled auto-merge (squash) December 2, 2023 00:31
@davidh44 davidh44 disabled auto-merge December 2, 2023 00:31
Copy link

sonarqubecloud bot commented Dec 2, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

79.9% 79.9% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@davidh44 davidh44 merged commit ae48b93 into master Dec 4, 2023
7 of 8 checks passed
@davidh44 davidh44 deleted the hdavidh/do-not-validate-checksums-twice branch December 4, 2023 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants