Flexible Checksums Updates #3845

landonxjames · 2024-09-22T01:37:29Z

Motivation and Context

Updates to flexible checksum functionality to align with the upcoming spec updates.

Note: this PR is to a feature branch since some of the changes here would be breaking before service deployments to align with the updated spec.

Description

Add two new SdkConfig fields response_checksum_validation and request_checksum_calculation that indicate the user's preferred checksum behavior
Use Crc32 as the default checksum algorithm
Update the checksum calculating/validating interceptors to respect the new SdkConfig settings and default checksum
Add two new interceptors (one for requests and one for responses) that each mutate the input to set the default checksum values. Note that these cannot be done in the existing inlineable interceptors because they require access to both runtime values and values from the modeled checksum trait

Testing

Added new suite of tests as modeled in the SEP in HttpChecksumTest.kt

There are a few CI tests failing an all of those are currently expected:

Check PR semver compliance: I marked an enum in aws-smithy-checksums as non-exhaustive. That crate is not yet 1.0 so this change should be fine (and is necessary since we will likely support different checksum types in the future).
Canary: This fails on MPU since the upload_part contains the default crc32 checksum but create_multipart_upload does not specify a checksum. S3 will fix this behavior before we deploy.
check-aws-sdk-smoketest-unit-tests: This fails on an S3 protocol test with missing required header: 'content-md5'. This should be fixed by bumping to the newest smithy version (in a separate PR) since MD5 is now deprecated.

Checklist

For changes to the smithy-rs codegen or runtime crates, I have created a changelog entry Markdown file in the .changelog directory, specifying "client," "server," or both in the applies_to key.
For changes to the AWS SDK, generated SDK code, or SDK runtime crates, I have created a changelog entry Markdown file in the .changelog directory, specifying "aws-sdk-rust" in the applies_to key.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

ysaito1001

Great work on this, leaving comments accumulated so far. Will circle back.

aws/rust-runtime/aws-config/src/default_provider/checksums.rs

rust-runtime/aws-smithy-types/src/checksum_config.rs

aws/rust-runtime/aws-inlineable/src/http_request_checksum.rs

aws/rust-runtime/aws-inlineable/src/http_response_checksum.rs

rust-runtime/aws-smithy-checksums/Cargo.toml

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/AwsCodegenDecorator.kt

rust-runtime/aws-smithy-types/src/checksum_config.rs

aajtodd · 2024-09-25T17:09:43Z

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/HttpRequestChecksumDecorator.kt

@@ -39,20 +47,49 @@ internal fun RuntimeConfig.awsInlineableHttpRequestChecksum() =
 CargoDependency.smithyHttp(this),
 CargoDependency.smithyRuntimeApiClient(this),
 CargoDependency.smithyTypes(this),
+ AwsCargoDependency.awsSigv4(this),


Why do we need awsSigv4 crate?

The http_request_checksum.rs inlineable uses the sigv4 crate. In previous contexts where it was used that crate was pulled in as a dependency by other decorators. When I started writing the codegen tests for this with a generic model that crate was not being pulled in. Since it is necessary for the inlineable to work I added it as a dependency here.

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/HttpRequestChecksumDecorator.kt

github-actions · 2024-09-25T20:37:53Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

github-actions · 2024-09-25T21:17:39Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

github-actions · 2024-09-25T22:21:03Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

Update codegen to point to new checksum config

To traverse the model via a knowledge index

Update PutBucketLifecycleConfiguration test it now checks the Crc32 checksum header instead of Md5

github-actions · 2024-09-27T06:26:10Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

ysaito1001 · 2024-09-27T17:01:01Z

Add two new interceptors (one for requests and one for responses) that each mutate the input to set the default checksum values. Note that these cannot be done in the existing inlineable interceptors because they require access to both runtime values and values from the modeled checksum trait

I'm thinking whether we can somehow avoid adding new interceptors, since we'd want all related logic to be bundled in one place. What if we augment existing interceptors in aws-inlineable/src/http_request_checksum.rs by introducing another generic parameter for mutating checksum, i.e.,

pub(crate) struct RequestChecksumInterceptor<AP, CM> {
    algorithm_provider: AP,
    checksum_mutator: CM, // this is new, similar to the above `AP`
}

In the impl block, we call the CM closure:

impl<AP, CM> Intercept for RequestChecksumInterceptor<AP, CM>
where
    AP: Fn(&Input) -> Result<Option<ChecksumAlgorithm>, BoxError> + Send + Sync,
    CM: Fn(&mut Input, &ConfigBag) -> Result<(), BoxError> + Send + Sync, // this is new
{
...
    fn modify_before_serialization(
        &self,
        context: &mut BeforeSerializationInterceptorContextMut<'_>,
        _runtime_components: &RuntimeComponents,
        cfg: &mut ConfigBag,
    ) -> Result<(), BoxError> {
        (self.checksum_mutator)(context.input_mut(), cfg)
    }
...
}

To wire things up, where we instantiate this struct in codegen, give this code to the closure for CM (we can borrow the idea from how to pass the argument for the closure AP). At that place, we should have access to things like checksumTrait.isRequestChecksumRequired.

The same argument goes for aws-inlineable/src/http_response_checksum.rs.

I haven't verified if this idea compiles or works. But the net benefit is that we don't have to render new interceptor structs per operation.

FYI, I'm ok with merging what we have and iterate on this to avoid churn (new interceptors are private so we can change later we want to).

github-actions · 2024-09-30T00:52:44Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

landonxjames · 2024-09-30T01:06:36Z

Good suggestion @ysaito1001, implemented in d682c8d and 6c9c3da

github-actions · 2024-10-02T01:28:06Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

aajtodd

Overall looks good. One non-blocking question.

aajtodd · 2024-10-02T17:10:04Z

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/HttpResponseChecksumDecorator.kt

+ ##[allow(clippy::wildcard_in_or_patterns)]
+ match response_checksum_validation {
+ #{ResponseChecksumValidation}::WhenRequired => {}
+ #{ResponseChecksumValidation}::WhenSupported | _ => {


Why | _ ?

The {Response | Request}ChecksumValidation enums are marked non-exhaustive as a safety measure so have to handle that case somehow. Since these should always be set before this point that case shouldn't actually be encountered, but if it is we treat it the same as the default (WhenSupported).

Velfi · 2024-10-15T18:17:43Z

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/HttpResponseChecksumDecorator.kt

+ if (!serviceHasHttpChecksumOperation(codegenContext)) {
+ listOf()
+ } else {
+ listOf(
+ adhocCustomization<SdkConfigSection.CopySdkConfigToClientConfig> { section ->
+ rust(
+ """
+ ${section.serviceConfigBuilder}.set_response_checksum_validation(${section.sdkConfig}.response_checksum_validation());
+ """,
+ )
+ },
+ )
+ }


obviously it's too late to update this PR now, but in the future we should avoid if nots and instead define this (and things like it) like this:

if (serviceHasHttpChecksumOperation(codegenContext)) { listOf( adhocCustomization<SdkConfigSection.CopySdkConfigToClientConfig> { section -> rust(""" ${section.serviceConfigBuilder} .set_response_checksum_validation( ${section.sdkConfig}.response_checksum_validation() ); """) }, ) } else { listOf() }

Velfi · 2024-10-15T18:22:40Z

aws/sdk/integration-tests/s3/tests/express.rs

+ assert_eq!(headers.get(0).unwrap().0, "x-amz-checksum-mode");
+ assert_eq!(headers.get(0).unwrap().1, "ENABLED");


For tuples, imo just compare them to tuples:

assert_eq!(headers.get(0).unwrap(), ("x-amz-checksum-mode", "ENABLED"));

Velfi · 2024-10-15T18:30:02Z

aws/rust-runtime/aws-inlineable/src/http_request_checksum.rs

+ let _ = (self.checksum_mutator)(context.input_mut(), cfg);
 let checksum_algorithm = (self.algorithm_provider)(context.input())?;
-
 let mut layer = Layer::new("RequestChecksumInterceptor");
- layer.store_put(RequestChecksumInterceptorState { checksum_algorithm });
+ layer.store_put(RequestChecksumInterceptorState {
+ checksum_algorithm: checksum_algorithm.0,
+ request_checksum_required: checksum_algorithm.1,
+ });


prefer destructuring and concise struct field assignment:

let _ = (self.checksum_mutator)(context.input_mut(), cfg); let (checksum_algorithm, request_checksum_required) = (self.algorithm_provider)(context.input())?; let mut layer = Layer::new("RequestChecksumInterceptor"); layer.store_put(RequestChecksumInterceptorState { checksum_algorithm, request_checksum_required, });

landonxjames force-pushed the landonxjames/flex branch 8 times, most recently from cf331e4 to 3175d0c Compare September 22, 2024 21:34

ysaito1001 reviewed Sep 23, 2024

View reviewed changes

aajtodd reviewed Sep 25, 2024

View reviewed changes

landonxjames force-pushed the landonxjames/flex branch from be5e2d4 to 6182401 Compare September 25, 2024 19:25

landonxjames force-pushed the landonxjames/flex branch from 2879b9e to bf37449 Compare September 25, 2024 20:59

smithy-lang deleted a comment from github-actions bot Sep 25, 2024

landonxjames force-pushed the landonxjames/flex branch from bf37449 to 8df2e4c Compare September 25, 2024 22:01

landonxjames force-pushed the landonxjames/flex branch from 8df2e4c to d0aedfe Compare September 26, 2024 04:31

landonxjames added 10 commits September 26, 2024 23:07

Add streaming checksum tests

50728c4

Remove S3MPU Decorator

e9d4c89

Add changelog

6ae5120

Cleaning up

ae0a7b9

Move checksum configs to aws_smithy_types

d1c6aad

Update codegen to point to new checksum config

Exclude x-amz-checksum-mode header from signing

b9d6ee0

PR feedback

06b384f

Add comment requested in PR feedback

c4b41bc

Update serviceHasHttpChecksumOperation

13d1626

To traverse the model via a knowledge index

Update some s3 integ and protocol tests

1f25cb4

Update PutBucketLifecycleConfiguration test it now checks the Crc32 checksum header instead of Md5

landonxjames force-pushed the landonxjames/flex branch from d0aedfe to 1f25cb4 Compare September 27, 2024 06:07

landonxjames mentioned this pull request Sep 29, 2024

Adding user-agent metrics for flexible checksums #3851

Merged

landonxjames added 2 commits September 29, 2024 17:00

Moving request checksum mutator to a closure

d682c8d

Move response mutator to closure

6c9c3da

landonxjames added 4 commits September 29, 2024 18:07

Adding user-agent metrics for flexible checksums

2027715

Add tests for checksum user-agent business metrics

aec2180

Update HttpChecksumTest to use UA test utils

d714eaa

Adding user-agent metrics for flexible checksums (#3851)

1339593

aajtodd approved these changes Oct 2, 2024

View reviewed changes

landonxjames merged commit 76f8a55 into feature/flexible-checksums Oct 2, 2024
42 of 44 checks passed

landonxjames deleted the landonxjames/flex branch October 2, 2024 21:57

Velfi changed the title ~~Flexible Checksums Updatess~~ Flexible Checksums Updates Oct 15, 2024

Velfi reviewed Oct 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flexible Checksums Updates #3845

Flexible Checksums Updates #3845

landonxjames commented Sep 22, 2024 •

edited

Loading

ysaito1001 left a comment

aajtodd Sep 25, 2024

landonxjames Sep 25, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 27, 2024

ysaito1001 commented Sep 27, 2024 •

edited

Loading

github-actions bot commented Sep 30, 2024

landonxjames commented Sep 30, 2024

github-actions bot commented Oct 2, 2024

aajtodd left a comment

aajtodd Oct 2, 2024

landonxjames Oct 2, 2024

Velfi Oct 15, 2024 •

edited

Loading

Velfi Oct 15, 2024

Velfi Oct 15, 2024

		assert_eq!(headers.get(0).unwrap().0, "x-amz-checksum-mode");
		assert_eq!(headers.get(0).unwrap().1, "ENABLED");

Flexible Checksums Updates #3845

Flexible Checksums Updates #3845

Conversation

landonxjames commented Sep 22, 2024 • edited Loading

Motivation and Context

Description

Testing

Checklist

ysaito1001 left a comment

Choose a reason for hiding this comment

aajtodd Sep 25, 2024

Choose a reason for hiding this comment

landonxjames Sep 25, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 27, 2024

ysaito1001 commented Sep 27, 2024 • edited Loading

github-actions bot commented Sep 30, 2024

landonxjames commented Sep 30, 2024

github-actions bot commented Oct 2, 2024

aajtodd left a comment

Choose a reason for hiding this comment

aajtodd Oct 2, 2024

Choose a reason for hiding this comment

landonxjames Oct 2, 2024

Choose a reason for hiding this comment

Velfi Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

Velfi Oct 15, 2024

Choose a reason for hiding this comment

Velfi Oct 15, 2024

Choose a reason for hiding this comment

landonxjames commented Sep 22, 2024 •

edited

Loading

ysaito1001 commented Sep 27, 2024 •

edited

Loading

Velfi Oct 15, 2024 •

edited

Loading