HEAD request not returning custom metadata fields

[akeustis]

I'm using S3Mock for integration tests and discovered that S3Mock apparently does not return client-supplied metadata fields when responding to a HEAD request.

When sending a PutObject request, any headers the client sends (up to 2KB, I think) starting with the X-Amz-Meta prefix are supposed to be persisted as object metadata and then returned on both GetObject and HeadObject requests. I know this works in my application with the real S3. In my application it's important to be able to retrieve metadata without having to read the object itself.

Unfortunately this did not work when I ran my integration test against S3Mock. I am using the Golang S3 SDK (https://docs.aws.amazon.com/sdk-for-go/api/service/s3) and the Metadata map (referenced below) was empty, even though I'm certain it was non-empty with the corresonding PutObject.

type HeadObjectOutput struct {
    . . .

    // A map of metadata to store with the object in S3.
    //
    // By default unmarshaled keys are written as a map keys in following canonicalized format:
    // the first letter and any letter following a hyphen will be capitalized, and the rest as lowercase.
    // Set `aws.Config.LowerCaseHeaderMaps` to `true` to write unmarshaled keys to the map as lowercase.
    Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"`
}

Here is the documentation for the corresponding PutObject field (which is also implemented via X-Amz-Meta headers):

type PutObjectInput struct {
   . . .

    // A map of metadata to store with the object in S3.
    Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"`
}

[afranken]

@akeustis
the behaviour you described is implemented and we have multiple integration tests verifying it works...

This is a simple example using the AWS CLI:

❯ docker run -p 9090:9090 -p 9191:9191 -e validKmsKeys="arn:aws:kms:us-east-1:1234567890:key/valid-test-key-ref" -e initialBuckets="bucket-a, bucket-b" -e LOGGING_LEVEL_ROOT=DEBUG -t adobe/s3mock:latest
[...]

❯ aws s3 cp ./test.json s3://bucket-a/ --endpoint-url=http://localhost:9090 --metadata '{"x-amz-meta-my-key":"MY_DATA"}'
upload: ./test.json to s3://bucket-a/test.json

❯ aws s3api head-object --bucket bucket-a --key test.json --endpoint-url=http://localhost:9090
{
    "AcceptRanges": "bytes",
    "LastModified": "2023-06-13T06:50:52+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ContentType": "application/json",
    "Metadata": {
        "x-amz-meta-my-key": "MY_DATA"
    }
}

[akeustis]

Thanks so much for your reply. I've replicated your test successfully with or without capitalizing x-amz-meta, which leads me to believe there's nothing wrong with s3mock and I must be using the Go SDK incorrectly.

Specifically I'm now 99% sure it has something to do with the EndpointResolver that I'm using with my local integration test but not with the real S3. It must be eating the headers somehow. Unless you have any helpful hints about using the Go SDK I'll go ahead and close this.

EDITED TO ADD: there is a still an issue. See next comment

[akeustis]

On further investigation, I'd like to reopen this. After a whole bunch of debugging I was finally able to discover the problem: the x-amz-meta header prefix is case-sensitive! Here is a minimal example to reproduce the error:

curl --request PUT 'http://localhost:9090/bucket-a/key' \
--header 'host: localhost' \
--header 'X-Amz-Meta-somekey: somevalue'

If we follow that with

curl --head 'http://localhost:9090/bucket-a/key'

then there is a 200 response with no x-amz-meta-somekey header.

On the the other hand, if we repeat the PUT request with lower-cased x-amz-meta-somekey, then the value persists as expected.

This is a problem because the real S3 API does not exhibit this case-sensitive behavior. Furthermore, many commonly used http clients (including Go's net/http package) automatically set headers to a "canonical" form with capitalized letters (i.e. X-Amz-Meta-Somekey). If so, then those custom metadata values will get lost in s3mock.

[afranken]

yeah, we are only looking for lower case header names:
https://github.com/adobe/S3Mock/blob/main/server/src/main/java/com/adobe/testing/s3mock/util/HeaderUtil.java#L68

also, we only test with the Java SDKv1 and SDKv2 which both send lower case headers when setting userMetadata in the PutObject requests:
https://github.com/adobe/S3Mock/blob/main/integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/GetPutDeleteObjectV1IT.kt#L151

Let me look into this, shouldn't be too hard to fix. :)

[akeustis]

Thanks again! That would be wonderful. I was able to come up with a workaround and use the Go SDK with a custom middleware that forces the headers to lowercase, but I'd love for no one else to have to face this frustration.

[afranken]

@akeustis this was an easy fix, just released 2.13.0 including the fix.