Changing S3 metadata x-amz-meta-x-amz-key and x-amz-meta-x-amz-iv

[davidsonff]

I am trying to do client-side encryption for S3 and then loading into Redshift. I have the following (partial) code:

input.Body = file
            input.Bucket = &bucket
            input.Key = &s3Key
            input.Metadata = map[string]*string{"x-amz-meta-x-amz-key": aws.String(encKeyStr), "x-amz-meta-x-amz-iv": aws.String(initVect)}

            log.Println("Uploading ", file.Name(), " to AWS S3 bucket ", bucket, ", key ", s3Key)

            output, err := uploader.Upload(&input)
            if err != nil {
                    if multierr, ok := err.(s3manager.MultiUploadFailure); ok {
                            // Process error and its associated uploadID
                            log.Fatal("Error:", multierr.Code(), multierr.Message(), multierr.UploadID())
                    } else {
                            // Process error generically
                            log.Fatal("Error:", err.Error())
                    }
            }

When I run it it fails with (after making some changes to unmarshal_error.go):

403 Forbidden
map[Date:[Fri, 07 Aug 2015 21:28:03 GMT] Server:[AmazonS3] X-Amz-Request-Id:[2495C9B900951A75] X-Amz-Id-2:[zHYa2qpTMKBxFhd5AwXFB1fq6dHNOOOoEnUwswUarCSWep8dCQ3XI8+mlsm9s9jL] Content-Type:[application/xml]]
map[]
ps_load.go:363: Error:SerializationError: failed to decode S3 XML error response

When I comment out the input.Metadata line of code, it runs just fine.

Am I doing something incorrectly? Why can I do everything else but add these metadata items? I think I have full authorization on the bucket...

Thanks,

Frank

[jasdel]

Hi @davidsonff initially the header metadata key names should be x-amz-key and x-amz-iv. The x-amz-meta- prefix will automatically be added by the SDK when using S3 operations.

Using this example I was able to write an object into S3 with the x-amz-key and x-amz-iv with dummy data.

package main

import (
    "bytes"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/service/s3"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
    "log"
)

func main() {
    svc := s3.New(nil)

    uploader := s3manager.NewUploader(&s3manager.UploadOptions{
        S3: svc,
    })

    result, err := uploader.Upload(&s3manager.UploadInput{
        Bucket: aws.String("bucketName"),
        Key:    aws.String("keyName"),
        Body:   bytes.NewReader(make([]byte, 5*1024*1024)),
        Metadata: map[string]*string{
            "x-amz-key": aws.String("encKeyStr"),
            "x-amz-iv":  aws.String("initVect"),
        },
    })
    if err != nil {
        log.Fatalln(err)
    }

    log.Println(result)
}

[jasdel]

In order to help debug your issue I suggest enabling debug logging with HTTP body. This will print the detailed response body before it is deserialized by the SDK.

svc := s3.New(aws.NewConfig().WithLogLevel(aws.LogDebugWithHTTPBody))

[davidsonff]

Thanks!!! That fixed it! Also, I was not using base64 to encode the ciphers... Now I just need to figure out how the encryption actually works!!!

[jasdel]

If you haven't already found it, take a look at this help doc it provides a few useful links at the bottom with examples how the Java AWS SDK handles client side encryption.