Decrypting non-encrypted content with AmazonS3EncryptionClientV2

[ashishdhingra]

Discussed in #63

^{Originally posted by simenstensas September 30, 2024}
Hi! I've upgraded to the latest preview of AmazonS3EncryptionClientV2 and so far it works out of the box with encrypted content.

However it fails when getting non-encrypted content with message "Amazon.Runtime.AmazonServiceException: Unable to decrypt data for object [object] in bucket [bucket]". Since the metadata contains information about whether the content is encrypted or not, could one just not decrypt it if no headers are found?

If this is not possible, what could be a solution for me?

[ashishdhingra]

We would need to check:

• Behavior on how it worked with previous obsolete version of encryption client, and also with Java encryption client.
• Need to figure out if the behavior was implemented as a result of security practice.
• If changing behavior would be a breaking change.

[ashishdhingra]

Findings when using Amazon.Extensions.S3.Encryption version 2.2.0
Reproducible using code below:

using Amazon.Extensions.S3.Encryption;
using Amazon.Extensions.S3.Encryption.Primitives;
using Amazon.S3;
using Amazon.S3.Model;

string bucketName = "<<bucket-name>>";
Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
Amazon.AWSConfigs.LoggingConfig.LogMetrics = true;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

S3EncryptionTest(bucketName, "<<kms-key-guid>>");

static void S3EncryptionTest(string bucketName, string kmsKeyId)
{
    var encryptionContext = new Dictionary<string, string>();
    var encryptionMaterial =
        new EncryptionMaterialsV2(kmsKeyId, KmsType.KmsContext, encryptionContext);
    var configuration = new AmazonS3CryptoConfigurationV2(SecurityProfile.V2);

    using (AmazonS3Client amazonS3Client = new AmazonS3Client())
    {
        var response = amazonS3Client.PutObjectAsync(new PutObjectRequest()
        {
            BucketName = bucketName,
            Key = "test-notencrypted-file.txt",
            ContentBody = "Testing S3 Encryption"
        }).Result;
    }
    
    using (var encryptionClient = new AmazonS3EncryptionClientV2(configuration, encryptionMaterial))
    {
        EncryptObject(encryptionClient, bucketName, "test-encrypted-file.txt", "Testing S3 Encryption");
        DecryptObject(encryptionClient, bucketName, "test-encrypted-file.txt");
        DecryptObject(encryptionClient, bucketName, "test-notencrypted-file.txt");
    }
}

static void EncryptObject(AmazonS3EncryptionClientV2 encryptionClient, string bucketName, string key, string contentBody)
{
    var putRequest = new PutObjectRequest
    {
        BucketName = bucketName,
        Key = key,
        ContentBody = contentBody
    };

    var response = encryptionClient.PutObjectAsync(putRequest).Result;
}

static void DecryptObject(AmazonS3EncryptionClientV2 encryptionClient, string bucketName, string key)
{
    var getRequest = new GetObjectRequest
    {
        BucketName = bucketName,
        Key = key
    };

    using (var response = encryptionClient.GetObjectAsync(getRequest).Result)
    {
        using (StreamReader reader = new StreamReader(response.ResponseStream))
        {
            string contents = reader.ReadToEnd();
            Console.WriteLine("Contents: " + contents);
        }
    }
}

Throws the exception System.AggregateException: 'System.AggregateException: 'One or more errors occurred. (Unable to decrypt data for object test-notencrypted-file.txt in bucket <<bucket-name>>)''.

Below is the stack trace:

AmazonS3EncryptionClientV2 59|2024-11-11T14:25:30.400Z|ERROR|An exception of type AmazonServiceException was handled in ErrorHandler. --> Amazon.Runtime.AmazonServiceException: Unable to decrypt data for object test-notencrypted-file.txt in bucket <<bucket-name>>
 ---> Amazon.S3.AmazonS3Exception: The specified key does not exist.
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.GetInstructionFileAsync(GetObjectRequest instructionFileRequest) in D:\source\GitHub\amazon-s3-encryption-client-dotnet\src\Internal\SetupDecryptionHandler.cs:line 220
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.DecryptObjectAsync(Byte[] decryptedEnvelopeKeyKMS, GetObjectResponse getObjectResponse) in D:\source\GitHub\amazon-s3-encryption-client-dotnet\src\Internal\SetupDecryptionHandler.cs:line 202
   --- End of inner exception stack trace ---
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.DecryptObjectAsync(Byte[] decryptedEnvelopeKeyKMS, GetObjectResponse getObjectResponse) in D:\source\GitHub\amazon-s3-encryption-client-dotnet\src\Internal\SetupDecryptionHandler.cs:line 206
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.PostInvokeAsync(IExecutionContext executionContext) in D:\source\GitHub\amazon-s3-encryption-client-dotnet\src\Internal\SetupDecryptionHandler.cs:line 151
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\source\GitHub\amazon-s3-encryption-client-dotnet\src\Internal\SetupDecryptionHandler.cs:line 127
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonS3EncryptionClientV2 60|2024-11-11T14:25:30.432Z|ERROR|AmazonServiceException making request GetObjectRequest to https://testbucket-issue1880.s3.us-east-2.amazonaws.com/. Attempt 1. --> Amazon.Runtime.AmazonServiceException: Unable to decrypt data for object test-notencrypted-file.txt in bucket testbucket-issue1880
 ---> Amazon.S3.AmazonS3Exception: The specified key does not exist.
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
...

Raw HTTPS response:

HTTP/1.1 404 Not Found
x-amz-request-id: 0FC12N5M2S52JH98
x-amz-id-2: gDohIlO6nQoEL316QyTIZnKXv1Yl95v45etM8Eph3O4WEGCm9Q0gNpjWIah2KggESrVURHFmpS4SCRJFdjp9L2gkCPux3iaNA86SwUcABz4=
Content-Type: application/xml
Date: Tue, 12 Nov 2024 18:05:52 GMT
Server: AmazonS3
Content-Length: 347

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>test-notencrypted-file.txtINSTRUCTION_SUFFIX</Key><RequestId>0FC12N5M2S52JH98</RequestId><HostId>gDohIlO6nQoEL316QyTIZnKXv1Yl95v45etM8Eph3O4WEGCm9Q0gNpjWIah2KggESrVURHFmpS4SCRJFdjp9L2gkCPux3iaNA86SwUcABz4=</HostId></Error>

---

Testing using Java amazon-s3-encryption-client-java version 3.1.2:

Make sure to add following dependencies in pom.xml (refer Amazon S3 Encryption Client for Java):

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>kms</artifactId>
</dependency>

<dependency>
  <groupId>software.amazon.encryption.s3</groupId>
  <artifactId>amazon-s3-encryption-client-java</artifactId>
  <version>3.1.2</version>
</dependency>

Used below code:

package org.example;

import software.amazon.awssdk.core.ResponseBytes;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.encryption.s3.S3EncryptionClient;

public class Handler {
    public void sendRequest() {
        String bucketName = "<<bucket-name>>";

        String encryptedObjectKey = "test-s3-encryption-key.txt";
        String nonencryptedObjectKey = "test-notencrypted-file.txt";
        String kmsKeyId = "<<kms-key-guid>>";
        S3Client v3Client = S3EncryptionClient.builder()
                .kmsKeyId(kmsKeyId)
                .build();

        v3Client.putObject(PutObjectRequest.builder()
                .bucket(bucketName)
                .key(encryptedObjectKey)
                .build(), RequestBody.fromString("Testing S3 Encryption"));

        ResponseBytes<GetObjectResponse> objectResponse = v3Client.getObjectAsBytes(builder -> builder
                .bucket(bucketName)
                .key(encryptedObjectKey));
        String output = objectResponse.asUtf8String();
        System.out.println(output);

        ResponseBytes<GetObjectResponse> nonencryptedObjectResponse = v3Client.getObjectAsBytes(builder -> builder
                .bucket(bucketName)
                .key(nonencryptedObjectKey));
    }
}

It throws below exception:

Exception in thread "main" software.amazon.encryption.s3.S3EncryptionClientException: Instruction file not found! Please ensure the object you are attempting to decrypt has been encrypted using the S3 Encryption Client.
	at software.amazon.encryption.s3.S3EncryptionClient.getObject(S3EncryptionClient.java:255)
	at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:9542)
	at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:9784)
	at org.example.Handler.sendRequest(Handler.java:50)
	at org.example.App.main(App.java:12)
Caused by: software.amazon.encryption.s3.S3EncryptionClientException: Instruction file not found! Please ensure the object you are attempting to decrypt has been encrypted using the S3 Encryption Client.
	at software.amazon.encryption.s3.internal.ContentMetadataStrategy$1.decodeMetadata(ContentMetadataStrategy.java:49)
	at software.amazon.encryption.s3.internal.ContentMetadataStrategy.decode(ContentMetadataStrategy.java:226)
	at software.amazon.encryption.s3.internal.GetEncryptedObjectPipeline$DecryptingResponseTransformer.onResponse(GetEncryptedObjectPipeline.java:120)
	at software.amazon.encryption.s3.internal.GetEncryptedObjectPipeline$DecryptingResponseTransformer.onResponse(GetEncryptedObjectPipeline.java:91)
	at software.amazon.awssdk.core.async.listener.AsyncResponseTransformerListener$NotifyingAsyncResponseTransformer.onResponse(AsyncResponseTransformerListener.java:87)
	at software.amazon.awssdk.core.internal.http.async.AsyncStreamingResponseHandler.onHeaders(AsyncStreamingResponseHandler.java:55)
	at software.amazon.awssdk.core.internal.http.IdempotentAsyncResponseHandler.onHeaders(IdempotentAsyncResponseHandler.java:103)
	at software.amazon.awssdk.core.internal.http.async.CombinedResponseAsyncHttpResponseHandler.onHeaders(CombinedResponseAsyncHttpResponseHandler.java:58)
	at software.amazon.awssdk.core.internal.http.async.AsyncAfterTransmissionInterceptorCallingResponseHandler.onHeaders(AsyncAfterTransmissionInterceptorCallingResponseHandler.java:67)
	at software.amazon.awssdk.core.internal.http.async.FilterTransformingAsyncHttpResponseHandler.onHeaders(FilterTransformingAsyncHttpResponseHandler.java:44)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ReadMetricsTrackingResponseHandler.onHeaders(MakeAsyncHttpRequestStage.java:305)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelRead0(ResponseHandler.java:102)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelRead0(ResponseHandler.java:75)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.channelRead(HandlerPublisher.java:396)
...
Caused by: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: 4YCCW3PNJTWFES45, Extended Request ID: u4ixnW71kxhLzVcvzf0pArGvFodvHlcdc3SpAjuIu1CdK1SThw6/HMLd1vXkQ+2xLOMnuY9+jkxSIzJ4wTUy4g==)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:66)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
	at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:5203)
	at software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:9063)
	at software.amazon.encryption.s3.internal.ContentMetadataStrategy$1.decodeMetadata(ContentMetadataStrategy.java:45)
	... 65 more

---

Looks like both Java and .NET clients throw the same exception.
For .NET, S3 service is returning <Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>test-notencrypted-file.txtINSTRUCTION_SUFFIX</Key>. Unsure why key is test-notencrypted-file.txtINSTRUCTION_SUFFIX.

---

Dive Deep:

• INSTRUCTION_SUFFIX in .NET is declared here.

• While getting S3 object, INSTRUCTION_SUFFIX in .NET is used here.

• The logic is able to get object from S3 at SetupDecryptionHandler.InvokeAsync(). The return type is GetObjectResponse.

  • For encrypted object, the raw response contains encrypted payload:

    HTTP/1.1 200 OK
    x-amz-id-2: mYKRjrrSPTUOZvwPZNbRfJNH+3wCD3lPs5Mz707UJyGuHCobbe4eHoWcv96piqVNKQZ51EqpfI0=
    x-amz-request-id: 3XGDND5DBQFGB32Y
    Date: Tue, 12 Nov 2024 18:51:10 GMT
    Last-Modified: Tue, 12 Nov 2024 18:50:58 GMT
    ETag: "c7326cf3620ddcc74c303f64f3dbfd42"
    x-amz-server-side-encryption: AES256
    x-amz-meta-x-amz-tag-len: 128
    x-amz-meta-x-amz-unencrypted-content-length: 21
    x-amz-meta-x-amz-wrap-alg: kms+context
    x-amz-meta-x-amz-matdesc: {"aws:x-amz-cek-alg":"AES/GCM/NoPadding"}
    x-amz-meta-x-amz-key-v2: AQIDAHi8yqejJR+QrQe/PgC2s6LGCKcm/5WNl0gJCQcyEZvhKAGP/C6PRG6lEGKMqkFnlSr2AAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM2PMDbhWgPH5x3+/5AgEQgDtfYbEHpy3Zk1MoIQzZS9Wm7SNCu3C3B/UuMshxmJHT1XcgscZmojwQps5hUkC5Fv7pvHcRm/YfuSLBiQ==
    x-amz-meta-x-amz-cek-alg: AES/GCM/NoPadding
    x-amz-meta-x-amz-iv: RM4mkwiOPuVAGCds
    Accept-Ranges: bytes
    Content-Type: text/plain
    Content-Length: 37
    Server: AmazonS3
    
     w/	 6 \  %h C  ()-  ` [   v  +  6x 
    

  • For not encrypted object, the raw response contains actual payload:

    HTTP/1.1 200 OK
    x-amz-id-2: 9qVnIBMN3ibXzvc5O9bIXP6WD3HFkWB7gF3ymjN72gk2LU3I0q3OViN9JB5wua6SDJM8K/F1ISLFVCc1yWd62w==
    x-amz-request-id: 8NG56RTQZ9PKB9WQ
    Date: Tue, 12 Nov 2024 18:52:34 GMT
    Last-Modified: Tue, 12 Nov 2024 18:50:56 GMT
    ETag: "b8952c8b639021bb47c1fee1d69cde25"
    x-amz-server-side-encryption: AES256
    Accept-Ranges: bytes
    Content-Type: text/plain
    Content-Length: 21
    Server: AmazonS3
    
    Testing S3 Encryption
    

• Thereafter, here it tries to call SetupDecryptionHandler.PostInvokeAsync(), which tries to decrypt response using SetupDecryptionHandler.DecryptObjectAsync().

• In SetupDecryptionHandler.DecryptObjectAsync(), for not encrypted file, since encrypted metadata is absent:

  • It first tries to invoke EncryptionUtils.GetInstructionFileRequest() here with EncryptionUtils.EncryptionInstructionFileV2Suffix (.instruction). It then makes subsequent call to SetupDecryptionHandler.GetInstructionFileAsync() here.
  • When the above call fails with EncryptionUtils.NoSuchKey (NoSuchKey) error code, it then tries to invoke EncryptionUtils.GetInstructionFileRequest() here with EncryptionUtils.EncryptionInstructionFileSuffix (INSTRUCTION_SUFFIX). Thereafter, making call to SetupDecryptionHandler.GetInstructionFileAsync() here. This ultimately fails.

  So, in network monitoring tool, we would see 2 failed requests with below responses:
  Response 1

  HTTP/1.1 404 Not Found
  x-amz-request-id: CJE4JDEHQY2DXYVV
  x-amz-id-2: 9Wb00m6Qdn36lezQUat54krE8PN5jhK730dqzvg+TdFNLNYdrSSS/OgfSvKMphPeICY+X8yxbpTfo6T3fSBb0fxNPjr29m5AalYzh1Q3LlA=
  Content-Type: application/xml
  Transfer-Encoding: chunked
  Date: Tue, 12 Nov 2024 19:12:39 GMT
  Server: AmazonS3
  
  155
  <?xml version="1.0" encoding="UTF-8"?>
  <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>test-notencrypted-file.txt.instruction</Key><RequestId>CJE4JDEHQY2DXYVV</RequestId><HostId>9Wb00m6Qdn36lezQUat54krE8PN5jhK730dqzvg+TdFNLNYdrSSS/OgfSvKMphPeICY+X8yxbpTfo6T3fSBb0fxNPjr29m5AalYzh1Q3LlA=</HostId></Error>
  0
  

  Response 2

  HTTP/1.1 404 Not Found
  x-amz-request-id: CMWSPS4CZ66XXVF8
  x-amz-id-2: ktAXd7/l7LlS67ancqdMUeQIuYOyerl4bq2MGwkf9lPM8NmV0qlERawK3foPSjWJBK38lGACXt6szBBICbjgYIoJnOV3Rdd6Jnjsx5spalE=
  Content-Type: application/xml
  Transfer-Encoding: chunked
  Date: Tue, 12 Nov 2024 19:12:47 GMT
  Server: AmazonS3
  
  15b
  <?xml version="1.0" encoding="UTF-8"?>
  <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>test-notencrypted-file.txtINSTRUCTION_SUFFIX</Key><RequestId>CMWSPS4CZ66XXVF8</RequestId><HostId>ktAXd7/l7LlS67ancqdMUeQIuYOyerl4bq2MGwkf9lPM8NmV0qlERawK3foPSjWJBK38lGACXt6szBBICbjgYIoJnOV3Rdd6Jnjsx5spalE=</HostId></Error>
  0
  

[ashishdhingra]

We had contacted Crypto Tools team (which owns Java S3 Encryption client) and got the below response:

---

The behavior to return plaintext violates the security guarantees of the library. A threat actor with write access to S3 can replace an encrypted object with a plaintext object, and the GetObject operation succeeds. This violates the integrity guarantee, i.e. that the original plaintext has not replaced with a different plaintext. Therefore, plaintext objects must be handled outside of the security boundary of the S3EC.

---

The current behavior makes sense. We would work on improving the error messaging to return exception with message like Please ensure the object you are attempting to decrypt has been encrypted using the S3 Encryption Client., rather than Amazon.Runtime.AmazonServiceException: Unable to decrypt data for object [object] in bucket [bucket] when instruction file is not found.

[ashishdhingra]

Change for improving error message is released in Amazon.Extensions.S3.Encryption version 2.2.1.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.