Unable to configure S3 Client when using DynamoDB S3Link

[Danny-UKDM]

Describe the bug

When implementing S3Link for both S3Link.Create() and myLinkedProperty.UploadStreamAsync() there appears to be no method for configuring the AmazonS3Client which is created under the hood, or for providing your own configured instance of an AmazonS3Client.

This appears fine for a production environment, but prevents you from being able to ensure the underlying AmazonS3Client is correctly configured for end-to-end integration testing via a service like localstack.

Despite best efforts to ensure the DynamoDBContext passed into S3Link.Create() is configured for localstack in a "Development" environment (which functions as expected elsewhere), the AmazonS3Client ultimately throws an error of:

"The AWS Access Key Id you provided does not exist in our records"

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Either:

1 - When correctly configuring the DI registration of DynamoDBContext for Development, which is passed into S3Link.Create(), the underlying functionality correctly clones the configuration and creates the correctly Development-configured AmazonS3Client under the hood when performing operations.

2 - S3Link allows the caller to pass their own configured instance(s) of AmazonS3Client which is known to be correctly configured for Development and end-to-end integration testing via a service like localstack.

Current Behavior

1 - Once calling myLinkedProperty.UploadStreamAsync() after using S3Link.Create() with a localstack-configured instance of DynamoDBContext, the following exception is ultimately thrown:

Amazon.S3.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records.
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.InvokeAsync[T](IExecutionContext executionContext) in /_/sdk/src/Handlers/AwsSdk/Internal/XRayPipelineHandler.cs:line 699
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Transfer.Internal.SimpleUploadCommand.ExecuteAsync(CancellationToken cancellationToken)
   at UD.Application.Vedrock.Persistence.Entities.Converse.Command.InsertConverseEntityCommand.Handler.CreateStorageLink(Message[] messages, ConverseEntity entity, CancellationToken cancellationToken) in D:\git\verification\apps\vedrock-service\src\UD.Application.Vedrock\Persistence\Entities\Converse\Command\InsertConverseEntityCommand.cs:line 75

The above happens regardless of manual AmazonDynamoDBClient instance creation with:

serviceCollection
    .AddSingleton<IAmazonDynamoDB>(_ =>
        new AmazonDynamoDBClient(new BasicAWSCredentials("localstack", "localstack"),
            new AmazonDynamoDBConfig
            {
                ServiceURL = "http://localhost:4566/",
                AuthenticationRegion = "eu-west-1"
            }));
            
serviceCollection
       .AddScoped<IDynamoDBContext>(provider =>
           new DynamoDBContext(
               provider.GetRequiredService<IAmazonDynamoDB>(),
               new DynamoDBContextConfig
               {
                   Conversion = DynamoDBEntryConversion.V2,
                   RetrieveDateTimeInUtc = true,
                   ConsistentRead = true,
                   IsEmptyStringValueEnabled = true
               }
           ));          

2 - There appears to be no specific configuration options or opportunities to pass my own configured AmazonS3Client to S3Link

Reproduction Steps

Minimal reproduction via .NET 8 Console App:

using Amazon;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.Runtime;
using SomeNamespace;

const string profile = "localstack";
const string serviceUrl = "http://localhost:4566/";
const string authenticationRegion = "eu-west-1";

var localstackCredentials = new BasicAWSCredentials("localstack", "localstack");
var dynamoDbConfig = new AmazonDynamoDBConfig
{
    Profile = new Profile(profile),
    ServiceURL = serviceUrl,
    AuthenticationRegion = authenticationRegion
};

// There is no opportunity to configure any facets of `S3Link` during the client or context construction, or to pass in your own `AmazonS3Client`
var dynamoDbClient = new AmazonDynamoDBClient(localstackCredentials, dynamoDbConfig);
var dynamoDbContext = new DynamoDBContext(dynamoDbClient,
    new DynamoDBContextConfig
    {
        Conversion = DynamoDBEntryConversion.V2,
        RetrieveDateTimeInUtc = true,
        ConsistentRead = true,
        IsEmptyStringValueEnabled = true
    });

var item = new SomeClass
{
    // There is no opportunity to configure the `AmazonS3Client` here, or to pass in your own
    Prop = S3Link.Create(dynamoDbContext, "bucket", "key", RegionEndpoint.EUWest1)
};

// This then throws "AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records."
await item.Prop.UploadStreamAsync(new MemoryStream("Hello World"u8.ToArray()));

namespace SomeNamespace
{
    public class SomeClass
    {
        public required S3Link Prop { get; set; }
    }
}

csproj:

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net8.0</TargetFramework>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="AWSSDK.DynamoDBv2" Version="3.7.400.21" />
        <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.301" />
        <PackageReference Include="AWSSDK.S3" Version="3.7.403" />
    </ItemGroup>

</Project>

docker-compose.yml:

services:
  localstack:
    image: localstack/localstack:3.7.2
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - DEBUG=${DEBUG:-0}
      - DEFAULT_REGION=eu-west-1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

Possible Solution

From attempting to step through the AWS SDK call stack after attempting to configure for localstack:

• S3Link CreatClientCacheFromContext(DynamoDBContext context) (typo here btw) did not seem to add the created new S3ClientCache to the S3Link.Caches dictionary despite not throwing an exception. This seemed to cause S3ClientCache GetClient to always fall in to if (!this.clientsByRegion.TryGetValue(region.SystemName, out output)) when S3Link operations are performed, where ServiceClientHelpers.CreateServiceFromAssembly() is then used.

• AmazonServiceClient CloneConfig(ClientConfig newConfig), which looks to construct the configuration for the AmazonS3Client created under the hood by S3Link, does not seem to respect client configuration which is necessary for localstack (e.g. maintaining the "localstack" accessKey and secreyKey, maintaining the localstack serviceUrl over using an AWS Region and allowing the necessary setting of ForcePathStyle = true for integration testing)

I believe offering the ability to either configure the created AmazonS3Client via S3Link or to provide your own configured instance of AmazonS3Client to S3Link would allow for end-to-end testing via localstack as expected.

Additional Information/Context

appsettings.Development.json read in as config for my specific code experiencing this issue:

{
    "AWS": {
        "ServiceURL": "http://localhost:4566/",
        "AuthenticationRegion": "eu-west-1",
        "Profile": "localstack"
    },
    "Logging": {
        "LogLevel": {
            "Default": "Information",
            "Microsoft.AspNetCore": "Warning"
        }
    }
}

and my service registrations (called in order):

    private static IServiceCollection ConfigureDelivery(this IServiceCollection serviceCollection, IConfiguration config)
    {
        serviceCollection.AddAWSService<IAmazonSQS>();

        if (string.Equals(Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase) ||
            string.Equals(Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase))
        {
            Console.WriteLine("Configuring S3 for 'Development'");

            serviceCollection
                .AddSingleton<IAmazonS3>(_ => new AmazonS3Client(new AmazonS3Config
                {
                    AuthenticationRegion = config.GetValue<string>("AWS:AuthenticationRegion"),
                    ServiceURL = config.GetValue<string>("AWS:ServiceURL"),
                    ForcePathStyle = true
                }));
        }
        else
        {
            serviceCollection.AddAWSService<IAmazonS3>();
        }

        serviceCollection.AddSingleton<AmazonSQSExtendedClient>(provider =>
        {
            var commonConfig = provider.GetRequiredService<IOptions<CommonConfig>>().Value;

            return new AmazonSQSExtendedClient(
                provider.GetRequiredService<IAmazonSQS>(),
                new ExtendedClientConfiguration()
                    .WithLargePayloadSupportEnabled(provider.GetRequiredService<IAmazonS3>(), commonConfig.BucketName)
                    .WithS3KeyProvider(new PrefixedGuidS3KeyProvider())
            );
        });

        return serviceCollection;
    }

    private static IServiceCollection ConfigurePersistence(this IServiceCollection serviceCollection)
    {
        if (string.Equals(Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase) ||
            string.Equals(Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase))
        {
            serviceCollection
                .AddSingleton<IAmazonDynamoDB>(_ =>
                    new AmazonDynamoDBClient(new BasicAWSCredentials("localstack", "localstack"),
                        new AmazonDynamoDBConfig
                        {
                            ServiceURL = "http://localhost:4566/",
                            AuthenticationRegion = "eu-west-1"
                        }));
        }
        else
        {
            serviceCollection
                .AddAWSService<IAmazonDynamoDB>();
        }

        serviceCollection
               .AddScoped<IDynamoDBContext>(provider =>
                   new DynamoDBContext(
                       provider.GetRequiredService<IAmazonDynamoDB>(),
                       new DynamoDBContextConfig
                       {
                           Conversion = DynamoDBEntryConversion.V2,
                           RetrieveDateTimeInUtc = true,
                           ConsistentRead = true,
                           IsEmptyStringValueEnabled = true
                       }
                   ))
               .AddKeyedSingleton<IDataRepository, DynamoDataRepository>(DataEngine.DynamoDb);

        return serviceCollection;
    }

NB - I have confirmed that it is indeed the Development routes which are invoked as part of my end-to-end testing

AWS .NET SDK and/or Package version used

AWSSDK.DynamoDBv2 3.7.400.21
AWSSDK.Extensions.NETCore.Setup 3.7.301
AWSSDK.S3 3.7.403

Targeted .NET Platform

.NET 8

Operating System and version

Windows 11 (with WSL)

[peterrsongg]

@Danny-UKDM We added a new configuration option called service-specific endpoints where you can set an environment variable or a profile for a specific service and that will set the serviceURL for that service. Can you try adding this in your application?

AWS_ENDPOINT_URL_S3 = "http://localhost:4566/";