SdkClientException: Unable to execute HTTP request: Only one connection receive subscriber allowed

[nikitsenka]

SdkClientException: Unable to execute HTTP request: Only one connection receive subscriber allowed

We are using java AWS SDK v2 for sending data to s3 using Java Http POST endpoint inside Reactive Spring Boot Webflux application.

Describe the issue

This error occurs 1-3 times in a week and only in production where the load is approx 10-30 RPS. It causes the dataloss as 1-3 HTTP json messages are not delivered. The SDK retry looks like applied for this case.

Steps to Reproduce

The reason of the issue is unknown
Unfortunately it happens only in production and not connected with the user data. Any recommendation how to simulate the issue in local environment will be really helpful.

Your Environment

• AWS Java SDK version used: 2.16.32
• JDK version used: OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.2+12, mixed mode, sharing)
• Operating System and version: Linux ingress-traffic-c5ddf5dd7-qm4vt 4.14.181-142.260.amzn2.x86_64 Add Travis config #1 SMP Wed Jun 24 19:07:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Full log log.txt
Snippet from the log:

426		... 55 common frames omitted
427	Caused by: java.lang.IllegalStateException: Only one connection receive subscriber allowed.
428		... 89 common frames omitted
429		at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:143)
430		at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:194)
431		at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:198)
432		at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
433		at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
434	Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Only one connection receive subscriber allowed.
435		at java.base/java.lang.Thread.run(Thread.java:832)
436		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
437		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
438		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
439		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
440		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
441		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)

Metrics details

10-04-2021 17:19:25.752 trace= span= �[35m[aws-java-sdk-NettyEventLoop-1-7]�[0;39m �[39mDEBUG�[0;39m c.n.e.o.t.c.a.LoggingMetricPublisher.? - aws sdk metrics: MetricCollection(name=ApiCall, metrics=[MetricRecord(metric=MarshallingDuration, value=PT0.000085333S), MetricRecord(metric=RetryCount, value=3), MetricRecord(metric=ApiCallSuccessful, value=false), MetricRecord(metric=OperationName, value=PutObject), MetricRecord(metric=ApiCallDuration, value=PT0.867155059S), MetricRecord(metric=CredentialsFetchDuration, value=PT0.000000895S), MetricRecord(metric=ServiceId, value=S3)], children=[MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=SigningDuration, value=PT0.000059589S), MetricRecord(metric=ServiceCallDuration, value=PT0.481107161S), MetricRecord(metric=AwsExtendedRequestId, value=Qdpz3ZCYk6RFbSYc0SRQu8cjhikHwH78oekWHw65vc8aFR1moZvuwEYP3re31+YohV2H7PubDqc=), MetricRecord(metric=HttpStatusCode, value=500), MetricRecord(metric=BackoffDelayDuration, value=PT0S), MetricRecord(metric=AwsRequestId, value=D41X7XHXFWAMT2KX)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=2), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])]), MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=BackoffDelayDuration, value=PT0.083S), MetricRecord(metric=SigningDuration, value=PT0.000098252S)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=1), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])]), MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=BackoffDelayDuration, value=PT0.194S), MetricRecord(metric=SigningDuration, value=PT0.000170456S)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=1), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])]), MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=BackoffDelayDuration, value=PT0.051S), MetricRecord(metric=SigningDuration, value=PT0.000090042S)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=0), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])])])

[debora-ito]

@nikitsenka a quick search for java.lang.IllegalStateException: Only one connection receive subscriber allowed and the results point to an issue with Spring Webflux:

• On StackOverflow: Spring WebFlux: Only one connection receive subscriber allowed
• On StackOverflow: Spring webclient throws “Only one connection receive subscriber allowed” when response does not contain media type header
• On spring-projects/spring-framework: WebClient throws "Only one connection receive subscriber allowed" when response has content but no Content-Type header [SPR-17482]

Looking at the logs the error is returned with a http status code 500, which the SDK will retry, as you mentioned. This doesn't look like an issue with the SDK itself, but if you'd like us to investigate further please provide a self-contained sample code we can use to reproduce.

[nikitsenka]

@debora-ito thanks for the links. The only helpful is the first one but unfortunately it is a WebClient case and we are using AWS SDK NettyNio client. Please see the code snippet below. It is not compilable but just to show the usage of SDK.

package example;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.config.ConfigurableBeanFactory;
import org.springframework.cloud.sleuth.instrument.async.TraceableExecutorService;
import org.springframework.context.annotation.Scope;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Component;
import org.springframework.util.Assert;
import reactor.core.publisher.Mono;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3AsyncClientBuilder;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.sts.StsClient;
import software.amazon.awssdk.services.sts.StsClientBuilder;
import software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;

import java.net.URI;
import java.time.Clock;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

import static java.lang.String.format;
import static org.apache.commons.lang3.StringUtils.defaultString;
import static org.apache.commons.lang3.StringUtils.isNotBlank;
import static software.amazon.awssdk.core.client.config.SdkAdvancedAsyncClientOption.FUTURE_COMPLETION_EXECUTOR;


/**
 * S3 implementation for sink adapter.
 */
@Component
@Getter
@Slf4j
@NoArgsConstructor
@AllArgsConstructor
@Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
@ToString(onlyExplicitlyIncluded = true, includeFieldNames = false)
public class S3SinkAdapter implements SinkAdapter<S3SinkConfiguration> {

    @ToString.Include
    private S3SinkConfiguration s3Sink;
    private Lock lock = new ReentrantLock(true);
    @Autowired
    private ObjectMapper objectMapper;
    private Map<String, S3AsyncClient> s3AsyncClientMap = new HashMap<>();
    @Autowired
    private TraceableExecutorService traceableExecutorService;
    @Autowired
    private TraceHelper traceHelper;
    @Autowired
    private Clock clock;


    /**
     * {@inheritDoc}
     */
    @Override
    public SinkType getType() {
        return SinkType.S3;
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public SinkAdapter<S3SinkConfiguration> withSink(JsonNode sinkConfiguration) {
        Assert.notNull(sinkConfiguration, "Null S3 sink configuration was provided");
        this.s3Sink = objectMapper.convertValue(sinkConfiguration, S3SinkConfiguration.class);
        return this;
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public S3SinkConfiguration getSink() {
        return this.s3Sink;
    }

    @Override
    public Mono<String> storeData(Payload payload) {
        if (Objects.nonNull(payload.getBytes())) {
            log.info("S3SinkAdapter start json data storing");
            // Aws sdk will throw socket connection timeout error if number of bytes set in the "Content-Length" header
            // is more than the actual file size
            // https://aws.amazon.com/premiumsupport/knowledge-center/s3-socket-connection-timeout-error/
            payload.setContentLength(payload.getBytes().length);
            return storeBody(payload, AsyncRequestBody.fromBytes(payload.getBytes()));
        } else {
            log.info("S3SinkAdapter start raw data storing");
            AsyncRequestBody requestBody = AsyncRequestBody.fromPublisher(payload.getDataBufferFlux().map(DataBuffer::asByteBuffer));
            return storeBody(payload, requestBody);
        }
    }

    private Mono<String> storeBody(Payload payload, AsyncRequestBody requestBody) {
        // We reuse connections for different request so we do not need to close clients
        @SuppressWarnings("PMD.CloseResource")
        S3AsyncClient s3AsyncClient = asyncClient(s3Sink.getRoleArn());
        PutObjectRequest build = PutObjectRequest.builder()
                .bucket(s3Sink.getS3BucketName())
                .contentLength(payload.getContentLength())
                .key(getKey(s3Sink.getFolderPath(),
                        payload.getEventId(),
                        s3Sink.getNamingTimeFormat(),
                        payload.getExtension()))
                .acl(s3Sink.getAcl())
                .build();
        return Mono
                .fromFuture(s3AsyncClient
                        .putObject(build,
                                requestBody
                        )
                )
                .transformDeferred(traceHelper::propagateTraceFromContextToMono)
                .onErrorMap(throwable -> {
                    if (throwable instanceof S3Exception
                            && ((S3Exception) throwable).statusCode() < HttpStatus.BAD_REQUEST.value()) {
                        return new IngestionValidationException("Cannot process request to S3", throwable);
                    }
                    log.debug("s3 client executor details: " + ((ExecutorAwareService) traceableExecutorService).getExecutor());
                    return new SinkIngestionException("S3 ingestion error. EventId - " + payload.getEventId(), throwable);
                })
                .doOnSuccess(s -> log.info("S3SinkAdapter stored data successfully"))
                .thenReturn(payload.getEventId())
                .name("s3sinkadapter_storedata")
                .metrics();
    }

    private S3AsyncClient asyncClient(String roleArn) {
        String key = StringUtils.isBlank(roleArn) ? "default" : roleArn;
        if (s3AsyncClientMap.get(key) == null) {
            lock.lock();
            try {
                if (s3AsyncClientMap.get(key) == null) {
                    log.info("s3 async client configuration start");
                    S3AsyncClientBuilder builder = S3AsyncClient.builder()
                            .credentialsProvider(getAwsCredentialsProvider(roleArn))
                            .asyncConfiguration(configuration -> configuration
                                    .advancedOption(FUTURE_COMPLETION_EXECUTOR, traceableExecutorService)
                                    .build()
                            )
                            .overrideConfiguration(c -> c.addMetricPublisher(LoggingMetricPublisher.builder().build()))
                            .region(getRegion());

                    addEndpointUrl(builder);
                    s3AsyncClientMap.put(key, builder.build());
                }
            } finally {
                lock.unlock();
            }
        }
        log.info("using s3 client with role: " + key);
        return s3AsyncClientMap.get(key);
    }

    private AwsCredentialsProvider getAwsCredentialsProvider(String roleArn) {
        if (StringUtils.isBlank(roleArn)) {
            return DefaultCredentialsProvider.create();
        }
        AssumeRoleRequest assumeRoleRequest = AssumeRoleRequest
                .builder()
                .roleArn(roleArn)
                .roleSessionName(UUID.randomUUID().toString())
                .build();
        StsClientBuilder stsClientBuilder = StsClient.builder()
                .region(getRegion());
        addEndpointUrl(stsClientBuilder);
        return StsAssumeRoleCredentialsProvider.builder()
                .stsClient(stsClientBuilder.build())
                .refreshRequest(assumeRoleRequest)
                .build();
    }

    private void addEndpointUrl(S3AsyncClientBuilder builder) {
        if (isNotBlank(s3Sink.getEndpointUrl())) {
            builder.endpointOverride(URI.create(s3Sink.getEndpointUrl()));
        }
    }

    private void addEndpointUrl(StsClientBuilder builder) {
        if (isNotBlank(s3Sink.getEndpointUrl())) {
            builder.endpointOverride(URI.create(s3Sink.getEndpointUrl()));
        }
    }

    private Region getRegion() {
        return s3Sink.getS3RegionName() != null ? Region.of(s3Sink.getS3RegionName()) : Region.US_EAST_2;
    }

    private String getKey(String folderPath, String eventId, String dateTimeFormat, String extension) {
        return format("%s/%s/%s.%s", defaultString(folderPath), getCurrentTimeFormatted(dateTimeFormat), eventId, extension);
    }

    private String getCurrentTimeFormatted(String dateTimeFormat) {
        return DateTimeFormatter.ofPattern(StringUtils.isEmpty(dateTimeFormat) ? "yyyyMMdd" : dateTimeFormat)
                .format(LocalDateTime.now(clock));
    }

}

Any recommendations will be helpful

[nikitsenka]

We are using s3AsyncClientMap to store client object in memory and reuse them for subsequent requests from the same client. The idea is to not create client every time but reuse existing one to improve performance. Not sure is it really make sense. Maybe we can just create new client for each request. I did't found any example or recommendation regarding it on AWS SDK User guide. Will be great if you can suggest better approach or provide some example for it.

[nikitsenka]

@debora-ito any ideas?

[pkgonan]

I have same issue, any solution?

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connection has been closed
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:179) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[sdk-core-2.17.36.jar!/:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.36.jar!/:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:104) ~[sdk-core-2.17.36.jar!/:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$WrappedErrorForwardingResponseHandler.onError(MakeAsyncHttpRequestStage.java:158) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:300) ~[netty-nio-client-2.17.36.jar!/:na]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$writeRequest$8(NettyRequestExecutor.java:254) ~[netty-nio-client-2.17.36.jar!/:na]
	at io.opentelemetry.javaagent.instrumentation.netty.common.FutureListenerWrappers$WrappedFutureListener.operationComplete(FutureListenerWrappers.java:67) ~[na:na]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: reactor.netty.channel.AbortedException: Connection has been closed
	at reactor.netty.http.server.HttpServerOperations.onInboundClose(HttpServerOperations.java:559) ~[reactor-netty-http-1.0.10.jar!/:1.0.10]
	at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:73) ~[reactor-netty-core-1.0.10.jar!/:1.0.10]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.handler.codec.MessageAggregator.channelInactive(MessageAggregator.java:438) ~[netty-codec-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) ~[netty-codec-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-codec-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	... 1 common frames omitted```

[debora-ito]

@nikitsenka I'm sorry, getting back to this issue after a long time. Are you still seeing the error?

It is recommended to reuse the same client whenever possible, but if you need different client configurations than you need to create different clients. Creating a new client for every request is not recommended. I don't see any obvious problems with your implementation of s3AsyncClientMap. We list some best practices in our Dev Guide and on Github.

Marking this to auto-close soon, but let us know if you're still getting this error.