S3 commands hang after uploading an empty stream using `@aws-sdk/lib-storage`

[martinslota]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Immediately ending the stream feeding the Upload utility from @aws-sdk/lib-storage sometimes leads to the S3 client getting "locked up": a subsequent command sent through it never finishes.

SDK version number

@aws-sdk/client-s3@3.637.0, @aws-sdk/lib-storage@3.637.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.15.0

Reproduction Steps

Detailed steps to reproduce are listed in https://github.com/martinslota/aws-sdk-lib-storage-race. Those steps run against Localstack, but the same code also works against actual AWS S3.

Observed Behavior

After uploading an empty stream, commands sent through the S3 client never finish.

Expected Behavior

After uploading an empty stream, commands sent through the S3 client should continue to work as usual.

Possible Solution

No response

Additional Information/Context

No response

[aBurmeseDev]

Hi @martinslota - thanks for reaching out.

Before proceeding, I'd like to request your assistance in confirming that the issue persists when running the code against the actual AWS service as the example you shared appears to be using a LocalStack endpoint (https://s3.localhost.localstack.cloud:4566). While LocalStack is a great tool for local development and testing, it's always a good practice to validate the behavior against the real AWS service before raising a formal issue.

Additionally, to better understand the underlying interactions between your code and the AWS SDK, please add middlewareStack to S3 client in your code to get request logs. This middleware will log the context, request input and response output for each AWS SDK operation, providing valuable insights into the data being sent and received.

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

Looking forward to your response!

[martinslota]

Hi @martinslota - thanks for reaching out.

👋

Before proceeding, I'd like to request your assistance in confirming that the issue persists when running the code against the actual AWS service as the example you shared appears to be using a LocalStack endpoint (https://s3.localhost.localstack.cloud:4566). While LocalStack is a great tool for local development and testing, it's always a good practice to validate the behavior against the real AWS service before raising a formal issue.

Yes, the same happens against the actual AWS S3 service. I also adapted https://github.com/martinslota/aws-sdk-lib-storage-race so it runs against S3 by default.

Additionally, to better understand the underlying interactions between your code and the AWS SDK, please add middlewareStack to S3 client in your code to get request logs. This middleware will log the context, request input and response output for each AWS SDK operation, providing valuable insights into the data being sent and received.

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

Looking forward to your response!

I added the code above but additionally redacted the Body field from response output. The output looks like this.

Previously I also tried with an hour-long timeout instead of 5 seconds and I still saw the script fail.

[fenos]

I'm experiencing this as well

[fenos]

Hey @aBurmeseDev sorry for the direct ping, do you know where the can problem might be? I can have a look, currently this issue is making requests randomly hang around 1% with high traffic

I have run the reproduction repo that @martinslota kindly provided and indeed it also happens locally after around 5k uploads in my case I tested with MinIO so it's definitely a client side issue

This happens also with the AWS S3 Service

[aBurmeseDev]

Hi @fenos - this's interesting! I haven't been able to reproduce it on my end with S3 itself but haven't tested with MinIO. I'll have to dig deeper into it. If there's any other specific case or pattern that I need to be aware of, please do share.

[fenos]

Yeah, it is really random, I don't really have a reliable way of reproducing it yet
Sometimes the done promise is never resolved from the Upload class, no requests are sent by the SDK for uploading.

I have otel telemetry on the aws SDK and added custom span in the uploader, and it simply hangs on the done without even sending any request. I have an idle timeout of 60 seconds that kicks in which then closes the request socket.

I can confirm that it is not something to do with the default agent option maxSockets of 50, since I've increased that number to 700 and instrumented the Agent. Free sockets stay at 60-75 whereas busy sockets are 20-30 at any given time.

I have tried with a couple of different setups for the Agent, using native and others. Currently, I'm using agentkeepalive which simply extends the native Agent with some better connection handling.

in my app I pretty much pass the request body as a stream to the Uploader

const s3Client = new S3Client({
   ...credentials,
   requestHandler: new NodeHandler({
      httpAgent: new Agent({
        maxSockets: 700,
        freeSocketTimeout: 15000
  })
})

fastify.post('/upload', async (req, reply) => {
    const upload = new Upload({
        client: s3Clients,
        params: {
            ...bucketAndKey,
            // body as Readable stream http.IncomingMessage
            Body: req.raw
        }
    })

   await upload.done()

   return reply.send('OK')
})

Files are particularly small ~1MB.
As soon as I get more findings I'll share, currently adding as much telemetry as possible

This issue only occurs with the Uploader

[aBurmeseDev]

Appreciate the details. Please do share once you have more findings while I further look into it on my end.