S3.headBucket doesn't throw NotFound error when bucket doesn't exist

[trivikr]

Describe the bug
S3.headBucket doesn't throw NotFound error when bucket doesn't exist

SDK version number

$ yarn list aws-sdk
└─ aws-sdk@2.693.0
$ yarn list @aws-sdk/client-s3
└─ @aws-sdk/client-s3@1.0.0-gamma.2

Is the issue in the browser/Node.js/ReactNative?
Node.js

Details of the browser/Node.js/ReactNative version

$ node -v
v12.17.0

To Reproduce (observed behavior)

Code

const AWS = require("aws-sdk");
const { S3 } = require("@aws-sdk/client-s3");

(async () => {
  let response;
  const region = "us-west-2";
  const Bucket = `test-bucket-${Math.ceil(Math.random() * 10 ** 10)}`;

  const v2Client = new AWS.S3({ region });
  const v3Client = new S3({ region });

  await v2Client.createBucket({ Bucket }).promise();
  await v2Client.waitFor("bucketExists", { Bucket }).promise();

  response = await v2Client.headBucket({ Bucket }).promise();
  console.log(`v2 headBucket: ${JSON.stringify(response, null, 2)}`);

  response = await v3Client.headBucket({ Bucket });
  console.log(`v3 headBucket: ${JSON.stringify(response, null, 2)}`);

  await v2Client.deleteBucket({ Bucket }).promise();
  await v2Client.waitFor("bucketNotExists", { Bucket }).promise();

  try {
    response = await v2Client.headBucket({ Bucket }).promise();
    console.log(`v2 headBucket: ${JSON.stringify(response, null, 2)}`);
  } catch (e) {
    console.log(`v2 Error:`);
    console.log(e);
  }

  try {
    response = await v3Client.headBucket({ Bucket });
    console.log(`v3 headBucket: ${JSON.stringify(response, null, 2)}`);
  } catch (e) {
    console.log(`v3 Error:`);
    console.log(e);
  }
})();

Details

v2 headBucket: {}
v3 headBucket: {
  "$metadata": {
    "httpStatusCode": 200,
    "httpHeaders": {
      "x-amz-id-2": "WESS+DcnNwmJOUmAUBquXI9U2TcNVliQKk1WPVoAHxFIwvXDVNSdI5Ya2B1MkqdIqwSBH3Wc+Ag=",
      "x-amz-request-id": "94F2FBB046EA4A8F",
      "date": "Wed, 08 Jul 2020 01:31:10 GMT",
      "x-amz-bucket-region": "us-west-2",
      "content-type": "application/xml",
      "transfer-encoding": "chunked",
      "server": "AmazonS3"
    },
    "retries": 0,
    "totalRetryDelay": 0
  }
}
v2 Error:
NotFound: null
    at Request.extractError (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/services/s3.js:811:35)
    at Request.callListeners (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/trivikr/workspace/aws-sdk-js-tests/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'NotFound',
  region: null,
  time: 2020-07-08T01:31:09.928Z,
  requestId: 'D28E236715974621',
  extendedRequestId: 'TKSth2YgdqhB7+Lk9yhI8P1flSVHhWxyiOifXmiLxAUFu8hnl7ZJc27+n3e++i/KenqTOla6570=',
  cfId: undefined,
  statusCode: 404,
  retryable: false,
  retryDelay: 25.782346166635396
}
v3 headBucket: {
  "$metadata": {
    "httpStatusCode": 200,
    "httpHeaders": {
      "x-amz-id-2": "WrOLC0sgp6EglKp1oTg5zBj2VUzT6u3AbgV1pt0n1dB/e4VDLCnNn+d+Ak0qoZPgQ1EuhZXKIl4=",
      "x-amz-request-id": "B38EE1F02316A32D",
      "date": "Wed, 08 Jul 2020 01:31:10 GMT",
      "x-amz-bucket-region": "us-west-2",
      "content-type": "application/xml",
      "transfer-encoding": "chunked",
      "server": "AmazonS3"
    },
    "retries": 0,
    "totalRetryDelay": 0
  }
}

Expected behavior
S3.headBucket should throw error when bucket doesn't exist

[alexforsyth]

Okay so interesting findings. I think this may no longer be a GA blocker, but is still an issue pending investigation.

Region is defaulted to us-west-2

import { S3 } from "@aws-sdk/client-s3";
(async () => {
    let s3Client = new S3({});
    let headBucketParams: HeadBucketRequest = {
        Bucket: "bucket-does-not-exist"
    }
    try {
        let result = await s3Client.headBucket(headBucketParams);    
        console.log("result", result);
    } catch (err){
        console.log(err);
    }
})();

I received a 200 response back

{ '$metadata':
   { httpStatusCode: 200,
     httpHeaders:
      { 'x-amz-id-2':
         '55As9QVGgwofkJ9MCu+yu5jlBU9jpfXX5Te+scb2RxHVttAnJj4adhWNjfwillzOVcFNogRx3hQ=',
        'x-amz-request-id': '14218166C3A99A07',
        date: 'Thu, 29 Oct 2020 16:34:45 GMT',
        'x-amz-bucket-region': 'us-west-2',
        'content-type': 'application/xml',
        'transfer-encoding': 'chunked',
        server: 'AmazonS3' },
     requestId: undefined,
     attempts: 1,

Then I set the region to us-east-1 (global for s3) and I got a 404 back.

 name: 'NotFound',
  '$fault': 'client',
  '$metadata':
   { httpStatusCode: 404,
     httpHeaders:
      { 'x-amz-request-id': '374BA5B6DF577714',
        'x-amz-id-2':
         'uT8Um7FB5rRC4/thCaLzkdNEVG2xX5tWVXtCm6qNnCCrWCLBWjAo64WGr7Ot1sc1eBsEGlqYDx0=',
        'content-type': 'application/xml',
        'transfer-encoding': 'chunked',
        date: 'Thu, 29 Oct 2020 19:27:00 GMT',
        server: 'AmazonS3' },
     requestId: undefined,
     attempts: 1,
     totalRetryDelay: 0 } }

Then I set the region back to us-west-2 and I continued to get a 404 back. But this is where It gets really weird: ALL buckets that do not exist now return a 404 for me. I can no longer reproduce the error thrown from this code and the sample code posted above is no longer reproducible for me.

[alexforsyth]

Okay so this looks like a caching issue with s3. Not a GA blocker. It looks like the cache gets invalidated on the global endpoint first (us-east-1) and then becomes eventually consistent elsewhere. To make this work immediately force your request to go to us-east-1

[trivikr]

Verified that the HTTPResponse remains the same in v3 after bucket deletion is successful.
This could be because v2 is calling us-east-1 endpoint while calling headBucket, as we use waiter from v2. The waiters are not supported in v3 yet.

    "@aws-sdk/client-s3": "1.0.0-rc.3",
    "aws-sdk": "2.782.0"

Code

const AWS = require("aws-sdk");
const { S3 } = require("@aws-sdk/client-s3");

(async () => {
  let response;
  const region = "us-west-2";
  const Bucket = `test-bucket-${Math.ceil(Math.random() * 10 ** 10)}`;

  const logger = { debug: console.log };
  const v2Client = new AWS.S3({ region });
  const v3Client = new S3({ region, logger });

  await v2Client.createBucket({ Bucket }).promise();
  await v2Client.waitFor("bucketExists", { Bucket }).promise();

  response = await v3Client.headBucket({ Bucket });
  // console.log(`v3 headBucket: ${JSON.stringify(response, null, 2)}`);

  await v2Client.deleteBucket({ Bucket }).promise();
  await v2Client.waitFor("bucketNotExists", { Bucket }).promise();

  // try {
  //   response = await v2Client.headBucket({ Bucket }).promise();
  //   console.log(`v2 headBucket: ${JSON.stringify(response, null, 2)}`);
  // } catch (e) {
  //   console.log(`v2 Error:`);
  //   console.log(e);
  // }

  try {
    response = await v3Client.headBucket({ Bucket });
    // console.log(`v3 headBucket: ${JSON.stringify(response, null, 2)}`);
  } catch (e) {
    // console.log(`v3 Error:`);
    // console.log(e);
  }
})();

Output

{
  httpRequest: HttpRequest {
    method: 'HEAD',
    hostname: 's3.us-west-2.amazonaws.com',
    port: undefined,
    query: {},
    headers: { 'Content-Type': '' },
    body: undefined,
    protocol: 'https:',
    path: '/test-bucket-7155967407'
  }
}
{
  httpResponse: HttpResponse {
    statusCode: 200,
    headers: {
      'x-amz-id-2': 'RsILGp4U4QkPhV9WQ+Ao/FcRkjvCiMnywqtaqYVWASWF+qcEk26Uj/KHPk4X2PbBTDPrHPQ4/2c=',
      'x-amz-request-id': '06562013940ED2EA',
      date: 'Thu, 29 Oct 2020 21:09:11 GMT',
      'x-amz-bucket-region': 'us-west-2',
      'content-type': 'application/xml',
      'transfer-encoding': 'chunked',
      server: 'AmazonS3'
    },
    body: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      headers: [Object],
      rawHeaders: [Array],
      trailers: {},
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 200,
      statusMessage: 'OK',
      client: [TLSSocket],
      _consuming: false,
      _dumped: false,
      req: [ClientRequest],
      [Symbol(kCapture)]: false,
      [Symbol(RequestTimeout)]: undefined
    }
  }
}
{
  httpRequest: HttpRequest {
    method: 'HEAD',
    hostname: 's3.us-west-2.amazonaws.com',
    port: undefined,
    query: {},
    headers: { 'Content-Type': '' },
    body: undefined,
    protocol: 'https:',
    path: '/test-bucket-7155967407'
  }
}
{
  httpResponse: HttpResponse {
    statusCode: 200,
    headers: {
      'x-amz-id-2': 'DT+GGRYtW4ylOxfPxXaQiZTjCJQuqWiNpsXemLm/8v9xTY/MUyabZeQcAE9fTb1KRHP4xmPSdnA=',
      'x-amz-request-id': 'E6465A2789D60155',
      date: 'Thu, 29 Oct 2020 21:09:12 GMT',
      'x-amz-bucket-region': 'us-west-2',
      'content-type': 'application/xml',
      'transfer-encoding': 'chunked',
      server: 'AmazonS3'
    },
    body: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      headers: [Object],
      rawHeaders: [Array],
      trailers: {},
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 200,
      statusMessage: 'OK',
      client: [TLSSocket],
      _consuming: false,
      _dumped: false,
      req: [ClientRequest],
      [Symbol(kCapture)]: false,
      [Symbol(RequestTimeout)]: undefined
    }
  }
}

[AllanZhengYP]

The global s3 client proposal will have fix this issue: #1807. Change this to documentation as we need to provide more detailed explanation of what happens under the hood.

[github-actions]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.