When deleting the empty bucket, BucketNotEmpty error is returned

[yinfangchen]

Describe the bug

When we use the SDK API DeleteBucketAsync, a BucketNotEmpty error is returned. However, before calling the DeleteBucketAsync, we have already used the DeleteObjectAsync to delete all the objects in the bucket.

For instance, suppose there is a for-loop to create a bucket and then delete it continuously, e.g.:

var bucketName = "xxx-xxx";
for (int i = 0; i < 100; i++) {
    await CreateBucketAsync(amazonS3Client, bucketName); // calls PutBucket
    ... // Some operations to upload objects and then download them
    await DeleteBucketContentsAsync(amazonS3Client, bucketName); // calls DeleteObjectAsync
    await DeleteBucketAsync(amazonS3Client, bucketName); // calls `DeleteBucketAsync` and fails here with BucketNotEmpty error code
}

The failure behavior is not deterministic, but we encountered this situation many times.

Expected Behavior

We are wondering whether reusing the same bucket name causes this problem since we noticed that the previous S3 document said that:

... it might take some time before the name can be reused ...

as shown in this post: https://serverfault.com/questions/554374/how-long-before-an-s3-bucket-can-be-created-with-same-name-after-deletion

However, we cannot find this description anymore, so we expect the same bucket name should be reusable immediately.

Current Behavior

Currently when deleting the content of the bucket, it will throw an AmazonS3Exception with the error message "The bucket you tried to delete is not empty".

The stack trace for the previous example

Unhandled exception. Amazon.S3.AmazonS3Exception: The bucket you tried to delete is not empty
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at AWSTest.Program.DeleteBucketAsync(IAmazonS3 client, String bucketName) 

Reproduction Steps

As described before, use a for-loop to create and delete the bucket continuously.

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.9.31

Targeted .NET Platform

.NET 6

Operating System and version

Windows 10 Pro

[yinfangchen]

I add a 100ms delay before the DeleteBucketAsync and the problem disappears.

Is this problem related to the consistency model of AWS S3?
I think the S3 provides strong read-after-write consistency, and the bucket has the eventual consistency model as mentioned here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel

Amazon S3 provides strong read-after-write consistency for PUT and DELETE requests of objects in your Amazon S3 bucket in all AWS Regions. This behavior applies to both writes to new objects as well as PUT requests that overwrite existing objects and DELETE requests.

Bucket configurations have an eventual consistency model. Specifically, this means that:
If you delete a bucket and immediately list all buckets, the deleted bucket might still appear in the list.

If I understand correctly, the S3 object write operation is guaranteed but the bucket write operation is not. However, my DeleteBucketContentsAsync only calls the DeleteObjectAsync which sends a DELETE request to remove objects.

• Shouldn't this operation follow the read-after-write consistency model, which means the object will become invisible after this delete operation? Why do I still encounter this "The bucket you tried to delete is not empty"?
• Is it because DeleteObjectAsync's following request is a write operation (i.e., DeleteBucketAsync), so the inconsistency still exists?

[ashishdhingra]

Hi @darknesschen,

Good morning.

The behavior documented by you is controlled by the S3 service. Deleting objects could take some time to synchronize. You could implement retry mechanism as per logic in UtilityMethods.DeleteS3BucketWithObjectsAsync().

Thanks,
Ashish

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.