s3scanner returns not_exist for everything on most default conditions

[fededamian]

Existing Issues

• I've searched the current issues

Command

docker run --rm ghcr.io/sa7mon/s3scanner -bucket s3scanner-bucketsize

Observed behavior (what happened?)

I am trying to run this tool, something I remember succeeded in before, but now, no matter what I do, I get the not_exist output. Either something is broken or I must be doing something very dumb but even in that case it may be helpful to have a couple command examples in the readme guaranteed to succeed to test the behavior.

I am using the docker image to make sure the infrastructure is not the culprit. See the output:

root@ip:~# docker run --rm ghcr.io/sa7mon/s3scanner -version
v3.0.4

root@ip:~# docker run --rm ghcr.io/sa7mon/s3scanner -bucket s3scanner-bucketsize
level=info msg="not_exist | s3scanner-bucketsize"

root@ip:~# docker run --rm ghcr.io/sa7mon/s3scanner -verbose -bucket s3scanner-bucketsize
level=debug msg="bucket not found" bucket_name=s3scanner-bucketsize method="aws.BucketExists()" region=
level=info msg="not_exist | s3scanner-bucketsize"

I know the testing buckets work fine because I can open and get the right response (https://s3scanner-bucketsize.s3.amazonaws.com/).

Am I doing something wrong? Or did something change that made the tool break these last days?

Thanks and apologizes for the inconvenience if this is on me.

Expected behaviour

Getting the exists output.

Debug output

docker run --rm ghcr.io/sa7mon/s3scanner -verbose -bucket s3scanner-bucketsize

level=debug msg="bucket not found" bucket_name=s3scanner-bucketsize method="aws.BucketExists()" region=
level=info msg="not_exist | s3scanner-bucketsize"

OS Info

Docker Image maintained by the project

Config file

No response

Additional info

No response

[sa7mon]

I was not able to reproduce the issue

# docker run --rm ghcr.io/sa7mon/s3scanner -bucket s3scanner-bucketsize
Unable to find image 'ghcr.io/sa7mon/s3scanner:latest' locally
latest: Pulling from sa7mon/s3scanner
9fda8d8052c6: Already exists
fb64270fff90: Pull complete
d6b51273a4f6: Pull complete
Digest: sha256:3fe783f409f3b44b748f8f7a69155b813681f3c2cffaf0ecd4e792c64401253e
Status: Downloaded newer image for ghcr.io/sa7mon/s3scanner:latest
level=info msg="exists    | s3scanner-bucketsize | us-east-1 | AuthUsers: [] | AllUsers: [READ]"
# docker run --rm ghcr.io/sa7mon/s3scanner -version                                                                                      
v3.0.4

I suspect the IP address you are running the container from is being blocked by AWS, but that's a complete guess. Is this a VPS? Or an IP that also hosts a Tor relay?

[fededamian]

Thanks a lot for your answer sa7mon. I ran the docker command on my laptop and it works fine.
The previous test I did, for which I found the issue, was in an AWS EC2 instance with Ubuntu 24.04, nothing else, so no TOR.
The same behavior occurs both running the go binary directly in the Ubuntu box and running the binary in the docker container.
I don't think it is a connectivity issue, if I curl the S3 URL I get the right response.

root@ip:~# docker run --rm ghcr.io/sa7mon/s3scanner -bucket s3scanner-bucketsize
level=info msg="not_exist | s3scanner-bucketsize"

root@ip:~# curl https://s3scanner-bucketsize.s3.amazonaws.com
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>s3scanner-bucketsize</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>test-file.txt</Key><LastModified>2019-09-16T23:37:22.000Z</LastModified><ETag>&quot;cbe923fd5deb54c85287df84fe028710&quot;</ETag><Size>43</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>root@ip-172-31-9-19:~#

root@ip:~# cat /etc/issue
Ubuntu 24.04 LTS \n \l

I am running other recon tools and they seem to work fine, so I don't think there is some sort of IP blocking on the egress point.

Any other ideas on what I can look for?

[sa7mon]

I'm not sure what's causing your issue. If you'd like to debug further and see the actual HTTP requests and responses, you can setup the development environment as explained in this section of the README.

[fededamian]

It's a very curious behavior. I ran the command in the EC2 server earlier today and got many successful results. After repeating everything exactly the same, I got nothing, all not_exist. I used MITM proxy and extracted the curl command. Then ran it in the EC2 and in my laptop:

AWS EC2 Run

curl -I -H 'User-Agent: aws-sdk-go-v2/1.27.0 os/linux lang/go#1.19.13 md/GOOS#linux md/GOARCH#amd64 api/s3#1.54.2' --compressed -H 'Amz-Sdk-Invocation-Id: edc28420-e27d-473b-9892-2fb3d3baf28f' -H 'Amz-Sdk-Request: attempt=1; max=3' -H 'X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' -X HEAD https://s3scanner-bucketsize.s3.us-east-1.amazonaws.com/

HTTP/1.1 404 Not Found
x-amz-request-id: R22WEAJX5WPA82RP
x-amz-id-2: MdATZv1FO5HCgs+OSRz/wV1nad+HKedtrSfK051X6QfWv4wTyHugF7j4VBEyJKxgcqEBmaPgBb4=
Content-Type: application/xml
Date: Thu, 13 Jun 2024 18:13:15 GMT
Server: AmazonS3

Laptop Run

curl -I -H 'User-Agent: aws-sdk-go-v2/1.27.0 os/linux lang/go#1.19.13 md/GOOS#linux md/GOARCH#amd64 api/s3#1.54.2' --compressed -H 'Amz-Sdk-Invocation-Id: edc28420-e27d-473b-9892-2fb3d3baf28f' -H 'Amz-Sdk-Request: attempt=1; max=3' -H 'X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' -X HEAD https://s3scanner-bucketsize.s3.us-east-1.amazonaws.com/

HTTP/1.1 200 OK
x-amz-id-2: OzEzx0vs7yUpjXV4tFK6B4yqYwvTGKqYFlUU4YBxogFh+/wjCdPFOf2+8jBB4CjtpCc3ou4GiWI=
x-amz-request-id: VM9MN0992V3ZHXPT
Date: Thu, 13 Jun 2024 18:12:31 GMT
x-amz-bucket-region: us-east-1
x-amz-access-point-alias: false
Content-Type: application/xml
Server: AmazonS3

The result from the EC2 machine is receiving a 404 Not found, but if I run the exact same curl from my laptop and home connection I get a 200 OK.

Do you think AWS may be detecting this in some way and be blocking it with the Not Found?

[sa7mon]

How strange. I just ran your curl command from my home IP and it worked fine (200 OK). I'm really not sure why AWS is behaving this way, but it's out of my hands at this point.

[m4ra7h0n]

Same problem.

[sa7mon]

Since this discussion began, I have also observed this behavior with AWS specifically. What I believe is happening is that AWS will not send a 429 if you have checked too many buckets in a given timeframe. It will just start saying that all buckets do not exist.

The only remedy for this that I have found is to switch to another IP address. Let me know if you find a way around this.