Issue upgrading 1.45.2 -> 1.46.0

[dhallatmaine]

I have the following docker-compose section:

  mockgcs:
    image: fsouza/fake-gcs-server:1.45.2
    ports:
      - "4443:4443"
    volumes:
      - ./mockgcs/buckets:/storage
    command: -scheme http -public-host localhost
    deploy:
      resources:
        limits:
          memory: 128M

I have a folder located in ./mockgcs/buckets/ called bucket-name.

I can fetch the bucket objects by visiting: http://localhost:4443/storage/v1/b/bucket-name/o

When I change the version 1.46.0. I can no longer view this bucket url or post to it, I am given the following response:

{"error":{"code":404,"message":"Not Found","errors":null}}

[fsouza]

@dhallatmaine this may be due to the introduction of bucket metadata. The format of the data in the disk is not guaranteed to work between upgrades. Can you try reseeding your server?

[fsouza]

I gave this another look and I don't think this is related to the bucket metadata change, as that didn't touch the listObjects endpoint. I tried to reproduce it and couldn't. Can you provide a complete reproducer?

Here's what I tried:

# terminal A
％ cd $(mktemp -d)
％ docker run -it -v ${PWD}:/storage -v ${fake_gcs_repo}/examples/data:/data -p 8080:8080 fsouza/fake-gcs-server:1.45.2 -data /data -scheme http -port 8080
INFO[0000] server started at http://0.0.0.0:8080
INFO[0004] 172.17.0.1 - - [01/Aug/2023:02:05:42 +0000] "GET /storage/v1/b/sample-bucket/o HTTP/1.1" 200 538
^C%
％ ls
sample-bucket
％ tree
.
└── sample-bucket
    ├── some_file.txt
    └── some_file.txt.metadata

2 directories, 2 files

# terminal B
％ curl -s localhost:8080/storage/v1/b/sample-bucket/o
{"kind":"storage#objects","items":[{"kind":"storage#object","name":"some_file.txt","id":"sample-bucket/some_file.txt","bucket":"sample-bucket","size":"33","contentType":"text/plain; charset=utf-8","crc32c":"4dDntg==","acl":[{"bucket":"sample-bucket","entity":"projectOwner-test-project","object":"some_file.txt","projectTeam":{},"role":"OWNER"}],"md5Hash":"DN5Abn4cTJdjZc1ldLIJqQ==","etag":"\"DN5Abn4cTJdjZc1ldLIJqQ==\"","timeCreated":"2023-08-01T02:05:38.7067Z","updated":"2023-08-01T02:05:38.706726Z","generation":"1690855538706736"}]}

Then restarted with 1.46.0 without setting -data:

# terminal A
％ docker run -it -v ${PWD}:/storage -p 8080:8080 fsouza/fake-gcs-server:1.46.0 -scheme http -port 8080
INFO[0000] server started at http://0.0.0.0:8080
INFO[0001] 172.17.0.1 - - [01/Aug/2023:02:07:16 +0000] "GET /storage/v1/b/sample-bucket/o HTTP/1.1" 200 538
％ ls
sample-bucket
％ tree
.
└── sample-bucket
    ├── some_file.txt
    └── some_file.txt.metadata

2 directories, 2 files

# terminal B
％ curl -s localhost:8080/storage/v1/b/sample-bucket/o
{"kind":"storage#objects","items":[{"kind":"storage#object","name":"some_file.txt","id":"sample-bucket/some_file.txt","bucket":"sample-bucket","size":"33","contentType":"text/plain; charset=utf-8","crc32c":"4dDntg==","acl":[{"bucket":"sample-bucket","entity":"projectOwner-test-project","object":"some_file.txt","projectTeam":{},"role":"OWNER"}],"md5Hash":"DN5Abn4cTJdjZc1ldLIJqQ==","etag":"\"DN5Abn4cTJdjZc1ldLIJqQ==\"","timeCreated":"2023-08-01T02:05:38.7067Z","updated":"2023-08-01T02:05:38.706726Z","generation":"1690855538706736"}]}

[sowmyav24]

I have been getting similar issue when I use

 mockgcs:
    image: fsouza/fake-gcs-server
    ports:
      - "4443:4443"

The error received in the spring application which tries to upload the file in the bucket is

com.google.cloud.storage.StorageException
 Caused by: java.io.IOException

However, when I use the version 1.45.2, it works fine

[raz-amir]

FYI, I have submited PR #1272 to revert the default port change that was caused by another PR I added #1215

[fsouza]

@sowmyav24 I think your issue is different. That's caused by a regression in 1.47.0, which @ramir-savvy fixed above (I'll release 1.47.1 shortly).

@dhallatmaine for the 1.45.2 -> 1.46.0 issue, let me know if you can provide a full reproducer or if I missed anything in my process.

[dhallatmaine]

@fsouza I will look into this more today. Currently I get the 404 from almost every version when visiting http://localhost:4443/storage/v1/b/bucket-name/o. With 1.45.2, my spring boot code that posts files to this bucket works, when I upgrade to 1.46.0, I get 404's for the posting.

My workflow process is such that I prune my docker system, volumes, and images before every run.

I almost wonder if @sowmyav24 and I are having the same issue and maybe I misdiagnosed the symptom. I apologize as I know this isn't much to go on now, when I have more time I will dig deeper into this.

1.46.0:

2023-08-01 11:51:00 time="2023-08-01T15:51:00Z" level=info msg="192.168.32.1 - - [01/Aug/2023:15:51:00 +0000] \"POST /upload/storage/v1/b/bucket-name/o?projection=full&uploadType=multipart HTTP/1.1\" 404 59"
2023-08-01 11:51:10 time="2023-08-01T15:51:10Z" level=info msg="192.168.32.1 - - [01/Aug/2023:15:51:10 +0000] \"POST /upload/storage/v1/b/bucket-name/o?projection=full&uploadType=multipart HTTP/1.1\" 404 59"
2023-08-01 11:51:23 time="2023-08-01T15:51:23Z" level=info msg="192.168.32.1 - - [01/Aug/2023:15:51:23 +0000] \"POST /upload/storage/v1/b/bucket-name/o?projection=full&uploadType=multipart HTTP/1.1\" 404 

1.45.2:

2023-08-01 11:56:22 time="2023-08-01T15:56:22Z" level=info msg="192.168.32.1 - - [01/Aug/2023:15:56:22 +0000] \"POST /upload/storage/v1/b/bucket-name/o?projection=full&uploadType=multipart HTTP/1.1\" 200 609"
2023-08-01 11:56:22 time="2023-08-01T15:56:22Z" level=info msg="192.168.32.1 - - [01/Aug/2023:15:56:22 +0000] \"POST /upload/storage/v1/b/bucket-name/o?projection=full&uploadType=multipart HTTP/1.1\" 200 609"
2023-08-01 11:56:22 time="2023-08-01T15:56:22Z" level=info msg="192.168.32.1 - - [01/Aug/2023:15:56:22 +0000] \"POST /upload/storage/v1/b/bucket-name/o?projection=full&uploadType=multipart HTTP/1.1\" 200 609"

[fsouza]

Oh, it's the /upload URL, yeah that's affected by the bucket metadata change. I've sent PR #1274 that makes the server compatible with the directory structure in 1.45.2. Let me know if that fixes the issue for you!

[fsouza]

I'm merging #1274 after confirming it fixes the issue. Validation steps:

1. create the directory structure on 1.45.2

％ cd $(mktemp -d)
％ docker run -it -v ${PWD}:/storage -v ${HOME}/Projects/os/p/fake-gcs-server/examples/data:/data -p 8080:8080 fsouza/fake-gcs-server:1.45.2 -data /data -scheme http -port 8080
INFO[0000] server started at http://0.0.0.0:8080
^C%
％ tree
.
└── sample-bucket
    ├── some_file.txt
    └── some_file.txt.metadata

2 directories, 2 files

2. reproduce the issue on fake-gcs-server 1.47.1

％ docker run -it -v ${PWD}:/storage -p 8080:8080 fsouza/fake-gcs-server:1.47.1 -scheme http -port 8080
Unable to find image 'fsouza/fake-gcs-server:1.47.1' locally
1.47.1: Pulling from fsouza/fake-gcs-server
8c6d1654570f: Already exists
8534ce12c14c: Already exists
fd438bd3e406: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:9934d11b52f51eaf37e108954aab67d2ad22548c6960cd533e37c5e374d8c2d6
Status: Downloaded newer image for fsouza/fake-gcs-server:1.47.1
INFO[0000] server started at http://0.0.0.0:8080
INFO[0003] 172.17.0.1 - - [01/Aug/2023:17:22:31 +0000] "GET /storage/v1/b/sample-bucket/o HTTP/1.1" 200 540
INFO[0036] 172.17.0.1 - - [01/Aug/2023:17:23:03 +0000] "POST /upload/storage/v1/b/sample-bucket/o?projection=full&uploadType=multipart HTTP/1.1" 404 59

# in another terminal
％ curl -XPOST 'localhost:8080/upload/storage/v1/b/sample-bucket/o?projection=full&uploadType=multipart'
{"error":{"code":404,"message":"Not Found","errors":null}}

3. confirm that backend/fs: return empty BucketAttrs if the metadata file is missing #1274 fixes the issue:

％ docker run -it -v ${PWD}:/storage -p 8080:8080 fsouza/fake-gcs-server:pr-1274 -scheme http -port 8080
INFO[0000] server started at http://0.0.0.0:8080
INFO[0002] 172.17.0.1 - - [01/Aug/2023:17:24:38 +0000] "POST /upload/storage/v1/b/sample-bucket/o?projection=full&uploadType=multipart HTTP/1.1" 400 77

# another shell
％ curl -XPOST -H 'content-type:application/json;boundary=abc' --data-binary '' 'localhost:8080/upload/storage/v1/b/sample-bucket/o?projection=full&uploadType=multipart'
{"error":{"code":500,"message":"multipart: NextPart: EOF","errors":null}}

(note: a 500 is ok as I just want to make sure that the upload endpoint can find the bucket and start processing the requst)