upload_from_file() with content type "application/json" not working

[tal-sim]

According to the method documentation of gcp_storage_emulator.storage.Storage.add_to_resumable_upload it expects parameter content to be of type bytes, but it is actually passed not as bytes (in my example, it is passed as list), because request.data (which is passed as parameter content) returns json.loads() when content_type is application_json.

1. Start the emulator

docker run  \
  -e PORT=9023 \
  -p 9023:9023 \
  --name gcp-storage-emulator-outtest \
  -v "$(pwd)/cloudstorage":/storage \
  oittaa/gcp-storage-emulator

2. Given file data.json with content:
   [{"a": 1}]

3. Try to upload the file (using Postman for example):

import uvicorn
from fastapi import FastAPI, UploadFile, File

import os

from google.cloud import exceptions, storage

HOST = "localhost"
PORT = 9023
BUCKET = "test-bucket"

os.environ["STORAGE_EMULATOR_HOST"] = f"http://{HOST}:{PORT}"
client = storage.Client(project="TEST")

try:
    bucket = client.create_bucket(BUCKET)
except exceptions.Conflict:
    bucket = client.bucket(BUCKET)

#####


app = FastAPI()


@app.post("/upload")
async def upload_json(file: UploadFile = File(...)):
    blob_bucket = bucket.blob("test_blob")
    blob_bucket.upload_from_file(file.file, content_type=file.content_type)  # this fails, and at this point file.content_type == "application/json"
 

if __name__ == '__main__':
    uvicorn.run("main:app", debug=True, reload=True)

4. The error I get on the emulator is:

Resource not found:
resource '355470393de4cb2d1491b7e5a7a4e47375563fb727c56ea585a2af657d5aa8ec' not found
An error has occurred while running the handler for PUT http://0.0.0.0:9023/upload/storage/v1/b/test-bucket/o?uploadType=resumable&upload_id=test-bucket%3Atest_blob%3A2022-04-09+08%3A08%3A08.176611
can't concat list to bytes
----------------------------------------
Exception occurred during processing of request from ('172.17.0.1', 59828)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python3.10/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 351, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/http/server.py", line 427, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.10/http/server.py", line 415, in handle_one_request
    method()
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 367, in do_PUT
    router.handle(PUT)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 337, in handle
    raise e
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 328, in handle
    handler(request, response, self._request_handler.storage)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/handlers/objects.py", line 324, in upload_partial
    data = storage.add_to_resumable_upload(upload_id, request.data, total_size)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/storage.py", line 249, in add_to_resumable_upload
    file_content += content
TypeError: can't concat list to bytes
----------------------------------------
Resource not found:
resource '355470393de4cb2d1491b7e5a7a4e47375563fb727c56ea585a2af657d5aa8ec' not found
An error has occurred while running the handler for PUT http://0.0.0.0:9023/upload/storage/v1/b/test-bucket/o?uploadType=resumable&upload_id=test-bucket%3Atest_blob%3A2022-04-09+08%3A08%3A08.176611
can't concat list to bytes
----------------------------------------
Exception occurred during processing of request from ('172.17.0.1', 59830)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python3.10/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 351, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/http/server.py", line 427, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.10/http/server.py", line 415, in handle_one_request
    method()
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 367, in do_PUT
    router.handle(PUT)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 337, in handle
    raise e
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/server.py", line 328, in handle
    handler(request, response, self._request_handler.storage)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/handlers/objects.py", line 324, in upload_partial
    data = storage.add_to_resumable_upload(upload_id, request.data, total_size)
  File "/usr/local/lib/python3.10/site-packages/gcp_storage_emulator/storage.py", line 249, in add_to_resumable_upload
    file_content += content
TypeError: can't concat list to bytes

[oittaa]

Thanks for the detailed report. I was also able to reproduce it without Docker:

import os

from google.cloud import storage
from gcp_storage_emulator.server import create_server

HOST = "localhost"
PORT = 9023
BUCKET = "test-bucket"

server = create_server(HOST, PORT, in_memory=True, default_bucket=BUCKET)
server.start()

os.environ["STORAGE_EMULATOR_HOST"] = f"http://{HOST}:{PORT}"
client = storage.Client()
bucket = client.bucket(BUCKET)

blob = bucket.blob("data.json")

with open("data.json", "w+b") as my_file:
    my_file.write(b'[{"a": 1}]')
    my_file.seek(0)
    blob.upload_from_file(my_file, content_type="application/json")

server.stop()

[oittaa]

The newest release v2022.04.09 should fix this.

Thanks again for the great report!

[tal-sim]

@oittaa Really awesome work! Thank you!