PUTing a full object to a resumable session does not trigger after instructions

[BenWhitehead]

PUTing a full object to a resumable session does not trigger after instructions such as return-503-after-256K[1].

after instructions appear to apply to the next request[2].

Environment details

• Package version: gcr.io/cloud-devrel-public-resources/storage-testbench:v0.36.0

Steps to reproduce

# start the container
docker run --rm -d --name repro gcr.io/cloud-devrel-public-resources/storage-testbench:v0.36.0
# create a bucket
xh --print=h :9000/storage/v1/b project==project-id name=bucket1
# create the resumable upload session
{ uploadId=$(xh --print=h POST :9000/upload/storage/v1/b/bucket1/o name==object1 uploadType==resumable | tee /dev/fd/3 | grep -i 'Location:' | cut -b11-) ; } 3>&1
# upload 768KiB of data and finalize the session
dd ibs=262144 count=3 if=/dev/urandom | xh --print=Hh PUT "$uploadId" Content-Range:'bytes 0-786431/786432' x-goog-emulator-instructions:return-503-after-256K
# stop the docker container
docker kill repro

[1] https://gist.github.com/BenWhitehead/07b65f51ac6e0f6421d0a7f41a4d5433#file-01_result-txt-L12-L28
[2] https://gist.github.com/BenWhitehead/07b65f51ac6e0f6421d0a7f41a4d5433#file-02_result-txt-L38-L53

[cojenco]

Is this referring to both the x-goog-emulator-instructions and Retry Test API in the testbench?

For the retry test instructions, I haven't had the chance to repro or investigate, but my first guess is that the after instructions are not triggered due to the current logic in handle_retry_test_instruction(). It relies on fetching len(upload.media) from the database in order to dequeue the instructions and force the failure.

storage-testbench/testbench/common.py

Lines 797 to 805 in 38e7cbc

	upload = database.get_upload(upload_id, None)
	if upload is not None and len(upload.media) >= after_bytes:
	database.dequeue_next_instruction(test_id, method)
	testbench.error.generic(
	"Fault injected after uploading %d bytes" % len(upload.media),
	rest_code=error_code,
	grpc_code=StatusCode.INTERNAL, # not really used
	context=None,
	)

[BenWhitehead]

I've observed the same behavior for x-goog-emulator-instructions:return-503-after-256K or for /retry_test