Chunked encoding with S3Client

[lordgamez]

Hi, I am using S3Client to upload files to S3 with PutObjectRequests. I see that the IsChunked() method of PutObjectRequest returns false and it does not use chunked encoding for transfer. If I create my own inherited version of PutObjectRequest where I override IsChunked() to return true I get the error that it is not implemented for this request:

[2023-04-20 14:58:59.834] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [error] [AWSXmlClient] HTTP response code: 501
Resolved remote host IP address: 52.95.142.82
Request ID: V6QD0FA0ZQGVRJ13
Exception name: NotImplemented
Error message: Unable to parse ExceptionName: NotImplemented Message: A header you provided implies functionality that is not implemented
7 response headers:
connection : close
content-type : application/xml
date : Thu, 20 Apr 2023 12:58:59 GMT
server : AmazonS3
transfer-encoding : chunked
x-amz-id-2 : /onVYYyui+huQoLGf4KXIqO9MyGPJYgzK6J35wfCXHEBiuD4q78dlf7eaOcJwH/2fSmvVY93+wE=
x-amz-request-id : V6QD0FA0ZQGVRJ13

In aws-sdk-java I see that the default upload option is chunked encoding, but there is a disableChunkedEncoding method to disable it. I saw a suggestion that this can be achieved in aws-sdk-cpp using multipart uploads, but that seems to different to me.

What is the right way to use chunked encoding for uploading to S3?

[lordgamez]

Is there any information how could chunked encoding be implemented for a PutObjectRequest? Are there any other examples of this for other requests? I would be glad to contribute to this in the codebase. Or if it is already implemented in another way, please correct me.

[jmklix]

I think you might be looking to use TransferManager. It will handle uploading large files in parallel for you. You can find an example for how to use this in the samples repo here. Let me know if you have any questions about this or if this is not what you are looking for.

[lordgamez]

Hi, thanks for the answer. In the meantime I implemented a solution using multipart upload requests. The reason I didn't use the TransferManager is I wanted to save the upload state and manually continue the part uploads in case the application is killed.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.