Uploading large files to AWS with aws_s3_bucket_object

[NathanDotTo]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

terraform -v
2021-05-12T09:50:18.204+0200 [DEBUG] Adding temp file log sink: /var/folders/18/h1984hd95ms_l22lvpwzc57r0000gn/T/terraform-log131399222
2021-05-12T09:50:18.204+0200 [INFO]  Terraform version: 0.15.3
2021-05-12T09:50:18.204+0200 [INFO]  Go runtime version: go1.16.3
2021-05-12T09:50:18.204+0200 [INFO]  CLI args: []string{"/usr/local/homebrew/bin/terraform", "-v"}
2021-05-12T09:50:18.205+0200 [TRACE] Stdout is a terminal of width 174
2021-05-12T09:50:18.205+0200 [TRACE] Stderr is a terminal of width 174
2021-05-12T09:50:18.205+0200 [TRACE] Stdin is a terminal
2021-05-12T09:50:18.205+0200 [DEBUG] Attempting to open CLI config file: /Users/nsowatsk/.terraformrc
2021-05-12T09:50:18.205+0200 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021-05-12T09:50:18.207+0200 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-05-12T09:50:18.207+0200 [DEBUG] ignoring non-existing provider search directory /Users/nsowatsk/.terraform.d/plugins
2021-05-12T09:50:18.207+0200 [DEBUG] ignoring non-existing provider search directory /Users/nsowatsk/Library/Application Support/io.terraform/plugins
2021-05-12T09:50:18.207+0200 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021-05-12T09:50:18.208+0200 [INFO]  CLI command args: []string{"version", "-v"}
Terraform v0.15.3
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.39.0

Affected Resource(s)

aws_s3_bucket_object

Terraform Configuration Files

resource "aws_s3_bucket_object" "object" {
  bucket = "dnac-ova"
  key    = "dnac/dnac.ova"
  source = "/Users/x/Virtual Machines.localized/dnac.ova"

  # See Re: [hashicorp/terraform] Uploading a large file to AWS with Resource: aws_s3_bucket_object (#28678)
  # etag = filemd5("/Users/x/Virtual Machines.localized/dnac.ova")
}

Note that dnac.ova is ~80GB

Also note that this command works, in that the test.txt file appears in the bucket.

echo test > test.txt; aws s3api put-object --bucket dnac-ova --key test/test.txt --region us-east-1 --body test.txt

As a further test, I also tried:

resource "aws_s3_bucket_object" "object" {
  bucket = "dnac-ova"
  key    = "test2/test2"
  source = "test2.txt"
}

That also worked as expected, in that the file was uploaded to the bucket.

Debug Output

https://gist.github.com/NathanDotTo/22c53af50e5eace1836fed7206eeedc1

Panic Output

NA

Expected Behavior

The dnac.ova file should have been uploaded to the s3 bucket.

Actual Behavior

The dnac.ova file was not uploaded to the s3 bucket.

Steps to Reproduce

rm -rf .terraform*; terraform init; terraform apply

Important Factoids

See hashicorp/terraform#28678 (comment)

The dnac.ova file is ~80GB.

References

hashicorp/terraform#28678
hashicorp/terraform#14693

[NathanDotTo]

Note that this code does work:

import boto3
from boto3.s3.transfer import TransferConfig

# Set the desired multipart threshold value (5GB)
GB = 1024 ** 3
config = TransferConfig(multipart_threshold=5*GB)

# Perform the transfer
s3 = boto3.client('s3')
s3.upload_file('/Users/x/Virtual Machines.localized/dnac.ova', 'dnac-ova', 'dnac/dnac.ova', Config=config)

[ewbankkit]

Related:

• Use of resource "aws_s3_bucket_object" with etag = filemd5() function results in large memory usage #11656
• EntityTooLarge: Your proposed upload exceeds the maximum allowed size #788
• Use of filemd5() function results in large memory usage terraform#23890

[ewbankkit]

@NathanDotTo Improvements to the Terraform file hashing functions (including filemd5) were made in hashicorp/terraform#28681 and released in Terraform v0.15.4.
Could you please upgrade to that version of the Terraform CLI, if possible, and retry your scenario?

[leonfs]

Are there any updates on this Issue? there seems to be a PR open for this fix.

https://github.com/hashicorp/terraform-provider-aws/pull/14382/files

[github-actions]

This functionality has been released in v3.71.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.