Chunked file transfers for azurerm_data_lake_store_file

[dyindude]

Files larger than 4MB fail to upload to ADLS with azurerm_data_lake_store_file due to lack of implementation of the API's buffersize parameter in the Azure SDK for go. There is an open issue for this on that project here: Azure/azure-sdk-for-go#3231

The referenced issue contains a workaround which transfers the file using 4MB chunks (which appears to be a static value in the API: https://docs.microsoft.com/en-us/rest/api/datalakestore/webhdfs-filesystem-apis )

I have implemented the workaround in this commit, and tested the change in a terraform project I was encountering this error with. I tried my best to make it match the style of the code already present, so let me know if something else needs to be changed.

I am relatively new to golang, so I welcome any constructive feedback! Thanks

[katbyte]

Hi @dyindude,

Thank you for fixing this bug! This looks pretty good to me, however could we get a new test for large files over 4Mb?

[katbyte]

Ideally generate a large local file rather then keep something in the repo itself

[dyindude]

I took a stab at it, using the existing TestAccAzureRMDataLakeStoreFile_basic test as a template. I also ran the tests myself with make testacc TESTARGS='-run=TestAccAzureRMDataLakeStoreFile'

[katbyte]

Thanks @dyindude!

I hope you don't mind but I changed it to use a tmp file and to remove it after the test so I could merge it 🙂 now LGTM

[dyindude]

I don't mind at all, I have learned a lot from this process :) thanks

[WodansSon]

👍 LGTM

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!