Feature request: gunzip function

[jeffb4]

Current Terraform Version

$ terraform version
Terraform v0.12.6
+ provider.aws v2.24.0

Use-cases

I am using base64gzip to compress and store certificates in AWS Secrets Manager. Later I am (using Terraform) reading those secrets and injecting into Helm values as Kubernetes secrets resources. It would be useful for me to base64decode the retrieved secret, gunzip it, then base64encode it so that no further processing is needed in my Kubernetes pod to deal with the secret.

I am compressing with gzip to get around AWS Secrets Manager size limitations, so using base64encode to it instead is not a viable option.

Attempted Solutions

Within Terraform a current solution would look like writing the gzip data out to temporary files, running gunzip with a provisioner, and then reading the data back in with file()

Proposal

a gunzip() function or possibly base64gunzip() (the latter mirrors base64gzip() directly)

References

[teamterraform]

Hi @jeffb4! Thanks for sharing this feature request.

The Terraform language does not include raw binary buffers, so base64 strings are the standard way to represent small binary objects in memory in situations where you need to pass them around as values, rather than referring to files on disk.

Therefore while we could potentially offer this functionality, it would have some constraints:

• It would need to accept base64-encoded gzip as its input because raw gzip data is unlikely to be a valid Unicode string.
• It would need to fail if the result is not a valid UTF-8 bytestream, similar to how file behaves, or its result would itself need to be base64-encoded. Given that base64gzip takes unicode string and returns base64 gzip, we'd likely prefer to keep this symmetrical and make it fail if the result isn't UTF-8.

base64gzip's name is implying the order of operations is like base64encode(gzip(x)) (although Terraform doesn't have gzip as a distinct operation). This converse function is essentially a fused version of gunzip(base64decode(x)) so perhaps its name ought to be gunzipbase64 to reflect that it's the input that is base64encoded, rather than the output. (Neither name is particularly clear unless it's explained directly, but at least these names are self-consistent.)

Doing this many encoding/decoding steps directly in Terraform configuration seems rather unfortunate, but we're assuming you're stuck doing this because Kubernetes wants a base64-encoded uncompressed object while your Parameter Store value is a base64-encoded compressed object, and neither of those services itself has enough information to know how to handle this compression difference.

[jeffb4]

Doing this many encoding/decoding steps directly in Terraform configuration seems rather unfortunate, but we're assuming you're stuck doing this because Kubernetes wants a base64-encoded uncompressed object while your Parameter Store value is a base64-encoded compressed object, and neither of those services itself has enough information to know how to handle this compression difference.

That is indeed precisely the case. For my case the result of the gunzip is a PEM-encoded certificate so the UTF-8 bytestream constraint is not problematic, although of course it would be great if tf could handle non-UTF-8 bytestreams in variables. Not that I have a use-case for the latter functionality.

[iancward]

There's also value in using such a gunzipbase64() function for debugging cloud-init, which is usually gzipped and then base64 encoded.

In my scenario, we use a module to generate user data and it gzips and base64 encodes the user data as an output. If the calling module could take this and decode/unzip the value, that would be really nice.

[smurfralf]

Still need this. Its common sense that if you offer the base64gzip function that you also offer its inverse.

[nomeelnoj]

bump--For us too it would be very helpful for debugging cloudinit files that are larger than the 16k max. Right now we have to wrap terraform in a script that grabs the json output of a tf plan, parses it, gunzipb64decode's it, runs it through diff, and presents it to the user so we can easily see the diff between what is there and what is being changed, since once you pass the gzip into user_data it is no longer human readable. Thanks!

[sebastianmacarescu]

This is a valid use case for storing more data in aws parameter store and reading it back to used it in terraform.
Are there any alternatives for this?

[Nuru]

@apparentlymart @jbardin Is there any way to get this feature on the roadmap? Specifically, I am looking for the inverse function of base64gzip, whatever you want to call it. It would be fine with me if it required the output from gunzip to be a properly UTF-8 encoded string, as it would still serve multiple use cases where we need to compress strings to meet storage limits.

[commarla]

bump--For us too it would be very helpful for debugging cloudinit files that are larger than the 16k max. Right now we have to wrap terraform in a script that grabs the json output of a tf plan, parses it, gunzipb64decode's it, runs it through diff, and presents it to the user so we can easily see the diff between what is there and what is being changed, since once you pass the gzip into user_data it is no longer human readable. Thanks!

If you are using the cloudinit_config data source, you can output the unzipped value of your user-data easily doing that:

data "cloudinit_config" "this" {
  part {
    content_type = "text/x-shellscript"
    content      = module.ext4.content
  }
  
  part {
    content_type = "text/x-shellscript"
    content      = module.soft.content
  }
}


output "user_data" {
  value = data.cloudinit_config.this.*.part
}

instead of using the data.cloudinit_config.this.rendered which is zipped and base64 encoded.

[crw]

Thank you for your continued interest in this issue.

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

Please see the provider-defined functions documentation to learn how to implement functions in your providers. If you are new to provider development, learn how to create a new provider with the Terraform Plugin Framework. If you have any questions, please visit the Terraform Plugin Development category in our official forum.

We hope this feature unblocks future function development and provides more flexibility for the Terraform community. Thank you for your continued support of Terraform!

[crw]

Marking the comment "off-topic" as that is a question best asked in their repo / forums.

[jeffb4]

Closing as not planned because you haven't done anything with this issue in 5 years.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.