Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature request: new built-in function gzip(string) #3407

Closed
glerchundi opened this issue Oct 5, 2015 · 5 comments
Closed

feature request: new built-in function gzip(string) #3407

glerchundi opened this issue Oct 5, 2015 · 5 comments

Comments

@glerchundi
Copy link
Contributor

Maybe this is a little bit opinionated but here it is.

Recently I run into an issue with cloud-config (yes, again (: ), and I solved it with a gzip function. I use GCE for deploying my machines and the maximum size for metadata is 32KB. I was deploying a whole Kubernetes cluster using CoreOS and cloud-config but when I tried to create the kube-master (which requires a bunch of files -certs,keys,cfgs-) it suddenly failed: fileSizeTooLarge.

Probably the best way to create this kind of clusters is to refer to a google cloud storage which contains everything I need and load them when booting but:

  • It's not as portable as a unique cloud-config file
  • and, it has dependencies I would like to avoid

Having a gzip built-in interpolation function solved my problem, I'm using it like this:

resource "template_file" "kube-master-userdata" {
    filename = "${format("%s/%s", path.module, "files/kube-master.yml")}"
    vars {

        #
        # parameters
        #

        dns-service-ip   = "10.3.0.10"
        etcd-endpoints   = "${var.etcd-endpoints}"
        service-ip-range = "10.3.0.0/24"
        pod-network      = "10.2.0.0/16"
        fleet-metadata   = "${var.kube-master-tags}"

        #
        # files
        #

        ca-cert               = "${base64encode(gzip(file(var.file-ca-cert)))}"
        api-server-cert       = "${base64encode(gzip(file(var.file-kube-api-server-cert)))}"
        api-server-key        = "${base64encode(gzip(file(var.file-kube-api-server-key)))}"
        etcd-ca-cert-file     = "/etc/ssl/etcd/ca.pem"
        etcd-ca-cert          = "${base64encode(gzip(file(var.file-kube-master-etcd-ca-cert)))}"
        etcd-client-cert-file = "/etc/ssl/etcd/client.pem"
        etcd-client-cert      = "${base64encode(gzip(file(var.file-kube-master-etcd-client-cert)))}"
        etcd-client-key-file  = "/etc/ssl/etcd/client.key.pem"
        etcd-client-key       = "${base64encode(gzip(file(var.file-kube-master-etcd-client-key)))}"
    }
}

And then in the cloud-config template like this:

[...]

  - path: "${etcd-client-key-file}"
    permissions: "0644"
    encoding: "gzip+base64"
    content: |
      ${etcd-client-key}

[...]

WDYT, is it worth it?

@sachincab
Copy link

sachincab commented Sep 5, 2016

This is definitely needed especially with Kubernetes TLS assets.
Anyone able to workaround it using template_cloudinit_config which support gzip and base64_encode?

I could not able to get working with template_cloudinit_config.

Please let me know your thoughts or suggestions.

@sachincab
Copy link

Any plan to incorporate gzip support in template_file for upcoming terraform release?

@cemo
Copy link

cemo commented May 21, 2017

We unfortunately need for TLS too. What is the current workaround without using storages?

@apparentlymart
Copy link
Contributor

Hi all! Sorry for the long silence here.

Since Terraform v0.10.3 we have had the base64gzip function, which gzips the given data and returns it base64 encoded. Base64 encoding is the conventional way to deal with small binary data blobs in Terraform config, since strings in Terraform are required to be UTF-8 strings and so cannot safely transfer arbitrary binary data.

When a resource type in a provider expects a small binary data blob to be provided in configuration, it must be written to accept it in base64 encoding. For example, the aws_instance resource type has both user_data for UTF-8 text and user_data_base64 for raw binary data, so users can choose which one is most appropriate.

Since we now have a gzip function, I'm going to close this out. Sorry we didn't catch this back when preparing the v0.10.3 release.

@ghost
Copy link

ghost commented Mar 31, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Mar 31, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants