Introduce a template() function that works on string content

[obones]

Current Terraform Version

Terraform v1.1.5

Use-cases

Imagine you have some text files placed in a given S3 bucket by an external process.
You can reference those files using a aws_s3_bucket_object data source and even access their (string) content with the body attribute.

How would you then apply template replacements on this content and then use the rendered result?

Attempted Solutions

Using the hashicorp/template provider, one can do this:

data template_file content_processor {
    template = data.aws_s3_bucket_object.source.body
    vars = {
        some_var = "replacement value"
    }
}

Then one can send the result to another s3 location like this:

resource aws_s3_bucket_object destination {
    bucket       = "mybucket"
    content      = data.template_file.content_processor.rendered
    content_type = "text/plain"
    etag         = md5(data.template_file.content_processor.rendered)
}

However, the documentation for hashicorp/template makes it clear that it is deprecated and recommends using the templatefile function.
But in that case, there is no file involved and so no file name to give to that function.

Proposal

To take into account the above use case, I suggest introducing the template function that works on string content, just like what is already available for hash functions:
For instance, we have the md5/filemd5, sha256/filesha256, sha512/filesha512 pairs that already cover the two possible use cases.

This would allow for full replacement of the hashicorp/template provider in the above use case.

References

I could not find any issue related to this but it's a bit hard to search with just the "template" keyword.

[crw]

Thanks for the request!

[apparentlymart]

I think #26838 might be covering the same thing, though with a different underlying use-case.

That issue frames the problem as one module passing a template to another in order to be rendered in a separate location to where it was defined, so it can have access to different values.

This issue describes a different problem where the template source code isn't a part of the configuration at all, but is instead loaded dynamically from somewhere else over the network. I don't think that was ever an intended use-case for the template_file data source -- we've historically not aimed to include any features where Terraform language code can be added dynamically at runtime -- but I can see that it happens to work that way due to a side-effect of how it was designed¹.

I feel a bit ambivalent about whether to merge these issues, because although you have proposed essentially the same solution as that other issue proposed, the other issue presents a problem for which there is a better solution than proposed: to have Terraform explicitly support compiling and running templates in different locations, so that the usage can still be checked statically rather than deferring failures until runtime.

That solution would not work for your underlying use-case, because you seem to explicitly want to introduce new Terraform language code into the program at runtime. My instinct for the moment is to keep this separate specifically because this one requires us to make a decision about whether dynamically loading and executing new code at runtime is a feature we intend to allow, but @crw feel free to override that determination if you'd rather consolidate these discussions around the broader problem. (If we do consolidate it, I think it's important to capture the new use-case in the other issue, since it implies some different requirements than what we were discussing in that other issue already.)

---

¹ Some historical context, in case it's useful for discussion here: template_file actually only ever existed as a data source because the Terraform v0.11-and-earlier language was not sophisticated enough to support a function like templatefile where the second argument has constraints that vary depending on the first argument.

The original discussion is over in #215, where the original use-case presented was rendering templates from other files on disk, rather than files from strings in memory. That discussion predates the concept of data sources and so you can see there that I originally proposed it being a managed resource since that was the only primitive Terraform had at the time that was able to take dynamic data in this way. #4169 later proposed data sources and proposed turning template_file into one, and then when I was working on HCL 2 for Terraform v0.12 I explicitly included the requirement that we be able to offer the templatefile function as originally requested, and so Terraform v0.12 finally managed to achieve what #215 originally proposed, and in turn deprecated the stopgap mechanism that had preceded it.

None of this is to say that we can't introduce something new to meet this requirement, but for any new requirement we first need to study the implications of it, part of which is understanding the historical context and checking whether there are any historical assumptions we need to revisit and challenge, and any possible constraints or considerations for increasing the scope.

[obones]

Thanks for taking the time to provide such a detailed explanation, it widens my (scarce) knowledge of how Terraform works.

I think #26838 might be covering the same thing

As you said it's not the same use case, but yes, the proposed templatestring method would achieve the same goal as my proposed template function.

you seem to explicitly want to introduce new Terraform language code into the program at runtime

I'm not sure to understand what "new language code" means in this context. To me there are no new resources created during the apply, only content that shows as (known after apply) in the plan.
But you have much better knowledge of Terraform's interior workings, so I must be missing something here.

I think it's important to capture the new use-case

In that case, let me give you two other use cases for the suggested template function.
The first is already in use in my setup and is coded like this:

data local_file template_needed {
    filename = "sourcefile.txt"
    depends_on = [ null_resource.sources_retrieval ]
}

data template_file template_needed {
    template = data.local_file.template_needed.content
    vars = {
        someVar = "Value"
        api_key = aws_api_gateway_api_key.api_key.value
    }
}

resource aws_s3_bucket_object template_needed {
    bucket       = local.s3_doc_bucket_name
    content      = data.template_file.template_needed.rendered
    content_type = "text/plain"
    etag         = md5(data.template_file.template_needed.rendered)
    depends_on   = [ null_resource.sources_retrieval ]
}

Now, you will surely say that I should use templatefile because I'm actually applying the template to a file. That is indeed correct, but I'm using hashicorp/local instead because it has the nice behavior that makes it wait for its depends_on items to have run before checking the existence of the file it has been given.
This is because the source file is coming from a zip file hosted on an S3 bucket and that there is currently no way to enumerate the content of a zip file, let alone if it is hosted on a S3 bucket. Incidentally, I already did a feature request for this.

And that leads me to my second use case, the thing that I'd love to be able to write:

locals {
    templated_file_names = ["firstfile.txt", "secondfile.json"]
}

data aws_s3_archive_object sources {
    bucket = data.s3_source_bucket.id
    key = "version/sources.zip"
    provider = aws.ci_read
}

resource aws_s3_object sources_files {
    for_each = data.aws_s3_archive_object.sources.files

    bucket       = aws_s3_bucket.destination.id
    key          = each.key
    provider     = aws.destination
    etag         = md5(content) // the md5 of the actual content, as generated by template() if applicable. Something akin to "auto etag" would be nice here
    content_type = each.value.content_type // this may not be available, in which case I would use a lookup based on file extension

    content = (contains(templated_file_names, each.key)) ? template(
        each.value.content, {
            someVar = "Value"  
            api_key = aws_api_gateway_api_key.api_key.value
        }
    ) : each.value.binarycontent
}

Basically, this would take a zip file from an S3 bucket, enumerate its content and create as many S3 objects in a destination bucket (using a different provider), applying a template only to some files from the zip file, while passing through the other files.
Here, once again, the suggested template function would work on "in memory" content and not on files present on disk before the apply phase.
I understand, however, that this might not be possible to do in any foreseeable future, but at least you now know what I'm trying to express in Terraform. Having the above possibility would allow for a much simpler terraform syntax on my side, completely removing the need for local-exec provisioners.

[dee-kryvenko]

Disclamer: no community rules are broken, no offense intended, just a constructive feedback on Hashicorp latest decision making logic. If my message below offends anybody - just remember that sunlight is the best disinfectant.

@apparentlymart the amount of time that took you to write that response probably exceeded the amount of time required to implement a template function to begin with. I love how everyone this days prefer to enjoy writing essays instead of GSD (me included).

This must be a no brainer - template_file data source took a string as an input. People created a lot of code based on that concept, that takes template string as an input from the upstream source, like the end user input or remote location. As of now there is no replacement solution that exists that would work universally because template_file is deprecated and not even build for darwin_arm64 and an equivalent backward compatible build-in function simply doesn't exists. You guys just deliberately breaking backward compatibility and putting users in an impossible position.

In your explanation you are basically telling us that you are protecting us from shooting our own foot by not giving us a template function, but since you broke backward compatibility - now we have to go through the length of hacks. Because if you didn't keep backward compatibility - someone else will have to down the chain. You just swept the problem under the rug so that downstream users have to worry about it now. And you are in the clear, right? All the potential ways I can shoot my own foot building back compatibility layer you broke somehow are lesser evil than a hypothetical template function that would bring back previously existed functionality and cost you literally nothing?

You guys are so ignorant lately that I am now understanding why so many users this days prefer to move to cdk/pulumi/crossplane etc rather than to deal with your opinionated overthinking and artificial limitations "for the greater good". Speaking from experience - made a lot of mistakes like that myself. Wake up before it's too late guys. You may soon have no one to write essays for anymore. Speaking from experience...

[crw]

@dee-kryvenko

Disclamer: no community rules are broken

Please see: https://www.hashicorp.com/community-guidelines, specifically:

As you are working with other members of the community, keep in mind the following guidelines, which apply equally to founders, employees, mentors, contributors, or anyone who is seeking help and guidance.

The following list is not exhaustive:

Be welcoming, inclusive, friendly, and patient at all times.
Be considerate.
Be respectful.
Be professional.

I do not plan to remove this post, but the final paragraph crosses the lines of "respectful" and "professional." In the future please refrain from the use of derisive language. Thanks for your consideration!

[dmikalova]

A use case for me is creating an aws_iam_policy_document in a central configuration which has a few templated values like region and account id, and then rendering the actual values in submodules that are using aliased providers in different accounts and regions. I can probably get away with replace() but would prefer to just render a template.

[jon-ward-unmind]

My current workaround to get templatefile-like functionality from a string is to use a heredoc and the format function. E.g:

locals {
  assume_role_policy_template = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "%s",
      "Effect": "Allow",
      "Principal": {
          "Service": "%s"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

  ecs_task_assume_role_policy      = format(local.assume_role_policy_template, "AllowECSTaskService", "ecs-tasks.amazonaws.com")
}

[jdwilly2001]

I am trying to re-use an ECS service module that was using a templatefile() approach to stamp out container_definitions for AWS ECS task definitions.

I have a use case that requires its own task definition. The module takes care of a lot of variables like t-shirt sizing, port mappings, etc. With this feature, I could implement easily a 'bring-your-own' template style.

You can use the ${path.root} in conjunction with a variable to let someone specify a file from the calling repo. Then use that in templatefile().

[Justin-DynamicD]

Just created a new issue that turns out is identical to this one already in play Linking here. Feel free to close mine as duplicate (assuming im adding voice to the issue)
#33335

[Justin-DynamicD]

That solution would not work for your underlying use-case, because you seem to explicitly want to introduce new Terraform language code into the program at runtime. My instinct for the moment is to keep this separate specifically because this one requires us to make a decision about whether dynamically loading and executing new code at runtime is a feature we intend to allow, but @crw feel free to override that determination if you'd rather consolidate these discussions around the broader problem. (If we do consolidate it, I think it's important to capture the new use-case in the other issue, since it implies some different requirements than what we were discussing in that other issue already.)

Just want to get clarity on the "introducing new Terraform code" statement. How is this any different from any other data reads? We are basically just saying 'this string should be rendered as a template" which is not too removed from say jsondecode() rendering.

I guess I'm curious how this is deviates from the intended design at all?

[roman-vynar]

Same here.

Our use case was using template_file() as non-existing template( template_string, vars ).

Then templatefile() was introduced as a replacement but it's not.
Who ever needed templatefile() if you can call file() anywhere you want to read from it and get into the string.

There is no function to implement template( template_string, vars ) at this moment.
Using replace() as suggested above just fine.

Please turn templatefile() into the built-in template(). Who wants to read from file they can add file().
Thanks!

[rossigee]

Please don't let this issue go stale! 🙏 Also, it would also be great to get some kind of update or feedback from HC on whether there are plans to fix this in-house, or whether it's just another one of those 'PRs accepted' issue (or not?). From what I can see, it seems to be a PITA problem that is adversely affecting a lot of people's workflows, not least mine.

In my case, I'm simply trying to fix up warnings that the 'template' module has now (long) been deprecated and archived and we shouldn't be using it. Having seen and used the 'templatefile' function, I just kinda assumed there would be an equivalent template function that took a string argument, so that it doesn't involve unnecessary file I/O. However, after I churned out a nice little patch to tidy up and update our module, I find myself here on this unfortunate thread 🤦 not really sure what best to do 🤔

Hashicorp, please advise!!!

--- a/modules/aws/s3/main.tf
+++ b/modules/aws/s3/main.tf
@@ -360,21 +360,13 @@ resource "aws_s3_bucket_policy" "double-facepalm-s3-policy" {
   })
 }
 
-data "template_file" "iam-user-policy" {
-  count = var.create_user ? 1 : 0
-
-  template = var.iam_user_policy != "" ? var.iam_user_policy : local.default_user_policy
-
-  vars = {
-    bucket_arn = aws_s3_bucket.double-facepalm-s3.arn
-  }
-}
-
 resource "aws_iam_policy" "double-facepalm-s3-policy" {
   count = var.create_user ? 1 : 0
   name  = "double-facepalm-${var.name}-s3-access-policy"
 
-  policy = data.template_file.iam-user-policy[0].rendered
+  policy = template(var.iam_user_policy != "" ? var.iam_user_policy : local.default_user_policy, {
+    bucket_arn = aws_s3_bucket.double-facepalm-s3.arn
+  })
 }
 
 resource "aws_iam_user" "double-facepalm-s3" {

[rossigee]

Hashicorp, please advise!!!

OK, I've now caught up with all the related comments and I now see that @apparentlymart is suggesting that this is not going to be implemented for various reasons but doesn't seem ready to offer a useful replacement. It sounds like we're being expected to write our strings to file and use the templatefile function or something as a workaround ?! 🤦 I haven't seen any better suggestions.

So I understand the arguments that templatefile and template functions may not be the 'perfect' solution for templating, but the community here has a lot of IaC that already works that way and I'm sure most of us would really like to keep it working that way until a such a better 'perfect solution' can be proposed.

By deprecating the 'template' module and only implementing a templatefile function (but not a corresponding template function) I think you've jumped the gun! From my perspective, you have only gone half way towards providing the community with a suitable replacement (or stepping stone) that the community can reasonably expect to work with in the mean time - and then given up!

I agree with another comment from earlier that suggested that if you'd have at least implemented the template function instead of templatefile, we could combine it with the file function if we need to. That'd work just fine in my case, and probably the majority of other people's cases too.

It's a real shame this doesn't appear to be much of a concern for HC, because it sucks for the rest of us who will now be stuck in an awkward limbo until further notice. Thanks 🙄

[jaimehrubiks]

Is there any official communication on why this is not being implemented or what the alternative is? I see many legit use cases for this function to be implemented

[roman-vynar]

@crw Can you please follow up on this? That's a small request to return things back but lots of inconveniences for community. Thank you for your time!

[crw]

@roman-vynar Thanks for your question. The question on the table for the team is whether we would rather have this happen in an external function via #27696 (comment) (assuming support for external functions is released), or have one official implementation that is supported as part of core Terraform. I will re-raise this with product, as this is a product decision. Thanks again.

[kristeey]

We have a similar use-case which could be solved with deprecated template_file but not the templatefile function:

1. We are creating github repositories from a github repository template using the github_repository tf resource.
2. Fetches a file from the newly created github repository using a github_repository_file data source.
3. Templating/inserting config variables into the content of that file.
4. Uploading the configured file to the github repository again using a github_repository_file resource.

It is desirable to have the file that is going to be templated as a part of the original template github repository in order to centralize the file that are templated.

[abower-digimarc]

I also have a use case for this needed but non-existent function.

My engineer has a file that will get passed to GCP Workflows. This file doesn't build terraform, it's not introducing new TF code. However, the file DOES reference values my terraform is responsible for, like 'GCP Project' and 'Workflow Task Queue'.

I have a terraform module that I use to deploy this application. I want this module to be able to get the developer's file from the application repo, and find/replace multiple variables with known terraform values, and then submit that string as the GCP workflow contents.

templatefile would work IF that file was in my module, but it isn't. I want to create dependency injection which allows my developer to change that workflow without constantly updating the .tf module (as long as they abide by the contract of which values are being replaced).

The proposed solution above seems like it would fill this need.

[fractos]

I've just hit this with a similar use-case to the engineer who was developing an AWS ECS module. It feels wrong to have to pass the path and filename of a template all the way down. Including a template function would be adding general utility that fits a bunch of use-cases - you can sense the gap where such a function should be. I don't really understand how this could be a product decision. Looking at the code, it confirms my suspicion that it would be a very straightforward implementation which allows code re-use.

[crw]

Thank you for your continued interest in this issue.

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

Please see the provider-defined functions documentation to learn how to implement functions in your providers. If you are new to provider development, learn how to create a new provider with the Terraform Plugin Framework. If you have any questions, please visit the Terraform Plugin Development category in our official forum.

We hope this feature unblocks future function development and provides more flexibility for the Terraform community. Thank you for your continued support of Terraform!

[fractos]

I think pigeon-holing this as something that must be implemented by an external function is complete overkill, real Heath Robinson territory, and a waste of my time to consider seriously. The current code even has separate load-from-file and replace-placeholder logic - all the components are already there waiting to be hooked up.

[rossigee]

Let me get this right. About two years ago, you took a simple, useful, generic, stable, well-used string templating function that many TF users in the community were actively using to keep their IaC code simple and concise - and just removed it!

Reasons were given by Hashicorp related to some underlying refactoring that was being done, but no actions appear to have been made by them to correct it. From the user's perspective we just saw a very useful function we had previously taken for granted get rug-pulled on us, for no good reason, with no good replacement.

Suddenly, we were without the ability to do 'in-memory' templating. So we all had to find some extra 'spare time' between our TF upgrades to add otherwise unnecessary IaC code (bloat) to our repos to mess around with temporary files etc. Now, all our codebases are that little bit less efficient, and are slightly more complicated to read, explain and maintain etc. Thanks a lot for that! 🙄

It should have been a simple decision to roll back the removal of the function, but sadly here we are two years later and you're now adding insult to injury by suggesting that the right way to fix this should be to write and maintain (and have all our repos depend on) a whole new plugin?! Seriously?!

[mbillow]

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

@crw This has the same problem that was pointed out six years ago in the template provider's source code. Since the provider isn't a part of the language parser itself, it has to package it's own copy of hashicorp/hcl and hashicorp/terraform/lang at pinned versions. This is not only inefficient from an artifact size standpoint, but it also is confusing to users because a function in their version of Terraform might not be in the provider because it has an older version of hashicorp/terraform/lang.

---

All in all, it feels incredibly weird to me to support a function like format, but then take a stance that such a similar function like this shouldn't be standard.

Don't get me wrong, I am more than happy to build a provider function to unblock my future self when 1.8 is released, but that feels like we are just kicking the can we decided was inefficient and confusing six years ago even further down the road.

[mbillow]

I am more than happy to build a provider function to unblock my future self when 1.8 is released

For those looking to also be unblocked, I built and published the provider: mbillow/string-template.

Example:

terraform {
  required_providers {
    string-template = {
      source = "mbillow/string-template"
      version = "0.1.0"
    }
  }
}

variable "demo_template" {
  type = string
  description = "User provided HCL template string."
  // Since we are defining a default inline, we need to use %% and $$ to get the literals % and $.
  // Your templates don't need double characters if they are coming from outside your HCL.
  default = <<-EOT
  %%{ for ip in split(",", ip_addresses) ~}
  $${ ip }
  %%{ endfor ~}
  EOT
}

locals {
  template_vars = {
    ip_addresses = "1.1.1.1,2.2.2.2,3.3.3.3"
  }
}

output "demo" {
  value = provider::string-template::template(
    var.demo_template,
    local.template_vars
  )
}

Output:

Changes to Outputs:
  + demo = <<-EOT
        1.1.1.1
        2.2.2.2
        3.3.3.3
    EOT

[ddaws]

I think the reason for exposing templating as a plugin instead of directly into the language is because there are many different templating languages that teams may want to use. By having a template function as part of the Terraform runtime Hashicorp has to pick and support a specific templating syntax. They've already done this by exposing a templatefile function, but that doesn't mean it was the best way to organize this functionality.

By allowing providers to expose their own template functions Terraform can support many different templating languages. Terraform still includes a templatefile function, but arguably this should be removed as well in favor of providers exposing a template function and templating files like provider::jinja::template(file("foo.txt", { ... })

[mbillow]

By having a template function as part of the Terraform runtime Hashicorp has to pick and support a specific templating syntax.

This would be a good point if we were in a pre-HCL Packer issue, sure. But to say that HashiCorp shouldn’t support a first party template function in the Hashicorp Configuration Language… That’s a stretch for me. They invented their own markup language with its own embedded templating language… of course they have to support it.

I think your point is incredibly valid for the examples given, that are outside the ecosystem, though.

[apparentlymart]

Hi all! Sorry for the long silence here.

Yesterday's v1.9.0 alpha release of Terraform CLI included an experimental solution to this feature request, and so it'd be very helpful if some of the folks who were interested in this issue would give it a try and share information about what happened.

For experiments we prefer to gather feedback in community forum topics rather than GitHub because GitHub issues don't really work well for multiple threads of discussion, so if you'd like to participate please take a look at this topic instead of posting new comments here:

Experiment Feedback: The templatestring function

For experiments like this it's helpful to see both successful and unsuccessful attempts to use it, so if you try it out and it works as you expected I'd still encourage you to leave a comment describing what you tried. Positive feedback on this will increase our confidence in stabilizing the current implementation as-is, as opposed to iterating on it further.

I'm going to lock this issue temporarily just to encourage keeping the experiment feedback all contained in the linked community forum topic, since it'll be harder to keep track of feedback in two different places. I'll unlock this again once our focus shifts away from feedback on this specific release and we're ready to discuss what might happen next.

Thanks!