Feature Request: Selective merge of HCL files for the build process

[ghost]

Feature Description

At https://www.packer.io/guides/hcl is a description how building blocks can be split in files, which is very useful for storing secrets separate from the build code. However, it seems that the only way to remerge these files in the build process is to put them in one directory and call the directory instead of the build file. Working with JSON I can specify a -var-file, bash allows to use lists and terraform supports modules and file based identification with .tfvars. None of these "tricks" seemed to work for Packer though.

Use Case(s)

As admin I'ld like to use a git repository without exposing my secrets. Creating separate directories for every image forces admins to copy the "sources.pkr.hcl" file multiple times - which is tedious and a prone of errors. Serving all builds with just one file would prevent that. The ability to store the file in bucket would be even better, because it adds versioning and key based authorization.

[SwampDragons]

Hi, thanks for reaching out. We're definitely aware that adding variable support to HCL2 is a top priority to make it usable, and are actively working on it. I think that in spirit, this is a duplicate of #8540 -- do you mind if I close this issue so we can track the work of adding variables support in one place?

[ghost]

Hi there, many thanks for getting back to me. I have had a look at "#8540 Support for variables in HCL2" and believe that this addresses a different issue. The use of variables is not my immediate concern, but the ability to manage the files that contain secrets. Today, I can only build artifacts from multiple files, when these files reside in the same directory. That means I can not share one sources block across image builds. I have to create multiple copies of the same file, even though the information remains the same. Not an issue for one or two image templates, but very cumbersome when managing a larger number of templates.

[azr]

Hello @torstenboe, thanks for reaching out. I think there are two different ideas here:

the ability to manage the files that contain secrets

I think that is exactly what variables support is going to bring and I'm currently working (reflecting for now) on this 🙂

in the same directory. I can not share one sources block across image builds

Just to be sure: you would like to be able to define a source only once and to import/reuse it in different folders ?
In any case, do you have a short example of how you would like to achieve this ?

One way I have in mind to achieve this would be to add an --import [folder|file|...] option (that would probably also load variables ?).
One other way to do this would be allow to define tags in a build block, then put all your files in the same folder ( 😬 ) to finally allow to select builds to run based on a tag.

I personally think that source block sharing is going to be a very important feature of packer and would like to think a bit more about it.

[azr]

Hello @torstenboe,

I want to say that variables are now a thing in Packer HCL2, as of #8588.

We still need to make some time to think about a feature to enable easy sources reusability.

[jetersen]

This does not only apply to sources re-usability but also variables re-usability.

For instance variables to our vsphere datacenter does not change but I need to reuse it for both Linux and Windows images.

[zenitraM]

I have a similar usecase where this would be useful: I want to reuse several sources but with different build definitions and sets of parameters passed using HCL variables.

A workaround I'm testing is symlinking the common .pkr.hcl files into a single directory for each job, a bit ugly but does the job for now.

[deric4]

Not sure if this deserves its own feature request, but once this capability is implemented would it be too much to hope for 'packer modules' similar to terraform? I'm in a similar situation as @torstenboe but have complicated matters further with "AMI builds" living in their respective repositories. Builds are then layered on top of one another (base, layer1, layerN...). I've successfully used a few different approaches to keep things in sync, allowing for the end user to only need to pass a few options consumed by the provisioners but native support for something like this... 🤤

If it helps, the biggest value prop for my use case though is being able to natively tie ref of the "base packer module" into the final artifact i.e artifact name, ami tag, manifest files, etc.

Thanks!

[SwampDragons]

That's definitely somewhere we want to go eventually, but the Packer team is a lot smaller than the Terraform team so it may take us a bit :D

[project0]

i was looking into something similar, i wish there would be something like terraform modules which can be re-used at several places.

[edqallen]

Having the same issue here. I have a single project that builds ~6 packer images, but it doesn't need to build every one of them every time, just the ones the user wants at that point. It works fine with one JSON file per target and one shared variable file for all of them, but trying to migrate to HCL2 has been pain point after pain point and as it stands now it seems like anything even somewhat resembling this setup is just impossible without duplicating a bunch of data -- e.g. by making a big variable declaration block, putting it in every one of the above files, and then creating a proper variable assignment file to get rid of these "A .pkrvar file is use to assign values..." messages.

At the risk of getting a little confrontational, I don't understand why Hashicorp has taken this "load all the files in the directory" approach that seems to be the cause of this, when just taking a list of filenames would have alleviated these issues. We're all *nix users, we know how to glob when we want to.

[mcgaw]

Some docker-compose type merging a list of files on the command line would be great instead of just a directory.

[jsloan117]

Is there any update on this?