File provisioner fails Packer build immediately if file to be uploaded is missing

[patrickdlee]

This behavior is logical in most cases and should certainly be the default. However, we have a use case in which a shell provisioner generates a file which we then want to upload using a file provisioner. In order to get around the error, we use zero-byte placeholder files to fool Packer into thinking all the files to be uploaded are already there when the build starts. Then we generate the real files and upload them. Yes, it feels quite icky to do this.

I propose an optional configuration option to indicate that the file will be generated and the Packer build command should not fail because it's missing...

{
  "type": "file",
  "source": "generated.tar.gz",
  "destination": "/tmp/generated.tar.gz",
  "generated": "true"
}

Using "generated" as the option name is just a suggestion. There are other choices that may be clearer.

[patrickdlee]

I should clarify that we use the shell-local provisioner and not the original shell provisioner in order to generate the file on the build server prior to uploading it with a file provisioner.

[cbednarski]

Thanks for the suggestion! This is an interesting use case. I'd like to understand it a bit better.

Can you explain why you need to create this file with the local shell provisioner? Why, for example, can it not be generated before packer runs? I imagine you could, for example, share config state through environment variables and use make to do the prep work before kicking off the packer build.

I'm aware that there are some difficulties around getting things out of the VM being built but this is the first time I've seen trouble going the other direction.

[patrickdlee]

@cbednarski Sure, I'd be happy to explain in more detail. We have a project in which we need to build several different AMI's for the various components of the infrastructure. We're using Jenkins to kick off Packer builds and we'd prefer to have a single parameterized Jenkins job where we can choose the AMI type from a select box. In order to do this, the commands that make up the build process need to be the same across the different AMI's. One convenient way to achieve that is to have a Packer template for each AMI that includes everything needed to create it: local shell scripts, file uploads, remote scripts, Puppet runs, Serverspec tests, etc. Then the appropriate Packer template can be called based on the parameter in the job and all the magic happens from there.

So, yes, we could call a shell script before the Packer build to generate the files we need. We'd just prefer for simplicity and uniformity to have the Packer build as the only step in the process. The new shell-local provisioner makes this possible as long as we have a way to appease the file provisioner.

[cbednarski]

Thanks for the feedback. I've heard a few other similar pieces of feedback. I think we're in a weird spot now where packer is subsuming a lot of make-like functionality, which I don't think is really packer's design goal. Can it do that? Yes. Should it? No. It breaks unix design philosophy.

Right now there's no notion of a pre-processor step, which is essentially what you're asking for. There are at least two other concrete use-cases that fall into this category so I don't want to rule this out as a possibility in the future. In the short term, though, it makes a mess of packer's internals when there's already a dozen solutions outside of packer -- make, bash, powershell, python, ruby, etc. -- that work just fine.

[activatedgeek]

@cbednarski I have an ansible-local provisioner. But for that, I need my playbook directory to be uploaded and I am giving relative paths. The build returns the following error:

Packer v0.8.6

vagrant-virtualbox output will be in this color.

2 error(s) occurred:

* playbook_file: ../playbook/devbox.yml is invalid: stat ../playbook/devbox.yml: no such file or directory
* playbook_dir: ../playbook is invalid: stat ../playbook: no such file or directory

The Packer file looks like this,

    {
      "type": "ansible-local",
      "playbook_file": "../playbook/devbox.yml",
      "playbook_dir": "../playbook"
    }

Isn't this wrong? Shouldn't the directory mentioned in playbook_dir be uploaded to the remote machine?

[phinze]

Hi folks! Just wanted to write down a use case that led me here today.

I'm iterating on a Docker build, and the code upload step is pretty slow, so I looked into duplicating the same "ignore .git" strategy I had when I was building with a Dockerfile.

That investigation ran me into #1811, so I decided to try and work around by doing a shell-local git archive of master and then uploading that tarball using the file provisioner, which brought me here.

IMHO moving this check to runtime is a reasonable change to make to the default behavior of the file provisioner, which would sidestep the concern of adding "yet another config option".

Here's my thinking: if we weigh the benefit of failing fast in the case of a typo or config problem against the benefit of enabling currently-impossible use cases described here, it seems to me that the former is only a small nicety and the latter is a very concrete improvement to Packer's utility.

Currently I'm going to work around by doing a touch on my target path for git archive while I iterate locally.

[jessica-pixvana]

I'm trying to do exactly what @cbednarski is. Was there a workaround for this?

[alexmootassem]

Trying to do exactly the same thing. Is there any workaround ?

[inkel]

Same use case here.

[xrage]

Same use case for me as well.

[mwhooker]

I would suggest maybe uploading a directory in the file provisioner. That way, it doesn't matter which files are in it. Does that seem like a reasonable solution to people?

@phinze would this solution work for your use-case?

[phinze]

@mwhooker ooo that's a good idea! I do believe that would work for the use case I described.

[mwhooker]

Thanks @phinze !

Going to close this since it sounds like we have a workable solution, and I don't want to make a change like this without greater consideration.

[robsonvn]

I want to compress a local folder before uploading it, this simple use case is failing, mind-blowing!

    {
      "type": "shell-local",
      "command": "tar -zcf /tmp/foo.tar.gz -C src ."
    },
    {
      "type": "file",
      "source": "/tmp/foo.tar.gz",
      "destination": "/tmp/foo.tar.gz"
    },

[SwampDragons]

@robsonvn This isn't on by default to prevent users from getting messed up by mistyped filepaths, but you can set the flag "generated": true in the file provisioner definition to turn this validation off.

[SwampDragons]

See https://packer.io/docs/provisioners/file.htm for full details

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.