packer: 0.7.5 -> 0.8.6

[zimbatm]

It took me 2 hours to package this thing.

First I cribbed the homebrew package dependencies. Wrote a little script for that.
Then I had 5 iterations with missing dependencies, trying to find the right version of the amazon SDK and such.

It compiles on OSX but that's as far as I've tested it for now.

[mention-bot]

By analyzing the blame information on this pull request, we identified @benley, @cstrahan and @bennofs to be potential reviewers

[copumpkin]

Would it not be easier to move this into the go-packages.nix ecosystem, rather than bundling custom dependencies for everything? cc @wkennington

We already have several other hashicorp things in there, including consul and vault.

[zimbatm]

I don't know. The more packages we add, the harder it's going to be to find a common set of working repositories. My guts tells me that we should rather follow the ruby approach and have a set of dependencies per project.

In this case for example the version of aws-sdk-go that packer uses is more recent than the one in go-packages because they have changed some of the interfaces.

What seem to be happening with the other hashicorp projects is that they add a Godeps/Godeps.json file that lists the whole set of dependencies with the exact revision. We can easily transform that into a nix expression. Bug report is also going to be easier if we use the same versions as everyone.

[kamilchm]

I've created https://github.com/kamilchm/go2nix PoC some time ago, but I don't have time to polish it up.
You should end up with similar default.nix after:

$ go get github.com/kamilchm/go2nix
$ go2nix get github.com/mitchellh/packer

[zimbatm]

It looks really nice to generate the nix dependency list.
I think that you can remove the get command and let another tool do the dependency resolution.
For example if a project uses godep then you just set a GOPATH, godep restore and then you could use go2nix <package-name> to generate the deps.nix file.

[kamilchm]

I'll try to use godep more in 2 cases:

if there's ``Godeps.json``:
    convert it do ``default.nix``
else:
    use ``godep`` to generate ``Godeps.json``
    convert it do ``default.nix``

But I don't know when I'll find some time for it :(

[copumpkin]

Isn't this the same set of points that applies anywhere though? You could say the same about the python ecosystem or any of the other ones we package. There are pros and cons to the two approaches but my usual fear is that you then lose out on fixes to libraries. For example assume one thing needed extensive patching to work under Nix, then any project depending on it now needs to figure out the issues every time, rather than once. Same with security updates to dependencies. In short, doing the ruby thing feels kind of un-distro-like to me.

On Jan 22, 2016, at 04:42, zimbatm notifications@github.com wrote:

I don't know. The more packages we add, the harder it's going to be to find a common set of working repositories. My guts tells me that we should rather follow the ruby approach and have a set of dependencies per project.

In this case for example the version of aws-sdk-go that packer uses is more recent than the one in go-packages because they have changed some of the interfaces.

What seem to be happening with the other hashicorp projects is that they add a Godeps/Godeps.json file that lists the whole set of dependencies with the exact revision. We can easily transform that into a nix expression. Bug report is also going to be easier if we use the same versions as everyone.

―
Reply to this email directly or view it on GitHub.

[copumpkin]

Like, I recognize the downsides to that point as well (equivalence to the outside world and entanglement) but what I mean by "un-distro-like" is that giving every package its own unique set of dependencies feels like a bit of a cop-out. Distributions are usually about picking canonical versions that they know work and don't have security issues and so on. If we start autogenerating those things, let's say someone finds a security bug in goamz because (hypothetically) it doesn't check certificates properly. Now I can't just fix it in go-packages.nix, but need to dig around in case someone else included a custom version inside a package like this. More likely, nobody will notice until a new version of packer is released, assuming that hashicorp noticed the advisory and remembered to update it themselves. Hashicorp is generally fairly good, but I have less faith in smaller and less actively maintained projects.

[zimbatm]

You're making a good point. I don't even trust Hashicorp. Not because they are bad developers but because the go community doesn't have any releases or tooling to actively track security issues. The fallback is to read all the new commits on all of the 54 dependencies of this project which is not something really reasonable to do.

As a packager, ideally I would be able to depend on a fixed set of package akin to Haskell's LTS, and only diverge when necessary (eg: depend on an unreleased feature, bugfix, ...). And have the tooling to support that with a way to consolidate the diversions in future LTS releases.

Right now because each Go developer has it's own GOPATH with a different set of checked-out revisions we are going to run into more conflicts that with Haskell where (probably) most developers test their package against a set of LTS releases (and get feedback from the LTS guys when compilation fails). Also because most packages aren't released with versioning, developers tend to break APIs without thinking too much about it. Also because of the missing versioning the best guess to select a commit is to take any commit that's younger than the parent project.

Sorry of being negative, I'm still feeling the pain of taking 2 hours to update this dependency set. Once I'll have recovered I will have another go at it and here is what I propose:
Use as much of the go-packages.nix as possible and keep the rest in the project. Would that be alright ?

[copumpkin]

@zimbatm yeah, sure. TBC, it's not as if I'll veto the merge if you don't even want to do what you're saying. Given that it's already doing all the custom versioning and you're bumping it up to a newer one, I'd probably be fine with a small note saying "TODO: figure out how to do this more nicely and refer to #12533 for more discussion" or something.

I really just want a good answer for this sort of thing going forward, and am hoping someone comes up with clever ways to get the best of both worlds 😄

[kamilchm]

It's worth to mention golang/go#12302 and golang/go#13517

[globin]

I agree with @copumpkin that this should be consolidated into top-level/go-packages.nix but I think it's out of scope of this PR but this definitely is worth merging.