1.2.0-M8 adds an ADD opt command to the Dockerfile - why?

[akauppi]

I've updated a project from sbt-native-packager 1.1.4 to 1.2.0-M8 and this is one of the changes observed. I'm pretty sure the added command does not come from my project. What's going on?

Steps:

• have a project using sbt-native-packager 1.2.0-M8
• sbt

> show dockerCommands
...
[info] * Cmd(WORKDIR,WrappedArray(/opt/docker))
[info] * Cmd(ADD,WrappedArray(opt /opt))
[info] * ExecCmd(RUN,List(chown, -R, daemon:daemon, .))
[info] * Cmd(USER,WrappedArray(daemon))
...
[success] Total time: 0 s, completed Jan 31, 2017 4:19:53 PM

The ADD command is about copying files from the host to the docker image - why would I want to do that for /opt?

I'm sorry that the info is a bit patchy. Maybe I'll find time to make a minimum viable repo to showcase the problem.

[akauppi]

The first aid to solve this in the application was:

dockerCommands := dockerCommands.value.filterNot {
  case Cmd("ADD", "opt /opt") => true
  case _ => false   // pass through
}

This removes the command from show dockerCommands, and sbt docker:publishLocal works for me, again. I tend to think that someone's tried to make sure the /opt folder exists on the Docker. Is that it?

[muuki88]

Thanks for your question. To be honest, I can't remember. I had this commit and this commit in mind, but both are released with version 1.0.0, so it must be even before that.

[muuki88]

@huntc do you have any insights on this? I remember the contribution I mentioned in my last comment that made sure the Dockerfile itself was included.
@fiadliel any ideas on your side?

[fiadliel]

ADD copies files from the host to the Docker image, as stated. The files added by this ARE the files described by mappings in Docker, i.e. the files you are attempting to put into the Docker image.

I added the lines mentioned above (filtering out ADD), and as expected, the result is an image with no application included.

So... I'm confused, what is the issue?

[huntc]

do you have any insights on this? I remember the contribution I mentioned in my last comment that made sure the Dockerfile itself was included.

I'm sorry, I don't.

[muuki88]

Ah, too bad 😩 thanks for taking a look ❤️

[fiadliel]

From the issue details:

[info] * Cmd(ADD,WrappedArray(opt /opt))

The ADD command is about copying files from the host to the docker image - why would I want to do that for /opt?

From the Docker documentation:

ADD has two forms:

ADD src... dest
ADD ["src",... "dest"] (this form is required for paths containing whitespace)

Have you misunderstood the ordering of arguments? This does not copy /opt from the host filesystem. It copies the opt directory from the directory in which the Dockerfile is, which is the directory target/docker/stage.

While there have been some changes to the code, this was its behavior since the earliest implementation.

[akauppi]

Update: I went back to using 1.1.5 in the project. To @fiadliel I think the issue was surprises when trying to update from 1.1.x to 1.2.0-M8, and eventually needing to yield back because of those.

My own "itch" is now covered, but I think it's important for an open source project to realize when transitions would be surprising to end users, and this probably is the "issue". Let me broaden it up a bit:

1. I was surprised to find that the documentation was about 1.2.0-SNAPSHOT, and not of the latest stable version. I was expecting otherwise, and this made it difficult to find the 1.1.5 documentation (I still don't know where it is, but would like to have a link).

2. The Docker Plugin documentation for 1.2.0-SNAPSHOT suggests using enablePlugins(DockerPlugin) so I replaced the earlier enablePlugins(JavaAppPackaging) with this. This is likely one of the reasons for the surprises. I didn't want to shake the boat, just upgrade to latest version.

3. The result was that sbt docker:publishLocal no longer worked (failed with an exception), since the opt folder is not there in my project folder.

As said, this is no longer an issue for me, personally. I'm back to using 1.1.5 and considering abandoning sbt-native-packager altogether, because all we need are Docker images. Using the sbt-native-packager plugin causes added learning curve and complexities (as indicated by the above) that we can do without. We can likely replace that by just sbt stage, a Dockerfile and some sbt glue code.

If you consider there is no "issue", from the project's point of view, I am of course fine in closing this issue.

---

I don't have any mappings in Docker explicitly mentioned in my .sbt files.

[fiadliel]

In reference to your points:

1. That would seem like a good thing to have.

2. The DockerPlugin (and other formats) need other plugins in order to work. I almost always go with http://www.scala-sbt.org/sbt-native-packager/gettingstarted.html#your-first-package which has always just worked for me.
   I browsed through the history, and the documentation doesn't seem to have changed (for what plugins to enable for Docker) between 1.1 and 1.2. Still, the documentation could be considered misleading, if you weren't familiar with the overall structure of the project.
   So as far as I can tell, there was no change here, the confusion comes from not having access to the older documentation (no way of telling if docs had changed), plus a page in the documentation which doesn't provide a full view on what's needed for a working Docker app.

3. This would be due to the overall structure of this project requiring more plugins to be enabled than just DockerPlugin for a working application. Seems like it would be nice to error out or something, rather than try and work, but give a misleading error.

For mappings in Docker, usually an archetype like JavaAppPackaging will add to the mappings (with your start script, JAR files, config files, etc.). An empty set of mappings will end up with an issue like this (no opt directory).

As for whether there's an issue here or not, I'm not a maintainer or anything, but I noted some items above. Still, it is confusing to manage multiple problems in a single issue. I think it would be easiest for everyone, to file separate issues for identified problems, and close this one (since there is no issue with the ADD command itself).

[akauppi]

Thanks, @fiadliel for the clarifications.

I agree about the "separate issues for identified problems", but feel that I might not be the best person to describe them in the project's terminology and language. Could some of the project people do this, refer to this issue and then we'll close this?

[muuki88]

Sorry for my late reply on this.

I was surprised to find that the documentation was about 1.2.0-SNAPSHOT, and not of the latest stable version. I was expecting otherwise, and this made it difficult to find the 1.1.5 documentation (I still don't know where it is, but would like to have a link).

The documentation with all the versions is linked in the readme.
http://sbt-native-packager.readthedocs.io/en/latest/

This is not ideal and I haven't had time to figure out with the sbt maintainers on how to change the domain to point the readthedocs.

The DockerPlugin (and other formats) need other plugins in order to work. I almost always go with http://www.scala-sbt.org/sbt-native-packager/gettingstarted.html#your-first-package which has always just worked for me.

This is a good starting point or the Installation section in the readme.

Still, the documentation could be considered misleading, if you weren't familiar with the overall structure of the project.

You are not the first one to be confused with the docs. Currently we have a getting started page that almost fits on a desktop screen, but people seem to skip this part and directly go into specific documentation parts. There were suggestions to put the documentation on "how to build a package" or link it from every other documentation. IMHO this is bloating the docs and would lead to a more fragmented and confusing documentation.

My hope was always that people start with the getting started page and simply copy and paste the code from there. Almost everything works if you start there.

However we have to add some further information to the docs that a format plugin is just a format. People get confused way to often :(

The Docker Plugin documentation for 1.2.0-SNAPSHOT suggests using enablePlugins(DockerPlugin) so I replaced the earlier enablePlugins(JavaAppPackaging) with this.

The documentation was the same in 1.1.5.

I'm back to using 1.1.5 and considering abandoning sbt-native-packager altogether, because all we need are Docker images. [...] We can likely replace that by just sbt stage, a Dockerfile and some sbt glue code

stage is defined by sbt-native-packager ;) However I met people that only use docker:stage and build the docker image in another step.

[akauppi]

stage is defined by sbt-native-packager ;) However I met people that only use docker:stage and build the docker image in another step.

Ended up doing exactly this. The main benefit I see in it is that we get to describe Docker things in a Dockerfile, and developers don't need to learn yet another format.