Docker mappings - incomplete? #424

huntc · 2014-12-15T10:43:12Z

I've just noticed that docker:mappings does not include the DockerFile that is produced. The sbt-bundle plugin uses docker:mappings as a general approach to including files when staging or producing a dist (it actually doesn't know anothing about the 'docker' config as this a setting to it). Should docker:mappings also include the DockerFile itself? I'm thinking so...

As an alternative the File result of stage could be used... only again it does not include the DockerFile as it points to /files. I'd prefer to use mappings though anyhow given that it retains the relative path.

What are your thoughts? mappings appears to be inclusive of the entire package with the other configurations doesn't it? /cc @jsuereth

huntc · 2014-12-02T23:30:48Z

Any thoughts anyone? Shall I raise a PR?

muuki88 · 2014-12-13T14:03:32Z

Sorry for responding so late.

Yes, the file is missing. I'm not sure about the meaning of packaging the
Dockerfile within the package. For me it sounds like packaging the build.sbt
into a jar. However if there are good reasons, feel free to raise a pr :)

cc @fiadliel

huntc · 2014-12-15T04:05:01Z

The reason to package the DockerFile is down to determining what the best way is to ship a Docker image. You can certainly publish to a repo, but what if I want to pass it around, perhaps even in some offline mode? I don't want to pass the image itself because they are too big. Instead, I want to package the DockerFile and its associated files so that I can pass it around and retain it for posterity. The archive is then quite small. Make sense?

I shall work on this PR right now. Thanks.

The Dockerfile wasn't being included in the `mappings` setting, but it now is. Along the way I took the opportunity of cleaning up how stage works, bringing it more inline with the approach taken by the Universal plugin. One change that occurred as a result of the above is that `target/docker` is now used to generate working files (`Dockerfile`) as well as holding a `stage` folder. The `stage` folder contains the `Dockerfile` for staging add its context files. This is very similar to the structure of the `target/universal` folder.

huntc · 2014-12-15T10:43:54Z

@muuki88 @fiadliel I've now morphed this issue into a PR. Lemme know what you think.

huntc · 2014-12-15T10:55:29Z

Provisionally released as 1.0.0-5d703a5ccc1e7a2dce305fdbe4260a80c5c6a676.

fiadliel · 2014-12-15T10:56:45Z

I considered the Docker target as a runtime "archive", as much as RPMs, Debs, etc. You wouldn't expect the debian/ folder to be present in a Debian package, it just describes how to create a Debian package. Same with RPM SPEC files. The Dockerfile is completely unnecessary for runtime, so I would prefer if it were a choice, not the default.

What would you think if we had something like a docker:packageSrc command which archives up target/docker/stage for convenience?

huntc · 2014-12-15T11:12:32Z

The native packager components have both a staging folder and an archive. Staging is where files are prepared for the purposes of archiving. In the case of Docker archiving is the act of building an image from the staged files and publishing it. There is no Dockerfile included in the image that is published.

All that I've done here (aside from some tidy-up) is added the DockerFile to the mappings property. mappings provide the input to the stage task.

Does that help?

muuki88 · 2014-12-15T12:24:15Z

src/main/scala/com/typesafe/sbt/packager/docker/DockerPlugin.scala

@@ -117,9 +116,10 @@ object DockerPlugin extends AutoPlugin {
    }

    val dockerCommands = Seq(
-      Cmd("ADD", "files /"),
+      Cmd("ADD", "* /"),


This will always include the Dockerfile, right?

Indeed it will, hence the removal of it further down. I could be more explicit about what is included if you want me to. This seemed like the simpler approach though.

Will the internal path change? I mean, currently it is /opt/docker/<app> and for me
it looks like /stage/opt/docker/<app> now

Ah... the stagingDirectory is used for this?

huntc · 2014-12-16T23:39:52Z

Any movement on this? Don't like?

huntc · 2014-12-17T06:22:14Z

Just to provide a little more background, we have another plugin named sbt-bundle which is for our new Reactive Runtime project. The idea of this plugin is to take all files for any archive produced by the native packager and we use mappings to source these files. It then "bundles" these files into a tgz, takes the digest and encodes the digest in the filename i.e. it fingerprints the filename so that you have a filename representing the contents. It also includes a "bundle descriptor" describing the thing that was bundled.

The user declares a bundleType setting when using sbt-bundle and assigns a config. By default Universal is used. However you can do this:

bundleType := Docker

The config declared above is then used to scope as mappings in bundleType.value effectively yielding mappings in Docker in the case above.

sbt-bundle itself has no knowledge of the types of archive that can be produced by the native packager. In fact all that it really wants is the mappings and the config as described. The plugin does depend on the native packager for a few other things though such as the archive name and stuff like that.

Does this make more sense now?

muuki88 · 2014-12-17T17:24:12Z

I'm willing to merge this pull request as we will do a lot more refactoring
withing 1.0.0-Mx as we have a lot of request. Actually I'm thinking about providing
more flexibility, but I haven't made up my mind yet what would be the best way.

Main points users want to customize (and are not yet available ) are

CMD
ENTRYPOINT

I'm not sure if there should be a single task which generates the list of docker commands.
E.g.

val dockerFileCommands = taskKey[Seq[Cmd]]

WDYT?

Docker mappings - incomplete?

huntc · 2014-12-18T03:48:15Z

Thanks for the review - should we do another release?

huntc · 2014-12-18T03:51:19Z

Main points users want to customize (and are not yet available ) are

CMD
ENTRYPOINT
I'm not sure if there should be a single task which generates the list of docker commands.
E.g.

val dockerFileCommands = taskKey[Seq[Cmd]]
WDYT

May be we could allow the user to specify a function as a task where the function receives the settings that we already use to form a DockerFile so that they can customise the Dockerfile easily within the context of sbt. So the goal of the function would be a string containing the contents of the Dockerfile. The default setting for this function could be something along the lines of the function we have presently.

muuki88 · 2014-12-18T10:07:08Z

I'll clean up the documentation (at least to a point where every example works for both the 0.8.x and the 1.0.x versions) and the we do another round of releases. Is this weekend early enough for you?

huntc · 2014-12-18T12:25:38Z

Sure thing. No rush! Thanks.

muuki88 · 2014-12-28T12:57:27Z

@huntc . This pull request broke most of the docker tests. They all fail with

[error] 2014/12/28 13:53:51 *: no such file or directory
java.lang.RuntimeException: Nonzero exit value: 1

Did this happen on your system as well? My configuration

Ubuntu 14.04 LTS
Docker version 1.2.0, build fa7b24f

huntc · 2014-12-28T19:35:42Z

The PR validator suceeded.

The Docker tests also suceeded on my machine (OS X).

huntc · 2014-12-28T19:36:57Z

The build is also passing according to the indicator in the README.

muuki88 · 2014-12-28T19:51:44Z

The docker tests don't run on the automated tests, We haven't figured out how to configure
travis-ci to run our docker tests.

We have two options.

I'm reverting the PR and we try a new one and I'm releasing the current state as M4
We fix this issue

IMHO, I prefer option 1 and I'll prepare a bigger refactoring for the docker plugin, which
includes your changes :)

muuki88 · 2014-12-28T19:55:47Z

DId you run sbt scripted docker/* ?

huntc · 2014-12-28T22:28:40Z

I can take a look right now

huntc · 2014-12-28T22:40:40Z

Running sbt clean "scripted docker/*" passes on my machine with the branch I had.

Now trying the master branch...

BTW: Is docker itself working correctly on your machine? Can you do a docker ps?

muuki88 · 2014-12-28T22:41:29Z

docker ps works fine on my machine.

huntc · 2014-12-28T22:46:51Z

OK, the master branch is passing on my machine... docker -v yields:

Docker version 1.3.2, build 39fa2fa

I dunno whether that'd make a difference...

huntc · 2014-12-28T22:47:40Z

Which of the docker tests are failing?

huntc · 2014-12-28T22:56:42Z

This is a long shot, but could you please try inserting the following:

stageDirectory.mkdirs

...before this line: https://github.com/sbt/sbt-native-packager/blob/master/src/main/scala/com/typesafe/sbt/packager/Stager.scala#L22

huntc · 2014-12-28T22:58:17Z

Also, could you trying running the test-project-docker and performing a docker:stage from there? It should be faster to debug things that way.

muuki88 · 2014-12-28T23:00:02Z

Already tried. I get the same error as in all the tests

*: no such file or directory

The ADD * / command doesn't seem to work correctly. When I run my docker client
directly from the cli I get the same error.

huntc · 2014-12-28T23:04:00Z

So you should have a test-project-docker/target/docker folder. cd into that and try a docker build --force-rm -t something . from there.

muuki88 · 2014-12-28T23:08:47Z

Yep. Already did that. I tried it again with this output:

[muki@Proteus test-project-docker]$ cd target/docker 
[muki@Proteus docker]$ ls
Dockerfile  stage
[muki@Proteus docker]$ docker build --force-rm -t something .
Sending build context to Docker daemon 7.153 MB
Sending build context to Docker daemon 
Step 0 : FROM dockerfile/java:latest
 ---> 93894ab6d6a4
Step 1 : MAINTAINER Gary Coady <gary@lyranthe.org>
 ---> Using cache
 ---> b8b64521c218
Step 2 : ADD * /
2014/12/29 00:08:22 *: no such file or directory

huntc · 2014-12-28T23:09:10Z

I just noticed a commit on Docker that may be having an effect: moby/moby#8076

Can you try upgrading your docker?

huntc · 2014-12-28T23:15:51Z

It looks as though 1.3.0 introduced the support for wildcards back in October: https://github.com/docker/docker/blob/master/CHANGELOG.md#130-2014-10-14

Docker is now also at 1.4.1. We should probably document what our baseline Docker support is WDYT?

muuki88 · 2014-12-28T23:15:58Z

My last ubuntu update removed the docker ppas. Still nice to know that we need docker 1.3 or higher?

Works now.

muuki88 · 2014-12-28T23:16:43Z

Yep. We need to document this. Most of the strange docker errors come from version gaps

huntc · 2014-12-28T23:18:49Z

What also really need are our tests working on CI so that we can enforce the version constraint.

Do you want me to raise a PR stipulating 1.3?

muuki88 · 2014-12-28T23:19:38Z

That would be awesome :)
If you have any experience running docker in travisci :) :)

huntc · 2014-12-28T23:20:43Z

Yeah, travis has a problem running Docker - I have another project that would benefit from this.

huntc · 2014-12-28T23:26:47Z

PR raised: #448

PR sbt#424 moved to a ADD wildcard approach, and then removed the Dockerfile. There were two problems with this: * ADD seems to behave weirdly regarding recursion. /opt/docker and /docker were both added to the output. * I had incorrectly set the current working directory to the parent of the context directory, where it really should be the context directory itself. This only worked because there was a Dockerfile in the parent also.

PR sbt/sbt-native-packager#424 moved to a ADD wildcard approach, and then removed the Dockerfile. There were two problems with this: * ADD seems to behave weirdly regarding recursion. /opt/docker and /docker were both added to the output. * I had incorrectly set the current working directory to the parent of the context directory, where it really should be the context directory itself. This only worked because there was a Dockerfile in the parent also.

huntc added the bug label Dec 2, 2014

muuki88 added the feature request label Dec 13, 2014

muuki88 reviewed Dec 15, 2014
View reviewed changes

huntc added a commit that referenced this pull request Dec 18, 2014

Merge pull request #424 from huntc/docker-stage

300ed33

Docker mappings - incomplete?

huntc merged commit 300ed33 into sbt:master Dec 18, 2014

huntc deleted the docker-stage branch December 18, 2014 03:47

huntc mentioned this pull request Jan 7, 2015

Back to specific directory for ADD command #455

Closed

huntc mentioned this pull request Jan 8, 2015

Back to specific directory for ADD command #458

Merged

Docker mappings - incomplete? #424

Docker mappings - incomplete? #424

Conversation

huntc commented Dec 15, 2014

huntc commented Dec 2, 2014

muuki88 commented Dec 13, 2014

huntc commented Dec 15, 2014

huntc commented Dec 15, 2014

huntc commented Dec 15, 2014

fiadliel commented Dec 15, 2014

huntc commented Dec 15, 2014

muuki88 Dec 15, 2014

Choose a reason for hiding this comment

huntc Dec 15, 2014

Choose a reason for hiding this comment

muuki88 Dec 17, 2014

Choose a reason for hiding this comment

muuki88 Dec 17, 2014

Choose a reason for hiding this comment

huntc commented Dec 16, 2014

huntc commented Dec 17, 2014

muuki88 commented Dec 17, 2014

huntc commented Dec 18, 2014

huntc commented Dec 18, 2014

muuki88 commented Dec 18, 2014

huntc commented Dec 18, 2014

muuki88 commented Dec 28, 2014

huntc commented Dec 28, 2014

huntc commented Dec 28, 2014

muuki88 commented Dec 28, 2014

muuki88 commented Dec 28, 2014

huntc commented Dec 28, 2014

huntc commented Dec 28, 2014

muuki88 commented Dec 28, 2014

huntc commented Dec 28, 2014

huntc commented Dec 28, 2014

huntc commented Dec 28, 2014

huntc commented Dec 28, 2014

muuki88 commented Dec 28, 2014

huntc commented Dec 28, 2014

muuki88 commented Dec 28, 2014

huntc commented Dec 28, 2014

huntc commented Dec 28, 2014

muuki88 commented Dec 28, 2014

muuki88 commented Dec 28, 2014

huntc commented Dec 28, 2014

muuki88 commented Dec 28, 2014

huntc commented Dec 28, 2014

huntc commented Dec 28, 2014