docker:publishLocal fails on and after 1.7.0

[kingslef]

Expected behaviour

sbt docker:publishLocal creates docker image.

Actual behaviour

sbt docker:publishLocal fails with:

[info] Step 5/25 : COPY opt /opt
[error] COPY failed: stat /var/lib/docker/tmp/docker-builder241894592/opt: no such file or directory
[info] Removing intermediate image(s) (labeled "snp-multi-stage-id=28adc33e-65a0-4e17-bfab-8d7c6bb635ca") 
[info] Deleted Images:
[info] deleted: sha256:da8f814f2c780c399dd625f224e79981270c399406b1b709181d814f1689087e
[info] deleted: sha256:4480b9d3b4deafbd7822413a1cdf02393ede09ee2083acda58d9a1f427e30a22
[info] deleted: sha256:43ecd5ee8e922e3fd5ca493cf0236ca07e75b52a37aada7079784c1c01d691a4
[info] deleted: sha256:7ea09f18c632d17c5f79f095a51931255ca93774fd1db81881d419aa63aa45be
[info] Total reclaimed space: 0B
[error] java.lang.RuntimeException: Nonzero exit value: 1
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.publishLocalDocker(DockerPlugin.scala:631)
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.$anonfun$projectSettings$49(DockerPlugin.scala:247)
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.$anonfun$projectSettings$49$adapted(DockerPlugin.scala:239)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error] 	at sbt.Execute.work(Execute.scala:290)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
[error] 	at java.base/java.lang.Thread.run(Thread.java:832)
[error] (foobar / Docker / publishLocal) Nonzero exit value: 1

Information

• What sbt-native-packager are you using
  1.7.5, but this first appeared on 1.7.0
• What sbt version
  1.3.13
• What is your build system (e.g. Ubuntu, MacOS, Windows, Debian )
  Ubuntu 16.04
• What package are you building (e.g. docker, rpm, ...)
  Docker
• What version has your build tool (find out with e.g. rpm --version)
  19.03.9
• What is your target system (e.g. Ubuntu 16.04, CentOS 7)
  Ubuntu 16.04

dockerGroupLayers in Docker := PartialFunction.empty makes it work but I would like to use the layered docker builds.

On another machine, using Mac OS 10.15.6 and Docker version 19.03.12, the same project works. (UPDATE: turns out it fails there too) I compared the differences with sbt docker:stage and on Ubuntu, the target/docker/stage/ folder looks like this:

1               2               Dockerfile      target

While on Mac, there is the opt directory, which doesn't contain anything else than empty docker/bin and docker/lib/ directories:

1               2               Dockerfile      opt             target

The error looks the same as in #1348, but there is no docker folder in the project.

[muuki88]

Thanks for the detailed report 🥰 . To summarize

• the build fails on Ubuntu 16.04, but not on Mac OSX
• docker version 19.03.9
• issue started appearing since sbt-native-packager version 1.7.0

The main change in 1.7.0 was the docker layering #1310 by @ppiotrow . As it's working on one OS, but not on another, I would say this is not a general bug, but some weird docker-os related thing (which there have been plenty enough).

Do you have a minimal reproducible test case? Maybe we can add ubuntu 16.04 as a test system to the integration tests as well and see if this breaks.

[kingslef]

Do you have a minimal reproducible test case?

Now I have: https://github.com/kingslef/sbt-native-packager-issue-1365! While creating it, I noticed that adding a file to the docker image actually causes the failure:

mappings in Docker += file(
  s"${baseDirectory.value}/src/main/resources/foobar.yaml"
) -> "/res/foobar.yaml"

dockerCommands += Cmd("COPY", "res", "/res")

I also noticed that I must have had dirty work directory, when I concluded that it works on Mac, as after git clean -xdff, it fails there too, so it wasn't anything specific to Ubuntu.

[muuki88]

Thanks a lot @kingslef . That clears up things a lot. I'll checkout the test case and let you know

[yanns]

I can confirm having this issue on MacOS and Linux.

[muuki88]

Does this still happen with 1.7.6 . #1370 changed the order in which layers are being used

[kingslef]

yeah, still fails.

[muuki88]

Thanks for the fast response 😘 . I'll see if I have time to dig deeper into this

[kingslef]

Happens also on 1.8.0.

[plavreshin]

@muuki88 - can confirm it fails for me as well on OSX with latest sbt-native-packager 1.8.0 and docker Docker version 20.10.2, build 2291f61 , any update on that one?

[Tvaroh]

Having this issue as well. It does happen on one of the apps and doesn't on others with exact same plugin setup. Very weird.

[muuki88]

I finally found the time to look into this. So the line that's causing this is

Docker / mappings += file(
  s"${baseDirectory.value}/src/main/resources/foobar.yaml"
) -> "/res/foobar.yaml"

which causes the DockerPlugin to add

COPY opt /opt
...
RUN ["chmod", "-R", "u=rX,g=rX", "/opt/docker"]

in the Dockerfile, which fails as these folders don't exist:

$ tree target/docker/stage
target/docker/stage
├── 1
│   └── opt
│       └── docker
│           ├── bin
│           │   ├── sbt-native-packager-issue-1365
│           │   └── sbt-native-packager-issue-1365.bat
│           └── lib
│               └── org.scala-lang.scala-library-2.13.1.jar
├── 2
│   └── opt
│       └── docker
│           └── lib
│               └── default.sbt-native-packager-issue-1365-1.0.jar
├── Dockerfile
└── res
    └── foobar.yaml

@ppiotrow it would be amazing if you find time to work on this 😍

I guess the issue is the getOrElse in this method:

sbt-native-packager/src/main/scala/com/typesafe/sbt/packager/docker/DockerPlugin.scala

Line 584 in 870e260

private final def pathInLayer(path: String, layer: Option[Int]) = layer.map(i => s"/$i$path").getOrElse(path)

[muuki88]

After tinkering a bit with this, I'm not sure if this a bug, a feature or missing documentation.

The build from the sample project can be fixed by adding a dockerGroupLayers rule and fix the mappings

Docker / dockerGroupLayers := {
  // map the "res" stuff in layer 3
  case (_, path) if path.contains("res") => 3
  // otherwise fallback to the predefined layering
  case in => (Docker / dockerGroupLayers).value(in)
}

// change the path from "/res/foobar.yaml" to "/opt/docker/res/foobar.yaml" as this is the 
Docker / mappings += file(
  s"${baseDirectory.value}/src/main/resources/foobar.yaml"
) -> "/opt/docker/res/foobar.yaml"

// copy the file also to /res
dockerCommands += Cmd("COPY", s"/${(Docker / defaultLinuxInstallLocation).value}/res", "/res")

This is... very cumbersome and unintuitive. If you use the copy command at the moment the Docker / defaultLinuxInstallLocation) is part of the the stage folder. I guess we could change this so the stage folder has a fixed directory (e.g. content`) where all the contents are located and the Dockerfile, e.g.

stage/docker
  content
     opt/docker/conf/
     opt/docker/lib/
     res/
  Dockerfile

Not sure how much work this is and how much depends on the installation path being part of the staging directory.

I want to take a step back and o understand the use case better.

Is the /res folder at the root of the image required or could it be inside the Docker / defaultLinuxInstallation path (/opt/docker) where the rest of the stuff is located?

[ppiotrow]

While implementing this feature I was surprised with Docker / mappings as I never had to use this. The implementation was mostly to cover the tests. I need to understand what was reported and then I can fix.

[kingslef]

I want to take a step back and o understand the use case better.

Is the /res folder at the root of the image required or could it be inside the Docker / defaultLinuxInstallation path (/opt/docker) where the rest of the stuff is located?

for me, the idea is to copy a file from the repo to a certain place in the docker image, so having it under /opt/docker wouldn't really work. something similar that people are asking in these SO questions:

• https://stackoverflow.com/questions/50253875/sbt-native-packager-dockercommands-copy
• https://stackoverflow.com/questions/40511337/how-copy-resources-files-with-sbt-docker-plugin
• https://stackoverflow.com/questions/28676006/add-copy-files-with-sbt-native-packagers-docker-support.

I'm not exactly sure if Docker / mappings is the best way to achieve it or would there be some other way.

[muuki88]

Thanks for the responses :)

for me, the idea is to copy a file from the repo to a certain place in the docker image

But why 😄 ? Is there a specific requirement for a third party library? As @ppiotrow pointed out ususally there's no need to add additional files via the mappings as they are either

• shipped inside a jar (and thus on the classpath)
• can be added via src/universal/conf for external config files

I do see that the behaviour is counter intuitive. Adding something to the mappings under res should make them available in the docker image via COPY("res", "/res").

sbt native packager has validations for your build configuration. Would it have helped you if there was a warning that says something like "you have non standard mapping entries that point to a directory that's not "Docker / linuxDefaultInstallation" and have group layers enabled. Either turn off group layering via ... or add a custom rules like ..." ?

[plavreshin]

Hi @muuki88 and @ppiotrow - any update on this issue?

I have tried proposed workaround to disabled grouped layers with dockerGroupLayers in Docker := PartialFunction.empty but it does not have any impact on build at all, still observing the same failure as above.

[info] Step 4/25 : WORKDIR /opt/docker
[info]  ---> Running in d1aa434f4044
[info] Removing intermediate container d1aa434f4044
[info]  ---> 9a4c8caafff8
[info] Step 5/25 : COPY 1/opt /1/opt
[info]  ---> 5f1a24057f9c
[info] Step 6/25 : COPY 2/opt /2/opt
[info]  ---> 83bf05bd8586
[info] Step 7/25 : COPY opt /opt
[error] COPY failed: file not found in build context or excluded by .dockerignore: stat opt: file does not exist
[info] Removing intermediate image(s) (labeled "snp-multi-stage-id=2f4d11a5-ac36-44cb-84f2-c24f9296cd4c")
[info] Deleted Images:
[info] deleted: sha256:83bf05bd8586f2d2f2c08aa5ae85c54d5cf683429aa1b39d6ed931d79b95ec51
[info] deleted: sha256:d3e807bdb1e50a606614aa2d4fb932230b44110dcef5f54e0ed11ae36578ad1a
[info] deleted: sha256:5f1a24057f9c481048069cfa6cc3bd9e1e78a6d315029ccb952aba77056d96d6
[info] deleted: sha256:b8eea208af067d419ddbadac7cb54b050f1fad6ad06c0b79e5d188782dfebd0f
[info] deleted: sha256:9a4c8caafff8672f839b4870fac57cca8c9f3d08e195243c8fa71bec0f56cc38
[info] deleted: sha256:62858bd8841e2d7ce8ef7c999a4a90b5a3bc023022323890c6eedde489f3bebf
[info] deleted: sha256:d2877a4e81a7f70ce95db54660af3e6054001cac969eaff56a0a05e76eefed61
[info] Total reclaimed space: 77.69MB
[error] java.lang.RuntimeException: Nonzero exit value: 1
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.publishLocalDocker(DockerPlugin.scala:672)
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.$anonfun$projectSettings$52(DockerPlugin.scala:251)
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.$anonfun$projectSettings$52$adapted(DockerPlugin.scala:243)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error] 	at sbt.Execute.work(Execute.scala:291)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] 	at java.base/java.lang.Thread.run(Thread.java:834)

[muuki88]

That's strange. The workaround with now layering work for me. Is the setting applied in the proper scope and actually used? From your log output it seems that it's still using layering.

[IvannKurchenko]

Recently I faced same issue and next worded for me.

I had mapping defined like:

mappings in Docker ++= {
 directory(baseDirectory.value / "angular-frontend" / "dist" / "angular-frontend")
}

And then I've added prefix for every file in directory, like:

mappings in Docker ++= {
  directory(baseDirectory.value / "angular-frontend" / "dist" / "angular-frontend").map {
      case (file, name) => file -> s"/opt/docker/${name.stripPrefix("/")}"
  }
}

Bottom line - you need to add /opt/docker/ prefix for additional mapping

sbt-native-packager version: 1.8.0
Scala version: 2.13
sbt version: 1.4
OS: macOS BigSur 11.1

[muuki88]

Thanks for sharing @IvannKurchenko 🤗 Let me refine the bottom line

You need to add s"/${(Docker / defaultLinuxInstallLocation).value}/" prefix for additional mappings

[BoopBoopBeepBoop]

@muuki88 thanks for taking care of this issue - I just wanted to mention quickly in response your earlier comment

But why 😄 ? Is there a specific requirement for a third party library?

My team manipulates both mappings, and in some cases manipulates or overrides dockerCommands, to make it convenient to build our entirely set of images at once (by issuing a single docker:publishLocal for the whole multiproject repo). We have 3rd party binaries that we need to integrate with, sometimes extend base images like jboss keycloak or flyway (which have specific locations that they desires extra jars and artifacts placed in the container), and in one instance simply override nearly everything & build a non-jvm container.

We could do this with separate builds & Dockerfiles, but we find it extremely useful to operate our entire stack as a single sbt-managed project, and be able to build/integration test from a single control plane.

Just wanted to give more color to how and why those end up getting touched in practice.

[muuki88]

Thanks for sharing @BoopBoopBeepBoop (an amazing display name 😂 ) 🤗

That makes a lot of sense. Especially 3rd party libraries that are not (yet) built for container environments and expect very fixed paths for their configuration or jar files.

I'm still not sure what's the way to go here. There are a few options that surfaced from this discussion

1. Leave as is and document how to properly add files when using layering ( prefix with Docker / defualLinuxInstallation )
2. Build a helper for adding docker mappings similar to the LinuxPackageMappingDLS
3. Append the Docker / defualLinuxInstallation to all paths that don't start with it. This has to be configurable in order to not break custom configurations in strange ways

I'm open to any other suggestions 😄

[BoopBoopBeepBoop]

My gut feeling, for what it is worth, is that documenting the usage of the default path variable is likely enough. Option 3 (prefixing paths) seems a bit too magic & likely to lead to more confusion.

[simao]

FWIW I am not using any mappings and I still get this error when using a custom dockerCommands.

All my sbt native packager config:

enablePlugins(JavaAppPackaging)

dockerRepository := Some("myrepo")

Docker / packageName := packageName.value

dockerUpdateLatest := true

dockerAliases ++= Seq(dockerAlias.value.withTag(git.gitHeadCommit.value))

Docker / defaultLinuxInstallLocation := s"/opt/${moduleName.value}"

dockerExposedPorts ++= Seq(9001)

dockerCommands := Seq(
  Cmd("FROM", "adoptopenjdk/openjdk11:jre-11.0.11_9-debianslim"),
  ExecCmd("RUN", "mkdir", "-p", s"/var/log/${moduleName.value}"),
  Cmd("COPY", "opt /opt"),
  Cmd("WORKDIR", s"/opt/${moduleName.value}"),
  ExecCmd("ENTRYPOINT", s"/opt/${moduleName.value}/bin/${moduleName.value}"),
  Cmd("RUN", s"chown -R daemon:daemon /opt/${moduleName.value}"),
  Cmd("RUN", s"chown -R daemon:daemon /var/log/${moduleName.value}"),
  Cmd("USER", "daemon")
)

This worked until 1.7.0, it works with 1.6.2

[fernanluyano]

can we get an update on this? I'm having the issue on Mac

[muuki88]

@simao thanks for the patience.

You are not using custom mappings, but you change the WORKDIR and ENTRYPOINT without changing the underlying file layout. The target/docker/stage folder now contains a layered layout for caching purposes. It's not safe to rely on the layout staying the same.

Options are to either deactivate layering entirely or to use the default dockerCommands and use Docker / mappings instead to rearrange the the built.

[muuki88]

@fernanluyano

can we get an update on this? I'm having the issue on Mac

As mentioned in #1365 (comment) we most likely will document this behaviour and users that change Docker / mappings or rely on the layout of target/docker/stage will need to fix their builds.

The docker build is already very complex and adding even more complexity will not solve this 😞

[jijingg]

I still have publish fail issue on MacOS

• sbt : 1.6.2
• OS : Mac-os 12.2.1
• docker-OS: centos7

[warn] 	You need to remove it from the cache manually to take effect.
[info] 	published ivy to /root/.ivy2/local/com.github.spinalhdl/spinalhdl-sim_2.11/1.6.5/ivys/ivy.xml
[error] java.lang.RuntimeException: Nonzero exit value: 129
[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:138)
[error] 	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:108)
[error] 	at $8952c887d8c4e436510f$.liftedTree1$1(build.sbt:83)
[error] 	at $8952c887d8c4e436510f$.gitHash(build.sbt:82)
[error] 	at $8952c887d8c4e436510f$.$anonfun$core$8(build.sbt:147)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)

[HC153Yuanyuan]

I still have publish fail issue on MacOS

• sbt : 1.6.2
• OS : Mac-os 12.2.1
• docker-OS: centos7

[warn] 	You need to remove it from the cache manually to take effect.
[info] 	published ivy to /root/.ivy2/local/com.github.spinalhdl/spinalhdl-sim_2.11/1.6.5/ivys/ivy.xml
[error] java.lang.RuntimeException: Nonzero exit value: 129
[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:138)
[error] 	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:108)
[error] 	at $8952c887d8c4e436510f$.liftedTree1$1(build.sbt:83)
[error] 	at $8952c887d8c4e436510f$.gitHash(build.sbt:82)
[error] 	at $8952c887d8c4e436510f$.$anonfun$core$8(build.sbt:147)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)

@jijingg
I have the same error when compiling spinalhdl 1.6.5 in centOS 6. How did you solve it ？

[dnychennnn]

I can confirm this when we tried to upgrade from 1.7.4 to the latest version.

• sbt : 1.6.2
• OS : Mac-os 13.0
• docker-OS: debian (openjdk:17-slim)

  val dockerDistribution: Seq[Setting[_]] = Seq(
    // Docker image configuration
    Docker / dockerRepository := Some(System.getProperty("app.docker.repository", "...")),
    dockerBaseImage := s"openjdk:${System.getProperty("app.java.version", "17")}-slim",
    Docker / dockerUpdateLatest := Try(System.getProperty("app.docker.update-latest").toBoolean).getOrElse(false),
    Docker / defaultLinuxInstallLocation := "/usr/src/app",
    dockerExposedPorts ++= Seq(3001, 2551, 19999),
    dockerCommands := Seq(
      Cmd("FROM", dockerBaseImage.value),
      Cmd("WORKDIR", (Docker / defaultLinuxInstallLocation).value),
      Cmd("COPY", "usr", "/usr"),
      Cmd("COPY", "1/usr", "/usr"),
      Cmd("COPY", "2/usr", "/usr"),
      ExecCmd("RUN", "chmod", "u+x", s"bin/${executableScriptName.value}"),
      ExecCmd("ENTRYPOINT", "bash", s"bin/${executableScriptName.value}"),
      Cmd("EXPOSE", "3001", "2551", "19999")
    )
  )