Support manifest list specified by digest #1360

dzwicker · 2018-12-18T10:07:25Z

Description of the issue:
As docker tags can be reference different container digest over time we want to use a digest for the base image.

Expected behavior:
Downloading the base image and build the new application image.

Steps to reproduce:

Use a digest instead of a tag for the from image
call gradle jib

Environment:

JVM: 11.0.1 (AdoptOpenJDK 11.0.1+13)
OS: Mac OS X 10.14.2 x86_64
Gradle: 5.0

jib-gradle-plugin Configuration:

jib {
    from {
        // image = 'openjdk:11-jre-slim'
        image = 'openjdk@sha256:8ab7b3078b01ba66b937b7fbe0b9eccf60449cc101c42e99aeefaba0e1781155'
    }
    to {
        image = "repo/project/${rootProject.name}-${project.name}:${project.version.toString().replaceAll('\\+', '_')}"
    }
}

Additional info:
The problem is that the docker registry returns an unknown mediatype for the mainfest. It is application/vnd.docker.distribution.manifest.list.v2+json. The list is the problem as it is unknown by the ManifestPuller.

Log output:


Getting base image openjdk@sha256:8ab7b3078b01ba66b937b7fbe0b9eccf60449cc101c42e99aeefaba0e1781155...
Building dependencies layer...
Building snapshot dependencies layer...
Building resources layer...
Building classes layer...
The base image requires auth. Trying again for openjdk@sha256:8ab7b3078b01ba66b937b7fbe0b9eccf60449cc101c42e99aeefaba0e1781155...
Retrieving registry credentials for registry.hub.docker.com...
No credentials could be retrieved for registry registry.hub.docker.com

> Task :sp-rest:jibDockerBuild
Task ':sp-rest:jibDockerBuild' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
Adding corresponding output directories of source sets to image
        '/Users/dzwicker/authada/backend/eservice-v3/sp/sp-rest/build/classes/java/main' (not found, skipped)
        '/Users/dzwicker/authada/backend/eservice-v3/sp/sp-rest/build/classes/kotlin/main'

> Task :sp-rest:jibDockerBuild FAILED
:sp-rest:jibDockerBuild (Thread[Execution worker for ':' Thread 7,5,main]) completed. Took 3.127 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sp-rest:jibDockerBuild'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build to Docker daemon failed

* Try:
Run with --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':sp-rest:jibDockerBuild'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:96)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:65)
        at org.gradle.api.internal.tasks.execution.ActionEventFiringTaskExecuter.execute(ActionEventFiringTaskExecuter.java:44)
        at org.gradle.api.internal.tasks.execution.TimeoutTaskExecuter.execute(TimeoutTaskExecuter.java:53)
        at org.gradle.api.internal.tasks.execution.SnapshotAfterExecutionTaskExecuter.execute(SnapshotAfterExecutionTaskExecuter.java:38)
        at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:49)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:61)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:44)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:35)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:49)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:44)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:325)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:318)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:304)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: org.gradle.internal.UncheckedException: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build to Docker daemon failed
        at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:66)
        at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:40)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:76)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:48)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:41)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:704)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:671)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:117)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:106)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:85)
        ... 35 more
Caused by: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build to Docker daemon failed
        at com.google.cloud.tools.jib.plugins.common.BuildStepsRunner.build(BuildStepsRunner.java:274)
        at com.google.cloud.tools.jib.gradle.BuildDockerTask.buildDocker(BuildDockerTask.java:142)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
        ... 48 more
Caused by: com.google.cloud.tools.jib.image.json.UnknownManifestFormatException: Unknown mediaType: application/vnd.docker.distribution.manifest.list.v2+json
        at com.google.cloud.tools.jib.registry.ManifestPuller.getManifestTemplateFromJson(ManifestPuller.java:141)
        at com.google.cloud.tools.jib.registry.ManifestPuller.handleResponse(ManifestPuller.java:82)
        at com.google.cloud.tools.jib.registry.ManifestPuller.handleResponse(ManifestPuller.java:40)
        at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:234)
        at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.callWithAllowInsecureRegistryHandling(RegistryEndpointCaller.java:152)
        at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:142)
        at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:339)
        at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest(RegistryClient.java:200)
        at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest(RegistryClient.java:208)
        at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.pullBaseImage(PullBaseImageStep.java:197)
        at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:168)
        at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:56)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:127)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:80)

The text was updated successfully, but these errors were encountered:

coollog · 2018-12-18T16:04:54Z

Hi @dzwicker , thanks for reporting this! Jib should be able to support using digests in addition to tags, so this seems like it is not working as intended.

coollog · 2019-01-17T21:19:50Z

So it seems that this is happening because you are using explicitly the digest for a manifest list for openjdk. This issue would be solved by using the digest for the specific manifest that is for the linux/amd64 architecture, which for your list would be openjdk@sha256:448cc779731236274982047655896a04e53673c448084a565c93fa772600aad4.

chanseokoh · 2019-03-15T15:35:50Z

Using the digest for a specific manifest will resolve the issue. However, I see docker pull works even on the digest for a manifest list (perhaps it ~~goes over the list and picks up a digest for the current OS/architecture~~ UPDATED: looks like the Docker Hub registry returns a Linux/amd64 manifest by default), so one could argue that ideally Jib should work similarly. But at the same time, this magic will undermine reproducibility to some extent, since builds will pick up different images on different architectures. But printing out a warning (as we currently do) that the build may not be reproducible could work.

briandealwis · 2019-03-21T19:23:13Z

The Docker Registry was changed such that if a client requests a tag that maps to a manifest-list but they don't support manifest-lists, then it returns a manifest corresponding to the defaultOS/defaultArch (linux/amd64). But if you reference a specific digest then you get that digest.

Unfortunately if you use the docker CLI to pull a something that has a manifest list, docker reports the digest of the manifest-list:

$ docker pull openjdk:11-jre-slim
11-jre-slim: Pulling from library/openjdk
[...]
Digest: sha256:afb3a7cd7214ac1731231ff5d0246615d50a68b43cdf8b89e9d56e5b8bbee7c5
Status: Downloaded newer image for openjdk:11-jre-slim

$ crane manifest openjdk@sha256:afb3a7cd7214ac1731231ff5d0246615d50a68b43cdf8b89e9d56e5b8bbee7c5
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
[...]

loosebazooka · 2019-06-28T18:10:35Z

So this issue should be fixed by #1811. Going to close for now, expect it to work with the 1.4.0 release

TadCordle · 2019-07-16T16:18:41Z

@dzwicker v1.4.0 has been released with manifest list support!

loosebazooka · 2019-07-16T19:10:56Z

@dzwicker it's limited support, but should work for your case (it will pick the amd64/linux base image from a manifest list automatically)

coollog added the priority: p1 label Dec 18, 2018

TadCordle removed the priority: p1 label Jan 17, 2019

marshallpierce mentioned this issue Jan 24, 2019

Configuring a base image built earlier in the Gradle build #1446

Closed

briandealwis mentioned this issue Mar 19, 2019

Support referencing base images from a manifest list #1567

Closed

1 task

chanseokoh added this to the v1.4.0 milestone Jun 25, 2019

loosebazooka mentioned this issue Jun 26, 2019

Support manifest lists when referenced by sha256 #1811

Merged

loosebazooka closed this as completed Jun 28, 2019

chanseokoh changed the title ~~Use digest instead of version tag for baseimage~~ Support manifest list specified by digest Jul 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support manifest list specified by digest #1360

Support manifest list specified by digest #1360

dzwicker commented Dec 18, 2018 •

edited

Loading

coollog commented Dec 18, 2018

coollog commented Jan 17, 2019 •

edited by chanseokoh

Loading

chanseokoh commented Mar 15, 2019 •

edited

Loading

briandealwis commented Mar 21, 2019

loosebazooka commented Jun 28, 2019

TadCordle commented Jul 16, 2019

loosebazooka commented Jul 16, 2019

Support manifest list specified by digest #1360

Support manifest list specified by digest #1360

Comments

dzwicker commented Dec 18, 2018 • edited Loading

coollog commented Dec 18, 2018

coollog commented Jan 17, 2019 • edited by chanseokoh Loading

chanseokoh commented Mar 15, 2019 • edited Loading

briandealwis commented Mar 21, 2019

loosebazooka commented Jun 28, 2019

TadCordle commented Jul 16, 2019

loosebazooka commented Jul 16, 2019

dzwicker commented Dec 18, 2018 •

edited

Loading

coollog commented Jan 17, 2019 •

edited by chanseokoh

Loading

chanseokoh commented Mar 15, 2019 •

edited

Loading