Error validating output TreeArtifact File when building an oci_image #425
Labels
need: investigation
A potential issue which we need to investigate first
Comments
|
Hm, I guess something about this |
alexeagle
added
the
need: investigation
|
Hmm interesting, there are no sha256 files. $ docker pull mcr.microsoft.com/dotnet/runtime-deps@sha256:24554fadd483d8305974ded44bb1dbe4916e2f02500b9e2d78e7beb557cfebd0
$ docker save mcr.microsoft.com/dotnet/runtime-deps@sha256:24554fadd483d8305974ded44bb1dbe4916e2f02500b9e2d78e7beb557cfebd0 > image.tar
$ tar -xvf ./image.tar
9629ad6e05e05dcbe2efeb2e21050346d75e7716a89097e1a99a6128644fdb7f.json
d3010e0e22efbe19c9653b0d1f5b923649a3bb9a809b1024e3557b1a0a4cd51a/
d3010e0e22efbe19c9653b0d1f5b923649a3bb9a809b1024e3557b1a0a4cd51a/VERSION
d3010e0e22efbe19c9653b0d1f5b923649a3bb9a809b1024e3557b1a0a4cd51a/json
d3010e0e22efbe19c9653b0d1f5b923649a3bb9a809b1024e3557b1a0a4cd51a/layer.tar
eb18f9a708a4f682536a7eaf85f73a9f706040047016a3a41a974cbbe9829323/
eb18f9a708a4f682536a7eaf85f73a9f706040047016a3a41a974cbbe9829323/VERSION
eb18f9a708a4f682536a7eaf85f73a9f706040047016a3a41a974cbbe9829323/json
eb18f9a708a4f682536a7eaf85f73a9f706040047016a3a41a974cbbe9829323/layer.tar
manifest.json
$ docker inspect mcr.microsoft.com/dotnet/runtime-deps@sha256:24554fadd483d8305974ded44bb1dbe4916e2f02500b9e2d78e7beb557cfebd0 [
{
"Id": "sha256:9629ad6e05e05dcbe2efeb2e21050346d75e7716a89097e1a99a6128644fdb7f",
"RepoTags": [
"bazel/image:image"
],
"RepoDigests": [
"mcr.microsoft.com/dotnet/runtime-deps@sha256:24554fadd483d8305974ded44bb1dbe4916e2f02500b9e2d78e7beb557cfebd0"
],
"Parent": "",
"Comment": "",
"Created": "2022-10-25T04:25:58.281747234Z",
"Container": "802335d3db455e68b4d872658e6bd032f7a56f24a0886d80f443c58cd37993f6",
"ContainerConfig": {
"Hostname": "802335d3db45",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"ASPNETCORE_URLS=http://+:80",
"DOTNET_RUNNING_IN_CONTAINER=true"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"ENV ASPNETCORE_URLS=http://+:80 DOTNET_RUNNING_IN_CONTAINER=true"
],
"Image": "sha256:1b869bb0bbfb8680c1b72689cfbef8d84eff5bc63f289b7038e1991506c2122f",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"DockerVersion": "20.10.17",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"ASPNETCORE_URLS=http://+:80",
"DOTNET_RUNNING_IN_CONTAINER=true"
],
"Cmd": [
"bash"
],
"Image": "sha256:1b869bb0bbfb8680c1b72689cfbef8d84eff5bc63f289b7038e1991506c2122f",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 116760754,
"VirtualSize": 116760754,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/0391c998bfdda0d14df4e3bdfbcb12ac4ee6432650f64a7e01c115ba32c6ce2a/diff",
"MergedDir": "/var/lib/docker/overlay2/6b7df02ff02ff0ee2239f98b246ff4eadba62996ce1ec8c9bfa976a06199aa90/merged",
"UpperDir": "/var/lib/docker/overlay2/6b7df02ff02ff0ee2239f98b246ff4eadba62996ce1ec8c9bfa976a06199aa90/diff",
"WorkDir": "/var/lib/docker/overlay2/6b7df02ff02ff0ee2239f98b246ff4eadba62996ce1ec8c9bfa976a06199aa90/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:a12586ed027fafddcddcc63b31671f406c25e43342479fc92a330e7e30d65f2e",
"sha256:55125ebb892069bd16e8dedc05e382be6a818ed99026f2e5f8578852b7c6d1cb"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
] |
|
hmm it's weird. looks like the blobs directory created by is missing. i wonder if the image is using different hash algo than sha256 |
|
We are running into the same issue, but using a different base image: The odd thing is that it works with version 1.4.0, but 1.4.2 and 1.4.3 fail. Not sure if it helps, but the generated file by oci_pul mentions that same shasum as being the 'manifest.json' being added to the container image. Could it be that that file is not present in the container image yet? |
|
I am migrating from |
|
@diedoman i am able to successfully build what you pasted above. Note that your platforms definition was incorrect. you needed |
|
I am not able to reproduce this... |
|
Same issue on my side. I will try to provide logs later on, but I guess this could be problem with |
|
Yes, this seems to be a bazel 7 thing. Example MODULE.bazel: module(
name = "bazel_test_area",
)
bazel_dep(name = "rules_oci", version = "1.4.3")
oci = use_extension("@rules_oci//oci:extensions.bzl", "oci")
oci.pull(
name = "distroless_base",
digest = "sha256:b31a6e02605827e77b7ebb82a0ac9669ec51091edd62c2c076175e05556f4ab9",
image = "gcr.io/distroless/base",
platforms = [
"linux/amd64",
"linux/arm/v7",
"linux/arm64/v8",
"linux/ppc64le",
"linux/s390x",
],
)
use_repo(oci, "distroless_base")With 7.0.0: With 6.4.0: Oddly, despite the message that cat /home/jrockway/.cache/bazel/_bazel_jrockway/73bbccb2c6bf9ab4d70b0ba8478635bc/execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_oci~1.4.3~oci~distroless_base_linux_amd64/blobs/sha256/6c871aa3c9019984dfd7f520635bd658d740ad20c6268a82faa433f69dfc9a0b
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":1596,"digest":"sha256:2e46274151... |
|
@jrockway since you have that repro handy, do you mind trying the 7.0.0rcX releases to see if there's a value of X where it's introduced - that would help us bisect |
|
Good idea! 7.0.0rc3: good. 7.0.0rc4: bad. Full transcript: (I don't think the |
|
Using bazel 7 stable release and facing this same issue MODULE.bazel bazel_dep(name = "rules_oci", version = "1.4.3")
oci = use_extension("@rules_oci//oci:extensions.bzl", "oci")
oci.pull(
name = "distroless",
digest = "sha256:6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b25",
image = "gcr.io/distroless/static",
platforms = [
"linux/amd64",
"linux/arm64/v8",
],
)
# For each oci.pull call, repeat the "name" here to expose them as dependencies.
use_repo(oci, "distroless")WORKSPACE # Declares that this directory is the root of a Bazel workspace.
# See https://docs.bazel.build/versions/main/build-ref.html#workspace
workspace(
# How this workspace would be referenced with absolute labels from another workspace
name = "driver-service",
)
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
## GO
http_archive(
name = "io_bazel_rules_go",
sha256 = "d6ab6b57e48c09523e93050f13698f708428cfd5e619252e369d377af6597707",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.43.0/rules_go-v0.43.0.zip",
"https://github.com/bazelbuild/rules_go/releases/download/v0.43.0/rules_go-v0.43.0.zip",
],
)
http_archive(
name = "bazel_gazelle",
sha256 = "b7387f72efb59f876e4daae42f1d3912d0d45563eac7cb23d1de0b094ab588cf",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.34.0/bazel-gazelle-v0.34.0.tar.gz",
"https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.34.0/bazel-gazelle-v0.34.0.tar.gz",
],
)
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")
load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
############################################################
# Define your own dependencies here using go_repository.
# Else, dependencies declared by rules_go/gazelle will be used.
# The first declaration of an external repository "wins".
############################################################
load("//:go_deps.bzl", "go_dependencies")
# gazelle:repository_macro go_deps.bzl%go_dependencies
go_dependencies()
go_rules_dependencies()
go_register_toolchains(version = "1.20.5")
gazelle_dependencies()
gazelle_dependencies(go_repository_default_config = "//:WORKSPACE")
# ## Buf.build
http_archive(
name = "rules_buf",
sha256 = "523a4e06f0746661e092d083757263a249fedca535bd6dd819a8c50de074731a",
strip_prefix = "rules_buf-0.1.1",
urls = [
"https://github.com/bufbuild/rules_buf/archive/refs/tags/v0.1.1.zip",
],
)
load("@rules_buf//buf:repositories.bzl", "rules_buf_dependencies", "rules_buf_toolchains")
rules_buf_dependencies()
rules_buf_toolchains(version = "v1.27.0")
load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
load("//:buf_deps.bzl", "buf_deps")
# gazelle:repository_macro buf_deps.bzl%buf_deps
buf_deps()
rules_proto_dependencies()
rules_proto_toolchains()
load("@rules_buf//gazelle/buf:repositories.bzl", "gazelle_buf_dependencies")
gazelle_buf_dependencies()Error: ERROR: /home/runner/.cache/bazel/_bazel_runner/83ec04a7f9edb0d4347368d686832579/external/rules_oci~1.4.3~oci~distroless_linux_amd64/BUILD.bazel:16:18: Error while validating output TreeArtifact File:[[<execution_root>]bazel-out/k8-fastbuild/bin]external/rules_oci~1.4.3~oci~distroless_linux_amd64/blobs/sha256 : Failed to resolve relative path d49f214e6f1bae819e24f651156552b073725592cae128a66eade0c6280f02e1 inside TreeArtifact /home/runner/.cache/bazel/_bazel_runner/83ec04a7f9edb0d4347368d686832579/execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_oci~1.4.3~oci~distroless_linux_amd64/blobs/sha256. The associated file is either missing or is an invalid symlink.
ERROR: /home/runner/.cache/bazel/_bazel_runner/83ec04a7f9edb0d4347368d686832579/external/rules_oci~1.4.3~oci~distroless_linux_amd64/BUILD.bazel:16:18: Copying files to directory rules_oci~1.4.3~oci~distroless_linux_amd64/blobs/sha256 failed: not all outputs were created or validAlthough the notable aspect is it's only happening in github action runner and not on local macOS machine even after I was able to retrieve the BUILD.bazel file causing this error "Generated by oci_pull"
load("@aspect_bazel_lib//lib:copy_to_directory.bzl", "copy_to_directory")
load("@aspect_bazel_lib//lib:copy_file.bzl", "copy_file")
load("@aspect_bazel_lib//lib:jq.bzl", "jq")
load("@bazel_skylib//rules:write_file.bzl", "write_file")
package(default_visibility = ["//visibility:public"])
copy_file(
name = "manifest",
src = "manifest.json",
out = "d49f214e6f1bae819e24f651156552b073725592cae128a66eade0c6280f02e1"
)
copy_to_directory(
name = "blobs",
# TODO(https://github.com/bazel-contrib/rules_oci/issues/73): other hash algorithms
out = "blobs/sha256",
include_external_repositories = ["*"],
srcs = ["07a64a71e01156f8f99039bc246149925c6d1480d3957de78510bbec6ec68f7a", "fe5ca62666f04366c8e7f605aa82997d71320183e99962fa76b3209fdfbb8b58", "b02a7525f878e61fc1ef8a7405a2cc17f866e8de222c1c98fd6681aff6e509db", "fcb6f6d2c9986d9cd6a2ea3cc2936e5fc613e09f1af9042329011e43057f3265", "e8c73c638ae9ec5ad70c49df7e484040d889cca6b4a9af056579c3d058ea93f0", "1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c", "4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f", "7c881f9ab25e0d86562a123b5fb56aebf8aa0ddd7d48ef602faf8d1e7cf43d8c", "5627a970d25e752d971a501ec7e35d0d6fdcd4a3ce9e958715a686853024794a"] + [
":d49f214e6f1bae819e24f651156552b073725592cae128a66eade0c6280f02e1",
":121a1df34efd36214f556caf5b11c7ec97a53f64285708d860aa175109bf7e6c",
],
)
copy_to_directory(
name = "distroless_linux_amd64",
out = "layout",
include_external_repositories = ["*"],
srcs = [
"blobs",
"oci-layout",
"index.json",
],
) |
|
Was able to identify the breaking commit (for our code base at least): |
|
Boiled it down to the Adding this to the |
|
Here's that delta bazel-contrib/bazel-lib@v1.35.0...v1.36.0 - I would guess bazel-contrib/bazel-lib#594 is the one that fixed it. |
alexeagle
added a commit
that referenced
this issue
Dec 18, 2023
|
I'm getting a similar issue: The problem is a symlink pointing to the wrong place: In particular, |
|
Yes, the 1.36.0 upgrade solves the issue. |
alexeagle
added a commit
that referenced
this issue
Dec 22, 2023
Pick up the change to execution requirements. Fixes #425
alexeagle
added a commit
that referenced
this issue
Dec 22, 2023
Pick up the change to execution requirements. Fixes #425
alexeagle
added a commit
that referenced
this issue
Jan 12, 2024
Pick up the change to execution requirements. Fixes #425
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am trying to migrate from
rules_dockerbut am running into some issues.Here is my setup:
.bazelversionWORKSPACEBUILD.bazelThe error:
The text was updated successfully, but these errors were encountered: