The associated file is either missing or is an invalid symlink.

[oh-tarnished]

Hey all, i am trying to build a multi-arch images for my project using rules_oci

Here's my MODULE.bazel using bazel version 7.0.2

bazel_dep(name = "aspect_bazel_lib", version = "2.4.1")
bazel_dep(name = "rules_oci", version = "1.7.2")

oci = use_extension("@rules_oci//oci:extensions.bzl", "oci")

oci.pull(
    name = "ubuntu_20.04",
    digest = "sha256:ed4a42283d9943135ed87d4ee34e542f7f5ad9ecf2f244870e23122f703f91c2",
    image = "docker.io/library/ubuntu",
    platforms = [
        "linux/amd64",
        "linux/arm64/v8",
    ],
)
use_repo(oci, "ubuntu_20.04")

Issue is while building the oci_image_index here is my BUILD.bazel file

load("@container_structure_test//:defs.bzl", "container_structure_test")
load("@rules_oci//oci:defs.bzl", "oci_image", "oci_image_index", "oci_push", "oci_tarball")
load("//bazel/docker:transition.bzl", "multi_arch")

oci_image(
    name = "image",
    base = "@ubuntu_20.04",
    cmd = [
        "saytime",
    ],
    entrypoint = ["bash"],
    tars = ["//bazel/package/cli:saytime_tar"],
)

multi_arch(
    name = "images",
    image = ":image",
    platforms = [
        "//platforms:linux_amd64",
        "//platforms:linux_arm64",
    ],
)

oci_image_index(
    name = "index",
    images = [
        ":images",
    ],
)

container_structure_test(
    name = "cli_docker_test",
    size = "small",
    configs = ["//bazel/docker/cli/tests:saytime_unit.yaml"],
    image = ":image",
)

Error Logs

developer@docker-desktop:/workspaces/droid-lab$ bazel build //bazel/docker/cli:index
INFO: Invocation ID: 75f84208-693e-4027-b2af-d9737af3eb07
INFO: Analyzed target //bazel/docker/cli:index (3 packages loaded, 958 targets configured).
WARNING: Remote Cache: Action output /home/developer/.cache/bazel/_bazel_developer/0a5c43b7e3021c7e26e00d79d66f2a33/execroot/_main/bazel-out/aarch64-fastbuild/bin/bazel/docker/cli/index/blobs/sha256/153b50335c298640651a77a03a9c64fc551c690d38e9eb25047ae3cd681dccc5 is a dangling symbolic link to /tmp/bazel-execroot/_main/bazel-out/aarch64-fastbuild-ST-4ee3311bb660/bin/bazel/docker/cli/image/blobs/sha256/153b50335c298640651a77a03a9c64fc551c690d38e9eb25047ae3cd681dccc5 
ERROR: /workspaces/droid-lab/bazel/docker/cli/BUILD.bazel:24:16: Error while validating output TreeArtifact File:[[<execution_root>]bazel-out/aarch64-fastbuild/bin]bazel/docker/cli/index : Failed to resolve relative path blobs/sha256/153b50335c298640651a77a03a9c64fc551c690d38e9eb25047ae3cd681dccc5 inside TreeArtifact /home/developer/.cache/bazel/_bazel_developer/0a5c43b7e3021c7e26e00d79d66f2a33/execroot/_main/bazel-out/aarch64-fastbuild/bin/bazel/docker/cli/index. The associated file is either missing or is an invalid symlink.
ERROR: /workspaces/droid-lab/bazel/docker/cli/BUILD.bazel:24:16: OCI Index //bazel/docker/cli:index failed: not all outputs were created or valid
Target //bazel/docker/cli:index failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.862s, Critical Path: 1.72s
INFO: 6 processes: 3 internal, 3 linux-sandbox.
ERROR: Build did NOT complete successfully

I'm unable to understand what is causing the issue i went thought this #425 issue. I even rolled backed the versions, didn't work.

[vicentsegui]

I'm getting a similar error in my repo, the oci_image_index fails on linux, but seems to work on macos M1.

[vicentsegui]

Algo got the same error when attempting to use Github Actions to enable CI for the repository.

[oh-tarnished]

Well i fixed it with this.

bazel_dep(name = "aspect_bazel_lib", version = "1.36.0")
bazel_dep(name = "rules_oci", version = "1.7.1")

# Right now we are using a fork of rules_oci until the bug is fixed from bazel team.
# The fork uses `yq` to pull and verify the checksum, instead of using the default `jq`
# `jq` with rules_oci has issues with linux arm64 hosts. (ie. MacOs arm64 hosts as well)
# The issue is tracked here: 
#   * https://github.com/bazel-contrib/rules_oci/issues/253
#   * https://bazelbuild.slack.com/archives/CA3NW13MH/p1707430756750789
# Date - 15-02-2024 @oh-tarnished
git_override(
    module_name = "rules_oci",
    commit = "e060819b2b19dd827f255900068e134b4914ff63",
    remote = "https://github.com/archen/rules_oci",
)

ext = use_extension("@aspect_bazel_lib//lib:extensions.bzl", "toolchains")
use_repo(ext, "yq_toolchains")  # For checksum verification

[vicentsegui]

Interesting, using --spawn_strategy=local seems to also fix the issue, but even more interesting is that once I have a successful run it will pass subsequent runs even with --spawn_strategy=linux-sandbox. I also attempted other sandboxes from a clean repo and got an interesting message when using the processwrapper about CopyFile:

bazel build --spawn_strategy=processwrapper-sandbox //...
INFO: Analyzed 82 targets (0 packages loaded, 0 targets configured).
WARNING: CopyFile uses implicit fallback from sandbox to local, which is deprecated because it is not hermetic. Prefer setting an explicit list of strategies, e.g., --strategy=CopyFile=sandboxed,standalone
ERROR: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/server/BUILD.bazel:130:19: Copying file server/server_image_linux_arm64 failed: CopyFile spawn cannot be executed with any of the available strategies: [processwrapper-sandbox]. Your --spawn_strategy, --genrule_strategy and/or --strategy flags are probably too strict. Visit https://github.com/bazelbuild/bazel/issues/7480 for advice
Use --verbose_failures to see the command lines of failed build steps.

Using verbose_failures doesn't output any further information. Note that in my case I'm building a multi arch image on a linux/amd64 host. The failure I get is:

bazel build --verbose_failures --spawn_strategy=linux-sandbox //server/... INFO: Analyzed 44 targets (0 packages loaded, 0 targets configured). ERROR: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/server/BUILD.bazel:130:19: Error while validating output TreeArtifact File:[[<execution_root>]bazel-out/k8-opt/bin]server/server_image : Failed to resolve relative path blobs/sha256/26e3e4b0848c0e87391a7a844b11fcd2e6d223bd516be6ac8b17bac4e6808609 inside TreeArtifact /usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/execroot/_main/bazel-out/k8-opt/bin/server/server_image. The associated file is either missing or is an invalid symlink. ERROR: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/server/BUILD.bazel:130:19: OCI Index //server:server_image failed: not all outputs were created or valid INFO: Elapsed time: 0.553s, Critical Path: 0.34s INFO: 3 processes: 2 internal, 1 linux-sandbox. ERROR: Build did NOT complete successfully

[vicentsegui]

I tried #501 (comment) but got a different error:

bazel build --verbose_failures //server/... INFO: Repository distroless_base_linux_arm64 instantiated at: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/WORKSPACE:27:9: in <toplevel> /usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/external/rules_oci~override/oci/pull.bzl:120:22: in oci_pull Repository rule oci_pull defined at: /usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/external/rules_oci~override/oci/private/pull.bzl:436:27: in <toplevel> ERROR: An error occurred during the fetch of repository 'distroless_base_linux_arm64': Traceback (most recent call last): File "/usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/external/rules_oci~override/oci/private/pull.bzl", line 356, column 36, in _oci_pull_impl downloader = _create_downloader(rctx) File "/usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/external/rules_oci~override/oci/private/pull.bzl", line 276, column 40, in _create_downloader "config": json.decode(rctx.read(rctx.attr.config)), Error in read: Unable to load package for @@oci_auth_config//:config.json: The repository '@@oci_auth_config' could not be resolved: Repository '@@oci_auth_config' is not defined ERROR: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/WORKSPACE:27:9: fetching oci_pull rule //external:distroless_base_linux_arm64: Traceback (most recent call last): File "/usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/external/rules_oci~override/oci/private/pull.bzl", line 356, column 36, in _oci_pull_impl downloader = _create_downloader(rctx) File "/usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/external/rules_oci~override/oci/private/pull.bzl", line 276, column 40, in _create_downloader "config": json.decode(rctx.read(rctx.attr.config)), Error in read: Unable to load package for @@oci_auth_config//:config.json: The repository '@@oci_auth_config' could not be resolved: Repository '@@oci_auth_config' is not defined INFO: Repository rules_oci~override~oci~oci_crane_linux_amd64 instantiated at: <builtin>: in <toplevel> Repository rule crane_repositories defined at: /usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/external/rules_oci~override/oci/repositories.bzl:51:37: in <toplevel> ERROR: no such package '@@distroless_base_linux_arm64//': Unable to load package for @@oci_auth_config//:config.json: The repository '@@oci_auth_config' could not be resolved: Repository '@@oci_auth_config' is not defined ERROR: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/server/BUILD.bazel:130:19: //server:server_image_linux_arm64 depends on @@distroless_base_linux_arm64//:distroless_base_linux_arm64 in repository @@distroless_base_linux_arm64 which failed to fetch. no such package '@@distroless_base_linux_arm64//': Unable to load package for @@oci_auth_config//:config.json: The repository '@@oci_auth_config' could not be resolved: Repository '@@oci_auth_config' is not defined ERROR: Analysis of target '//server:server_image' failed; build aborted: Analysis failed INFO: Elapsed time: 0.651s, Critical Path: 0.00s INFO: 1 process: 1 internal. ERROR: Build did NOT complete successfully FAILED: Fetching https://github.com/google/go-containerregistry/releases/download/v0.14.0/go-containerregistry_Linux_x86_64.tar.gz

[thesayyn]

This is due to Bazel bug i believe.

in bazel-lib 2.x we don't have any action requirements to make copy_to_directory work with BWOB https://github.com/aspect-build/bazel-lib/blob/aa6a58f73cfc7b6833e2d5f7a0c289f293e96637/lib/private/copy_common.bzl#L4

in bazel-lib 1.36.0 this should work.

[vicentsegui]

I did a quick try with bazel lib 1.36.0:

bazel_dep(name = "aspect_bazel_lib", version = "1.36.0")

But still getting error:

ERROR: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/simple_client/BUILD.bazel:30:19: Error while validating output TreeArtifact File:[[<execution_root>]bazel-out/k8-opt/bin]simple_client/simple_client_image : Failed to resolve relative path blobs/sha256/383e1c5dd0c1830143b1230e90292ebd4219911e0512b70d250c8907c4899110 inside TreeArtifact /usr/local/google/home/segui/.cache/bazel/_bazel_segui/2573d3d54a61eba68f67ce21195763d5/execroot/_main/bazel-out/k8-opt/bin/simple_client/simple_client_image. The associated file is either missing or is an invalid symlink. ERROR: /usr/local/google/home/segui/Projects/bower/bower-service-ml-mixer-go/simple_client/BUILD.bazel:30:19: OCI Index //simple_client:simple_client_image failed: not all outputs were created or valid INFO: Elapsed time: 2.833s, Critical Path: 1.51s INFO: 649 processes: 295 internal, 128 linux-sandbox, 226 local. ERROR: Build did NOT complete successfully

[vicentsegui]

Another data point, I'm using the rules in a bazel macro, if I add the tags=["no-sandbox"] to the oci rules it also seems to work.

[thesayyn]

#525 fixes this. Can someone try and see if it fixes this?

[ghost]

I don't know if I'm exactly in this same case because I don't have exactly the same error but it's certainly due to the difference between bazel 7.0 and 7.1.

Bazel: 7.1.0 (non bzlmod)
rules_oci: 1.7.4
aspect_bazel_lib: 2.5.3

I have a simple rule returning all OCI images created by a rule equivalent to oci_image with custom transition:

def _impl(ctx):
    return [DefaultInfo(files = depset(ctx.files.image))]

_multi_arch_images = rule(
    implementation = _impl,
    attrs = {
       ...
        "image": attr.label(
            cfg = _multi_arch_images_transition,
            mandatory = True,
            allow_files = True,
        ),
        "_allowlist_function_transition": attr.label(
            default = "@bazel_tools//tools/allowlists/function_transition_allowlist",
        ),
    },
)

No problem to build all these images. Blobs files are all ok.

And I have a oci_image_index rule taking these images as inputs:

oci_image_index(
    ...
    images = [multi_arch_images_rule_name],
    ...
)

$ bazel build <rule>
ERROR: .../BUILD.bazel:87:17: Error while validating output TreeArtifact File:[[<execution_root>]bazel-out/k8-fastbuild/bin]<name> : Child blobs/sha256/4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f of tree artifact <execroot>/<workspace>/bazel-out/k8-fastbuild/.../ is a dangling symbolic link

All blobs files are symbolic links targetting non existing files in /tmp.

It's ok with --noincompatible_sandbox_hermetic_tmp option.

The patch #525, applied to rules_oci does not seem to fix this issue in my case:
cat: bazel-out/k8-fastbuild/.../blobs/sha256/0601a2e7a519c9044fb8f6e9f3423b2e454b15ceb2c5791b37ea4961a9d1a247: No such file or directory (os error 2)

EDIT: I didn't say anything. #525 works fine 👌

[thesayyn]

EDIT: I didn't say anything. #525 works fine 👌

Can i close this now?

[vicentsegui]

I'm ok with closing for now, I have a workaround in place. I will wait for #525 to be released and once I update I will try to remove the workarounds and see if it fixes, if not I'll reopen the issue.

[thesayyn]

This is released in https://github.com/bazel-contrib/rules_oci/releases/tag/v1.7.5

Feel free to reopen if it's still an issue.