Dynamically linked cc binary doesn't work in genrule with remote execution

[meteorcloudy]

Consider the following example:
Under directory <workspace>/foo:

cc_binary(
  name = "main",
  srcs = ["main.cc"],
  deps = ["//foo"],
)

genrule(
  name = "run",
  cmd = "$(location :main) > $@",
  outs = ["output"],
  tools = [":main"],
)

cc_binary(
  name = "foo.so",
  srcs = ["foo.cc"],
  linkstatic = True,
  linkshared = True,
)

cc_import(
  name = "foo",
  shared_library = ":foo.so",
)

bazel build :run will succeed locally but fail with remote execution with:

bazel-out/k8-opt/bin/foo/main: error while loading shared libraries: foo.so: cannot open shared object file: No such file or directory

The reason is that //:main is dynamically linked to foo.so, and the rpath is ../_solib_local/_U_S_Sfoo_Cfoo___Ufoo/foo.so.
Locally it works, because bazel-out/k8-opt/_solib_local is always created for all so libs.

$ ldd bazel-out/k8-opt/bin/foo/main
	linux-vdso.so.1 (0x00007ffda3dc6000)
	foo.so => /home/kbuilder/workdir/tensorflow/bazel-out/k8-opt/bin/foo/../_solib_local/_U_S_Sfoo_Cfoo___Ufoo/foo.so 
(0x00007f3d84491000)

But with remote execution, the same relative path is not created. Therefore the foo.so cannot be found.

However, inside the runfiles tree of //:main, the so file exists (also in remote execution), but you'll have run the main binary from the runfiles tree to make it work. So in genrule, you can use "$(location :main).runfiles/<workspace name>/foo/main > $@", but this is obviously a hack.

$ tree bazel-bin/foo/main.runfiles
bazel-bin/foo/main.runfiles
├── MANIFEST
└── org_tensorflow
    ├── _solib_local
    │   └── _U_S_Sfoo_Cfoo___Ufoo
    │       └── foo.so -> /home/kbuilder/.cache/bazel/_bazel_kbuilder/b35e56146bb385d966771bab3ed3e8cb/execroot/org_tensorflow/bazel-out/k8-opt/bin/_solib_local/_U_S_Sfoo_Cfoo___Ufoo/foo.so
    └── foo
        └── main -> /home/kbuilder/.cache/bazel/_bazel_kbuilder/b35e56146bb385d966771bab3ed3e8cb/execroot/org_tensorflow/bazel-out/k8-opt/bin/foo/main

Possible solutions:

• Create the _solib_local directory for all RBE actions, but how do we figure out which so libs should be there?
• Have a similar function to locate the runfiles path of the binary in genrule, like $(rlocation //:foo)
• Write a wrapper script around the cc binary to run it from the runfiles directory, then use it in the genrule.
• Add extra rpath to locate the so file from the runfiles directory. eg. linkopts = ["-Wl,-rpath,$$ORIGIN/main.runfiles/org_tensorflow/_solib_local/_U_S_Sfoo_Cfoo___Ufoo"],

[meteorcloudy]

FYI @coeuvre @oquenchil

[keith]

Sounds like this might be the same case as #13819 ?

[meteorcloudy]

@keith Yes, thanks for the link, looks like the same issue.

[quval]

I'm not sure this is the same issue... The problem there was missing rpath entries, whereas here it's about resolving the path of the tool to run.

As far as I understand,

1. Given the way the runfiles libraries work, Bazel-built binaries expect to find a runfiles directory at $0.runfiles (this is a potential problem with the rlocation solution).
2. rpath entries are relative to the location of the actual binary, and given the way the rpath is set, Bazel-built binaries expect to be located under the runfiles dir (theoretically rpath entries could be added to support running from both outside and inside the runfiles dir, but this would double the number of rpath entries, which is rather wasteful).

This leads me to think a general solution has to be something like putting a symlink to or a wrapper script outside the runfiles dir, pointing to or invoking the actual binary under the runfiles dir. This is obviously far from trivial. Not wanting to diverge too much from mainline Bazel, our solution so far was to do the former in packaging rules, and if we'd ever need to run dynamically linked binaries in genrules, I imagine we'd go with the manual wrapper script.

[pcjanzen]

another related issue: #10809

[moroten]

Any progress on this issue? I've been running bazel test --remote_executor=... @abseil-hello//:hello_test and that of course fails.

[meteorcloudy]

/cc @tjgq @coeuvre

[coeuvre]

No progress since I don't have capacity to work on this yet. PR is welcome.

[cameron-martin]

This is also an issue when executing a binary from a custom rule, using ctx.actions.run. I think this stops possible solutions 2 & 3 from being viable.

[jheaff1]

I created a runnable_binary macro in rules_foreign_cc to facilitate the usage of dynamically linked binaries in build rules. Hope it helps.

[fmeum]

@meteorcloudy Your example works with Bazel 6.0.0, presumably due to c7aa392.

[meteorcloudy]

Good to know, I'm glad my bug lived long enough to be fixed.
/cc @oquenchil I'll assign this issue to you and close this one now, feel free to open if you think there are more issues to fix.