Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg_tar rule break the symlink and cause link error at runtime on MacOS #821

Open
bfsoyc opened this issue Feb 13, 2024 · 4 comments
Open

Comments

@bfsoyc
Copy link

bfsoyc commented Feb 13, 2024

The attribute include_runfiles = True is useful for me since I would like to keep things organized as they are in workspace dir tree.

However, I found the pkg_tar rule keep two copys of dynamic lib:

  • One in the dir tree as we declare them by BUILD file.
  • Another is located in _solib_darwin_arm64 for MacOS (_solib_k8 for Linux).

In the project workspace, the one in bazel-bin/_solib_darwin_arm64 is actually a symlink to the one in dir tree. However, in package tar both are duplicated files.

It introduces linking error when I run the binary. (The binary links the .so lib with relative path by using @loader_path)

This behavior makes no sense to me.

Could anyone tell if it's desired behavior or not? many thanks.

An simple example:

- bin/executable
- lib
  |- module_a
    |- a.so
  |- module_b
    |- deeper_dir
      |- b.so 

We have executable deps on b and b deps on a.

After pack with pkg_tar, we have the following in tar ball:

- _solib_darwin_arm64
  |- encoded_path_to_a
    |- a.so. # no longer a symlink to lib/module_a/a.so
  |- encoded_path_to_b
    |- b.so. # no longer a symlink
- bin/executable
- lib
  |- module_a
    |- a.so
  |- module_b
    |- deeper_dir
      |- b.so 

When running the executable binary, it try to link _solib_darwin_arm64/encoded_path_to_b/b.so, which is no longer a symlink but the same content with the origin one.
Therefore every relative path broken when b.so trying to load a.so by relative path.

@dieortin
Copy link
Contributor

Are those shared libraries prebuilt? Or are you building them with Bazel?

In my case, I’m packaging binaries with lots of shared libraries and haven’t encountered this problem. But I don’t have any prebuilt .so in my sources tree: most libraries are built by Bazel, and those that are not are included from external repositories.

@bfsoyc
Copy link
Author

bfsoyc commented Feb 15, 2024

Are those shared libraries prebuilt? Or are you building them with Bazel?

In my case, I’m packaging binaries with lots of shared libraries and haven’t encountered this problem. But I don’t have any prebuilt .so in my sources tree: most libraries are built by Bazel, and those that are not are included from external repositories.

Thanks for your attention at first.

We are working on a huge project, where there's some prebuilt .so library.

However, I think that not the point. As long as we have a binary require dynamic linking to lib B, and lib B require dynamic linking to A. The linking phase will fail as I mention above.
(I use otool -L to query the dynamic link of a binary or .so.)

# the way we generate a lib B, needed to link lib A at runtime.
cc_binary(
    name = "libb.so",
    srcs = ["b_static"],
    linkshared = 1,
    linkstatic = 1,
    visibility = ["//visibility:public"],
    deps = [
        "//lib/module_a:a",  # this explicit deps forces B linking to A, such that we can reproduce the problem.
    ],
)

So my point is, if we can preserve the object in _solib_darwin_arm64 symlink to the object in real dir tree.
The behavior can remain the same as they were in Bazel workspace.

@aiuto
Copy link
Collaborator

aiuto commented Feb 28, 2024

I don't know enough about how mac dynamic linking as you used it (with both linkstatic and linkshard) is supposed to work to comment yet. To me, the cc_binary example above seems like module_a should be statically linked.
Can you point to a small reproduction? Is module_a a cc_shared_library?

@bfsoyc
Copy link
Author

bfsoyc commented Mar 3, 2024

@aiuto , thanks for your attention.
FYI: https://github.com/bfsoyc/bar_repo/tree/main

bazel build //bin:deploy

tar -zxf bazel-bin/bin/deploy.tar --directory {test_dir}

cd {test_dir}
./bin/main  # this should crash due to library loading failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants