Skip to content

Conversation

rickeylev
Copy link
Collaborator

@rickeylev rickeylev commented Oct 8, 2025

It seems $ORIGIN resolves prior to symlink resolution. This makes it resolve
differently depending on if the directory or file itself is symlinked.

To fix, special case shared libraries and have them symlinked directly. Since an
explicit file is the target, VenvSymlinkEntry.link_to_file is added to hold the
File object that will be linked to.

An unfortunate side-effect of this logic is any package with lib*.so files
will be more expensive to build (depset flattened at analysis time, more files
symlinked), but it beats not working at all. Optimizing that can be done in
another change.

Tests added to generate libraries that look like what something from PyPI
does. Manually verified a case using jax and jax plugins.

Fixes #3228

@rickeylev rickeylev added the do not merge Tag that prevents merging label Oct 8, 2025
@rickeylev rickeylev force-pushed the fix.symlink.shared.libs.directly branch 4 times, most recently from 9b137a5 to f044f34 Compare October 9, 2025 21:40
@rickeylev rickeylev force-pushed the fix.symlink.shared.libs.directly branch from f044f34 to 438076b Compare October 9, 2025 21:44
@rickeylev
Copy link
Collaborator Author

@hartikainen Give this a try? I tried this with your repro and things appear to work (I don't have a gpu on my machine, so the jax call to get gpu devices failed, but otherwise everything imported successfully). This commit has the patch to apply: https://github.com/bazel-contrib/rules_python/commit/add0c262ed6cabe50df9266157ac74aca384f3f9.patch

Otherwise, this is almost ready, just need to do some cleanup for CI.

@rickeylev rickeylev removed the do not merge Tag that prevents merging label Oct 11, 2025
@aignas aignas mentioned this pull request Oct 11, 2025
7 tasks
@rickeylev rickeylev marked this pull request as ready for review October 11, 2025 04:13
@rickeylev rickeylev requested a review from aignas as a code owner October 11, 2025 04:13
@rickeylev
Copy link
Collaborator Author

PTAL. Cleaned up and ready

Copy link
Collaborator

@aignas aignas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a CHANGELOG item to mark that this has been fixed?

@rickeylev
Copy link
Collaborator Author

@gemini-code-assist /review

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses an issue with shared library loading in venvs by introducing a mechanism to symlink them directly, which is a solid approach to resolving the problem with $ORIGIN resolution. The changes are well-structured and include comprehensive tests for the new logic, which is excellent.

I've identified a critical issue: some leftover debugging code that will cause build failures. Additionally, there's a high-severity issue regarding platform-specific logic that needs to be addressed for correctness on Windows, and a minor typo in a new test file. Once these points are addressed, the PR will be in great shape.

Copy link
Collaborator Author

@rickeylev rickeylev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL

@aignas aignas added this pull request to the merge queue Oct 11, 2025
Merged via the queue into bazel-contrib:main with commit f84640b Oct 11, 2025
4 checks passed
@shayanhoshyari
Copy link

Very nice! Thank you.

@rickeylev rickeylev deleted the fix.symlink.shared.libs.directly branch October 12, 2025 00:19
@hartikainen
Copy link

Very nice indeed! I just tested this from main...hartikainen:rules_python:venv-site-packages-investigation and this work as expected. Thank you so much @rickeylev and @aignas 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Attempt to install nvidia packages with the new site-packages layout

4 participants