py_binary should support a PEX output
#11931
Comments
gregestren
added
team-Rules-Python
lberki
added
P4
|
Found this bazel rule repo: https://github.com/benley/bazel_rules_pex |
|
Why not solve for this in rules_python with a rule that ingests a Also, seems related to bazelbuild/rules_python#436 |
|
I learned from @groodt that #9453 indicates that this is already possible with the existing python rules, by requesting the Here's a demo: It can be run with So the remaining question for this issue is, do we actually need any other implementation like PEX? Maybe we are just missing some documentation for this output group. |
|
I feel like it being a command-line option is the wrong way to go. It should just be part of setting up py_binary. The all-or-nothing nature of this is kind of lacking. It's really why I still use .par (subpar). Also, PEX supports actually having the interpreter in the file as well, plus .so support. Its not just python only. |
|
It's included in py_binary, you can use a filegroup to select the zip output group without a command line flag. It's a bit awkward compared with java_binary's _deploy.jar implicit output, but not unusable. Seems like embedding interpreter, cross-platform, and .so are all reasons that the current zip implementation isn't sufficient, so this FR should stay open. |
|
I can clarify a few things here as well.
I wonder if we can add a macro or something for it, but yes the output filegroup is available within Bazel to add the zip to a Docker container or
Both
So, by this, are you really asking for multi-platform? Some archive format that will support all OS in a single archive with all the platform-specific interpreters and binaries? Wouldn't that get quite bloated if it had to ship 3 interpreters and the binaries for all 3 platforms? Not even sure that is possible in Bazel at the moment. How would it transition to produce the different binaries and then finally switch back to bundle them all together?
Here is the output from a zip produced by a |
|
Here is my par version: This works, other than par/py, I did not have to do anything special. Does NOT just work: I am probably doing something wrong here but people keep seeing zip is easy, just add this or do this, but it does not seem to work out of the box. |
|
Please try to execute it as follows: The functionality being leveraged is similar to: So to execute a To package your artifact for execution outside the source tree, you can build and ship the "python_zip_file" output group. |
|
Thanks @groodt, I did not realize it had to be run differently. Seems like some documentation around this feature would be good. A google search does not seem to yield anything but maybe its already out there. I will keep plugging and this and see if we can get some non-python in here working, if so, then I would be less likely to want pex. PAR does not support anything but python currently which has worked for us, so far. |
I actually got the interpreter in by defining my own python toolchain as type This not only made sure that these files made it in as runfiles, but also the Allows one to go into the past, based on what you put :) The remaining issue are the .so libraries the Python interpreter itself would depend on - it still tries to hit |
|
OT: Maybe as a bit of side-feedback, this (as in zip file via output group and the entrypoint generation) should be rolled into rules_py rather than rely on Bazel itself? In the absence of missing documentation, people are likely to search source files, and I can see that happening for rules, but usually not for Bazel source itself. Also makes it easier to iterate on. |
|
If you use the Sadly py_library and PyInfo are stuck in the Bazel sources, so users have to put the You can then write a I think we can post an example under rules_python soon, I have a client making this work. |
|
Happy to offer any help with PEX rule if people are interested. I already have a minimal functioning example - https://github.com/borancar/mishmash/tree/928cd7bb75c93472c3931f3f8f56941786ec6116, and here's the changes needed to make rules pex work for Bazel 4 borancar/mishmash@928cd7b#diff-6b9d2d187ff982cd6100e3a5e0813e450351d5164262d0c1d9acc43522d994db |
|
At twitter we have a rule that does exactly this, though it takes a py_library and outputs a pex. I can speak with the team about the potential to open source it. |
this is very useful, is there any progress now ? |
|
Hey @libratiger I haven't got around to bringing this up sorry. Its still a possibility. A couple things need to happen. Permission from close source repo and agreement on direction with the rest of rules_python maintainers. |
|
@hrfuller is there any progress on open-sourcing rule that make pex? |
|
No longer at Twitter. However, I'm open to working on a clean slate design based on use of the PEX CLI, which probably would a better fit for the OSS world. |
Ftr this is basically how https://github.com/arrdem/rules_zapp functions |
|
https://github.com/aspect-build/rules_py/blob/main/docs/pex.md solves this, if you're willing to adopt a ruleset on top of rules-python. I think this issue should be closed, or at least transferred to rules_python. |
Description of the problem / feature request:
A
java_binaryhas an implicit*_deploy.jarthat creates a single file that can be executed. There is no equivalent implicit output for apy_binary. Any such equivalent should support packaging native libraries (for multiple OSs) for obvious reasons.PEX provides a mechanism to do this.
Feature requests: what underlying problem are you trying to solve with this feature?
Create a single deployable artifact from a
py_binarytarget without specifying additional command line options.What's the output of
bazel info release?3.4.1
Have you found anything relevant by searching the web?
There's a
par_binaryrule that already exists, but it handles neither native files, nor runfiles. In addition, it's a different rule thanpy_binaryrather than implicit output.FB have also released XAR, but this relies on squashfs and so is pretty hostile to Windows users.
The text was updated successfully, but these errors were encountered: