-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
monorepo versioning for python libraries using pants #10633
Comments
Hello @adabuleanu, great question. This is indeed possible with Pants through creating a macro: https://www.pantsbuild.org/v2.0/docs/macros. You could create a macro like this: def my_org_setup_py(**kwargs):
if "version" not in kwargs:
kwargs["version"] = "2.1.1"
return setup_py(**kwargs) Then, in your BUILD file: python_library(
...
provides=my_org_setup_py(
name=...
) Would this work? There's another option if you want to define the version in some other file than your macro definition file, such as setting up a |
I could not get pants macros to work. I have put My pants.toml
My pants-build/macros.py
and my BUILD file
Macros seem to be helpful to set up automation on versioning. But I was interested in a pants feature that accomplishes this out of the box. Basically, something like: if you change lib C, during the setup-py build it will bump the semver patch version and it will also bump the semver versions for other dependent libraries (lib B and lib C) during build. Something similar to how java artifacts are handled during publish: https://v1.pantsbuild.org/publish.html. |
Hm, I gave you a bad recommendation on using a macro like the above. Normally, macros are meant for creating target types, like Instead, This should work instead. # pants-plugins/setup_py/register.py
from pants.backend.python.python_artifact import PythonArtifact
from pants.build_graph.build_file_aliases import BuildFileAliases
def scm_setup_py(version: str = "2.1.1", **kwargs) -> PythonArtifact:
return PythonArtifact(version=version, **kwargs)
def build_file_aliases():
return BuildFileAliases(objects={"scm_setup_py": scm_setup_py}) This is inspired by pants/pants-plugins/src/python/internal_backend/utilities/register.py Lines 17 to 66 in d5801ae
which shows some more complex things you could do like setting default values. Because you can have imports, you would be able to use from pathlib import Path
from pants.base.environment import get_buildroot
def scm_setup_py(...):
version = Path(get_buildroot(), "_VERSIONS").read_text() Activate this plugin in your Then, you can use -- The above approach wouldn't fully solve what you're asking for, though. The above "object" will allow you to do things like set the If you want the more complex functionality, you'd want to set up a custom goal using the Rules API, which will allow you to do things like: a) Safely write to the build root We'd be happy to help you write this plugin. |
Thx. I got my setup running. I've hit a blocker regarding the mechanism on how pants calls the version function. This is my version implementation per python libraries: BUILD file
scm_setup_py implementation in register.py
It looks like the versioning function does not work when pants "caches" setup-py runs. See below example:
Also, is there any way of "deducting" the library path from the BUILD file (something similar to
|
Hey @adabuleanu, thanks for sharing all this code. It's helpful to see in code the logic that you're aiming for. The caching issue sounds like an instance of #10360. I'll ask another contributor if fixing #10360 would fix this issue. Otherwise, you'd want to create a plugin using the Rules API. https://www.pantsbuild.org/v2.0/docs/rules-api-concepts. Once I hear back from the contributor, I'll be happy to help you port the above logic into a plugin.
There is, with a little clunky of an API called "Context Aware Object Factory" (CAOF). You'd have something like this: from pants.base.build_environment import get_buildroot
from pants.backend.python.python_artifact import PythonArtifact
from pants.build_graph.build_file_aliases import BuildFileAliases
class ScmSetupPy:
def __init__(self, parse_context):
self._parse_context = parse_context
def __call__(self, name: str, **kwargs) -> PythonArtifact:
build_root = get_buildroot()
# This is relative to the build root, e.g. `src/python/demo`.
setup_py_rel_path = self._parse_context.rel_path
version = get_version(build_root, setup_py_rel_path)
return PythonArtifact(name=name, version=version, **kwargs)
def build_file_aliases():
return BuildFileAliases(context_aware_object_factories={"scm_setup_py": ScmSetupPy}) Note that this also uses the function |
@adabuleanu Eric is out for a couple of days, so I will pick this up. |
Thanks for the detailed information above, that made it easy to figure out why you're seeing what you're seeing. To explain the caching problem: Pants only runs your So what we need is to figure out a way to invalidate correctly here. Will think about it for a bit and reply here. |
OK, I think I have an idea here. @Eric-Arellano let's discuss Friday? |
Update: we've sketched out a plan to make it easy to use custom logic to generate a version (or any other setup() kwarg). Will post here when it's ready for use, within the next couple of days. |
We often request a target in tests for some address. This is common enough to be worth factoring up. We also port `run_setup_py_test.py` to use Pytest style, and to use this new util. In addition, we make some small improvements to `run_setup_py.py`. This is prework for #10633.
Hey @adabuleanu , @Eric-Arellano, just wondering where this is at right now? |
Closing thanks to the plugin hook at https://www.pantsbuild.org/docs/plugins-setup-py. A couple users have used this, and it seems there are far too many different versioning schemes for Pants to ship with one canonical approach. Thanks for prompting us to add this hook, Adrian! |
Hi all,
Does pants support any versioning for python libraries? From what I have read, you have to update the version number for each python library in the BUILD files, under the setup_py construct
e.g.
This because tedious and error prone when you have multiple internal python library dependencies.
e.g. : you have lib A -depends-> lib B -depends-> lib C . If you modify lib C and you want to build lib A, lib B and lib C, you have to update the version in the BUILD files for each of the 3 libraries. If not, you will overwrite existing artifacts.
Or at least this is my understanding after reading the documentation. I found something related to semantic version in the docs, but it is only available for java artifacts https://v1.pantsbuild.org/publish.html. I know versioning inside a monorepo is a hot topic when you move to a monorepo, but I was wondering how pants handles this for python libraries.
I am just starting with pants and I am currently using the latest stable version (1.30)
Sorry if I did not respect any ticket format, but I did not found any guide on this.
Thank you,
Adrian
The text was updated successfully, but these errors were encountered: