-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add the target fingerprint to the version of each local dist so that we don't use the first cached one #6022
add the target fingerprint to the version of each local dist so that we don't use the first cached one #6022
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a few comments.
This is a good starting point but this problem should really be solved at the pex resolver cache level by something along the lines of my PR at pex-tool/pex#453.
I’d be interested in @kwlzn’s take on the refactoring of the python_dist target to inherit from PythonTarget. There was a reason we avoided this but I cannot recall at the moment.
Overall, looks good, just wondering whether this is the right thing to be doing/the right UX change we want to introduce.
from setuptools import setup, find_packages | ||
from distutils.core import Extension | ||
|
||
|
||
c_module = Extension(str('c_greet'), sources=[str('c_greet.c')]) | ||
cpp_module = Extension(str('cpp_greet'), sources=[str('cpp_greet.cpp')]) | ||
|
||
public_version = '1.0.0' | ||
local_version = os.getenv('_SETUP_PY_LOCAL_VERSION') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This places responsibility of this existence on the author, so we would need to document this or throw a warning if it is not present.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would probably be super easy to just check the version when we make the local wheels product and make sure it has a PEP 440 "local" identifier at the end -- then, we allow the possibility of using setup.py projects generated in a different way if we ever need to, but still avoid the name clashing issue!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, as was very effectively and thoroughly discussed in #5449, and as I have thought about before, there are many more ways to potentially make this part of the setup.py more ergonomic/natural (including making temporary files that the setup.py could import, for example?). I think the env var is a simple and effective answer right now but I'm 100% ready to go deeper into setuptools as need be.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TIL the egg_info setup command. Maybe this helps?
$ ./pants setup-py --run='egg_info -b+bob sdist bdist bdist_wheel' src/python/pants:pants-packaged
...
$ ls -1 dist/pantsbuild.pants-1.9.0.dev0/dist/
pantsbuild.pants-1.9.0.dev0+bob.linux-x86_64.tar.gz
pantsbuild.pants-1.9.0.dev0+bob-py2-none-any.whl
pantsbuild.pants-1.9.0.dev0+bob.tar.gz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jsirois I didn't see that comment until now -- thanks! It would be utterly fantastic to be able to avoid making any changes to the setup.py to support this. Will look into using that now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Of course it works -- added in 1fd8db1 so we can do this without any changes to setup.py at all! I should have probably expected this to exist and looked beforehand, so thanks again for mentioning it!
""" | ||
if not 'setup.py' in sources: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why strip these out? Are you planning on adding setup.py checking back or is it happening elsewhere?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be the exact same as the previous target, just without unnecessarily repeated code, which I think should be a goal. It doesn't seem like too gross of a refactoring because it's limited to a single file. We call the superclass constructor (PythonTarget
), which already does all of the checking that was removed from this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I remember vaguely that we don't subclass PythonTarget
due to some method in pex_build_util.py
using PythonTarget
to mean something that we don't want to include python_dist()
in. I don't know if that is still the case with the changes to that file. Will look into and leave a comment if this is ok or revert the change / do something else if not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed on the redundancy, but I think that PythonTarget will not check for setup.py presence as of right now, so it would be good to add that back in to this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand -- we are still checking if setup.py
is in sources, just doing it before we initialize anything else. I can move it back to after the super constructor call -- it's not clear which ordering would be desired here, I just usually try to do argument validation first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, weird, I must have read this wrong. This looks good to me regarding setup.py checking.
That's what everyone said on #5479, which is why I closed it, but I don't think it's correct, and I don't think this particular issue is the same as the one resolved in pex-tool/pex#453. I actually wrote another pex PR based on pex-tool/pex#453 first, but after spending several hours inserting debug statements into various places, realized that does not solve the issue. EDIT: to clarify, using pex-tool/pex#453 will end up ensuring the python requirement is updated after a If you build the |
pex-tool/pex#453 should fix this issue though because of what it is doing regarding the cache key strategy. Because that PR gets the size of the wheel and incorporates it into a cache key, when Pants uses the the pex resolver it will not find it in the cache (different code, same version), crawl the repo attached to the PythonRequirement object (the .pants.d location for build-local-dists), and use that one. The cache could also be made more robust by using actual hashing of the wheel contents. Afaict, invalidation is taken care of by Pants detecting changed sources, so wouldn’t pex/#453 solve the problem? |
I have a PR that I can put up that hashes the wheel contents (a two line change from pex-tool/pex#453), which I haven't yet because I wanted to get this in first. If you attempt to use pants with pex-tool/pex#453, you will find that it does not solve the problem. This is why I decided to make this PR in the first place, because I branched off of pex-tool/pex#453 and found that it did not solve the problem. If you use pex-tool/pex#453, it will stop using the first built version of the You can test a development version of pex with pants by deleting the pex line from |
I'm going to implement @jsirois's suggestion to use the |
1fd8db1
to
298ea5e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great, thanks.
I suspect that Chris's comments on the python_dist
arguments were a misunderstanding, so you're good to land this.
interpreter compatibility for this target, using the Requirement-style | ||
format, e.g. ``'CPython>=3', or just ['>=2.7','<3']`` for requirements | ||
agnostic to interpreter class. | ||
:type sources: :class:`twitter.common.dirutil.Fileset` or list of strings. Must include |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that this changes the behaviour of sources, you might want to merge from master and re-test post #5908
298ea5e
to
aaa9cbf
Compare
aaa9cbf
to
14e2f0f
Compare
So the OSX shard continues to fail in CI, although I've cleared caches twice. I can't seem to repro this locally and suspect that (as was mentioned earlier) it may require pex-tool/pex#453 (or a modified version of it). That should be done anyway, so I will put up a new pex PR to take that over from @CMLivingston (we discussed this previously) and come back to this one. |
486d3ee
to
d486af4
Compare
dca9887
to
af00a12
Compare
This was rebased against master but otherwise unchanged from the last time -- #6104 fixed the issue I was banging my head against the wall about. There was a lint error which I just fixed, which was because I removed the use of |
setup( | ||
name='ctypes_test', | ||
version='0.0.1', | ||
version='1.0.0', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1.0.0 typically means stable, so this may not be yet? (your call)
A good reference would be https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/project_info/tasks/export.py#L51-L63
so maybe 0.1.0
if there is backward incompatibility?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, I hadn't thought about the significance of this at all, mostly changed it to mimic the other python_dist()
targets. I don't think it's stable yet, I was planning to switch the target names from ctypes_compatible_cpp_library()
to just cpp_library()
at that time as well. I'll revert this change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for the reference!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is backwards incompatibility, but I don't think anyone would be relying on the broken behavior this PR fixes, except accidentally, so I think 0.0.1
is probably the right move here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reverted the version change in f88e427!
@@ -53,9 +56,25 @@ def test_pants_run(self): | |||
# Check that text was properly printed to stdout. | |||
self._assert_native_greeting(pants_run.stdout_data) | |||
|
|||
@staticmethod | |||
def rewrite_hello_c_source(orig_content): | |||
return re.sub('"Hello from C!"', '"Hello from C?"', orig_content) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the usage of this function is fairly contained in the particular test below, I'd recommend moving it into the test itself. @staticmethod
or not isn't important.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, see the new version of the test.
self.assertIn('Hello from C!\n', unmodified_pants_run.stdout_data) | ||
|
||
c_source_file = '{}/c_greet.c'.format(self.fasthello_install_requires) | ||
with self.rewrite_file_content(c_source_file, self.rewrite_hello_c_source): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
non-blocking: since you are modifying the content of the repo, it might create surprises if later tests happen to run against the touched content. It's best to create a target on the test run, but if it's hard to do, it would be good to restore it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, and I've had the file stuck in the intermediate "rewritten" state at least once while testing -- I'm not sure what caused that since it is wrapped in a try
but I'll see if I can change the implementation to make a target for the test (and make sure that that implementation fails on current master so it's still a valid test). I have a hunch this won't be that big of a change, and for the reasons you mentioned I would prefer not to leave the rewrite_file_content
method in the integration test base class.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ended up using mock_buildroot()
from PantsRunIntegrationTest
, which is much more clear anyway -- see the new test here. Thanks for pointing this out!
@@ -8,6 +8,7 @@ | |||
import c_greet | |||
import cpp_greet | |||
|
|||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can leave this file untouched.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in a7d2f23!
@@ -23,6 +23,7 @@ def get_generated_shared_lib(lib_name): | |||
asdf_c_lib = ctypes.CDLL(asdf_c_lib_path) | |||
asdf_cpp_lib = ctypes.CDLL(asdf_cpp_lib_path) | |||
|
|||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can leave this file untouched.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in a7d2f23!
@@ -4,5 +4,6 @@ | |||
|
|||
from __future__ import absolute_import, division, print_function, unicode_literals | |||
|
|||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can leave this file untouched.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in a7d2f23!
@@ -7,6 +7,7 @@ | |||
import c_greet | |||
import cpp_greet | |||
|
|||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can leave this file untouched.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in a7d2f23!
4f8e9dc
to
a7d2f23
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR description could use a major rewrite at this point.
|
||
setup_py_command = ( | ||
egg_info_snapshot_tag_args + bdist_whl_args + platform_args + dist_dir_args) | ||
return cls(source_dir, setup_py_command, **kw) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all seems very specific to your use case. Any reason not to just move all this logic there and call the constructor directly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're totally right. Will do that now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved to _generate_snapshot_bdist_wheel_argv()
of BuildLocalPythonDistributions
in 2aa1a5f!
@@ -115,6 +117,26 @@ def _retrieve_single_product_at_target_base(self, product_mapping, target): | |||
single_product = all_products[0] | |||
return single_product | |||
|
|||
# TODO: This snapshot version tagging is only performed for python_dist() targets, but the | |||
# setup.py invocation command line is generated in SetupPyRunner in setup_py.py. Could this |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And I think we agree on the problem I mentioned above!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed! Made the change in 2aa1a5f -- also moved some comments here into docstrings.
Rewrote the OP to reflect what's currently happening in this PR, and also to be nicer (I think I had a wild headache, still not an excuse -- I'm sorry about that and will make an effort to recognize it in the future). I may have made a mistake when moving setup.py argv generation into I'm VERY pumped that this is finally close to being mergeable. |
The osx shard is failing again, but with a different error:
I can investigate this -- the error message is more revealing this time. There is another failure on Linux that I can repro locally (edit: fixed in 09442b8). The other failures appear to be spurious timeouts ( |
@cosmicexplorer the most useful debug is the edit |
Will do! |
And ... likely #6141 fixes although I cannot yet explain all the interactions. @cosmicexplorer I'd be obliged if you try out the patch; although I'll land the PR regardless since it eliminates cruft. |
f5f5b9d
to
5f108db
Compare
Rebased after #6141 landed and running through CI again. I had forgotten the point about how to identify the use of the apple-provided python interpreter and why that was a concern, so thank you for elaborating on that. We shall be able to see hopefully that #6141 makes things work as suspected -- will dig in if not. |
…rmalized snapshot version
TRAVIS IS GREEN (without me having to do anything at all)!!! @CMLivingston @kwlzn if you could take another look at this that'd be neat -- I don't think there is anything of concern I missed but since this went through several iterations since last time (in response to changing conditions, namely the resolution errors getting fixed elsewhere in pants) I'd feel more comfortable if you could take a quick look again. Thanks. I also want to specifically shout out @jsirois and @kwlzn for responding to a lot of questions I had on this PR in this thread and also on slack earlier when I didn't really know what was happening at all and was mega frustrated -- I really appreciated that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Problem
I made #5479 because the dist resolved for a
python_dist()
would not change after changing any of the source files, and this persisted even after aclean-all
. This issue is more thoroughly explained in #5449 (which I literally didn't see before making this PR, but describes almost exactly what this PR does, it seems -- great minds think alike?). We were building thepython_dist()
targets each time they were invalidated, but caching the package locally in~/.cache/pants/
after we built it the first time and using that instead, because the version string was the same for both the first and subsequent builds of thepython_dist()
target.Solution
BuildLocalPythonDistributions
.python_dist()
target's fingerprint to the version string using--tag-build
. This conforms to the "local" version specifier format as per PEP 440. This tagged version is then used in the syntheticPythonRequirement
stitched into the build graph for the local dist.test_invalidation()
usingmock_buildroot()
(which fails on master, but not in this PR) where we run a binary depending on apython_dist()
, rewrite the contents of a source file to change the output, then run pants again to verify that the new dist is resolved.Separately:
PythonDistribution
subclassPythonTarget
to avoid duplicated logic. It's not clear what the original reason for not subclassing it was at first, but it seems to be not be in effect anymore.dir
field name for theManager
namedtuple inmock_buildroot()
to benew_buildroot
so it doesn't get highlighted as a builtin in certain text editors.Result
When we resolve local dists in any python requirements task, we get the dist corresponding to the target we just built. Resolves #5449.