-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Digest does not exist when switching repos with pantsd off in one, on in the other #10719
Comments
sometime, I see this extra error info:
|
. @asherf mentioned that this occurs on a machine where some repositories are using Added to the |
I got this when running in the TC codebase (no pantsd) after just having had run in Pants's codebase (with pantsd):
|
…ng (#10789) ### Problem #10719 likely describes two different variants of "we hit the local process cache, but then failed to actually use the result because it had been garbage collected". In one of the cases it is crystal clear that the result was collected, because it is the stdout of the process that is missing. In the other case, the failure occurs while attempting to merge directories produced by process runs. ### Solution When hitting the local process cache, ensure that all of the process outputs exist (and as a sideffect, that they are downloaded locally if a remote cache is configured). Added and fixed a test for this case. An alternative implementation would have been to guarantee that a cache entry must exist only if all of the digests it requires are transitively reachable. But the local cache and the filesystem store use two different LMDB stores, which means that we cannot transactionally update them in a way that would rule out a cache entry existing even though its file content had been garbage collected... and it's not clear that merging those stores is desirable. ### Result Fixes #10719. In addition to the test, I lowered the lease time and garbage collection times and validated that the case described on #10719 is no longer reproducible.
Still happens on latest master (in the pants repo).
|
2.0.0b0 does not include #10789, meaning that the Toolchain codebase is still "wrong", even if Pants's is correct. Is this expected? Do both need to have the fix? |
this is on latest master in the pants repo. |
Yes, but the hypothesis for this issue is that it's when you have two repos, and one uses Pantsd and the other does not. My point is that you're using the right version of the pantsbuild/pants repo, but Toolchain is still "broken", so it might be expected for this to still be an issue. It will be more informative once we do the beta1 release. If things are still broken once Toolchain upgrades to beta1, then #10798 did not fix things. |
Right. The consuming ("read side") repo needs to be consuming the patch. I expect that we should re-close this unless we observe it after TC is actually using the patch. |
I'm going to re-close this until/unless we observe it in |
…ng (pantsbuild#10789) When hitting the local process cache, ensure that all of the process outputs exist (and as a sideffect, that they are downloaded locally if a remote cache is configured). Added and fixed a test for this case. An alternative implementation would have been to guarantee that a cache entry must exist only if all of the digests it requires are transitively reachable. But the local cache and the filesystem store use two different LMDB stores, which means that we cannot transactionally update them in a way that would rule out a cache entry existing even though its file content had been garbage collected... and it's not clear that merging those stores is desirable. Fixes pantsbuild#10719. In addition to the test, I lowered the lease time and garbage collection times and validated that the case described on pantsbuild#10719 is no longer reproducible.
Cherrypicking to |
…ng (cherrypick of #10789) (#10879) ### Problem #10719 likely describes two different variants of "we hit the local process cache, but then failed to actually use the result because it had been garbage collected". In one of the cases it is crystal clear that the result was collected, because it is the stdout of the process that is missing. In the other case, the failure occurs while attempting to merge directories produced by process runs. ### Solution When hitting the local process cache, ensure that all of the process outputs exist (and as a sideffect, that they are downloaded locally if a remote cache is configured). Added and fixed a test for this case. An alternative implementation would have been to guarantee that a cache entry must exist only if all of the digests it requires are transitively reachable. But the local cache and the filesystem store use two different LMDB stores, which means that we cannot transactionally update them in a way that would rule out a cache entry existing even though its file content had been garbage collected... and it's not clear that merging those stores is desirable. ### Result Fixes #10719. In addition to the test, I lowered the lease time and garbage collection times and validated that the case described on #10719 is no longer reproducible.
The text was updated successfully, but these errors were encountered: