Digest does not exist when switching repos with pantsd off in one, on in the other #10719
Comments
|
sometime, I see this extra error info: |
|
. @asherf mentioned that this occurs on a machine where some repositories are using Added to the |
|
I got this when running in the TC codebase (no pantsd) after just having had run in Pants's codebase (with pantsd): |
Eric-Arellano
changed the title
stuhood
added a commit
that referenced
this issue
Sep 16, 2020
…ng (#10789) ### Problem #10719 likely describes two different variants of "we hit the local process cache, but then failed to actually use the result because it had been garbage collected". In one of the cases it is crystal clear that the result was collected, because it is the stdout of the process that is missing. In the other case, the failure occurs while attempting to merge directories produced by process runs. ### Solution When hitting the local process cache, ensure that all of the process outputs exist (and as a sideffect, that they are downloaded locally if a remote cache is configured). Added and fixed a test for this case. An alternative implementation would have been to guarantee that a cache entry must exist only if all of the digests it requires are transitively reachable. But the local cache and the filesystem store use two different LMDB stores, which means that we cannot transactionally update them in a way that would rule out a cache entry existing even though its file content had been garbage collected... and it's not clear that merging those stores is desirable. ### Result Fixes #10719. In addition to the test, I lowered the lease time and garbage collection times and validated that the case described on #10719 is no longer reproducible.
|
Still happens on latest master (in the pants repo). |
|
2.0.0b0 does not include #10789, meaning that the Toolchain codebase is still "wrong", even if Pants's is correct. Is this expected? Do both need to have the fix? |
|
this is on latest master in the pants repo. |
|
|
|
Yes, but the hypothesis for this issue is that it's when you have two repos, and one uses Pantsd and the other does not. My point is that you're using the right version of the pantsbuild/pants repo, but Toolchain is still "broken", so it might be expected for this to still be an issue. It will be more informative once we do the beta1 release. If things are still broken once Toolchain upgrades to beta1, then #10798 did not fix things. |
Right. The consuming ("read side") repo needs to be consuming the patch. I expect that we should re-close this unless we observe it after TC is actually using the patch. |
|
I'm going to re-close this until/unless we observe it in |
stuhood
added a commit
to stuhood/pants
that referenced
this issue
Sep 29, 2020
…ng (pantsbuild#10789) When hitting the local process cache, ensure that all of the process outputs exist (and as a sideffect, that they are downloaded locally if a remote cache is configured). Added and fixed a test for this case. An alternative implementation would have been to guarantee that a cache entry must exist only if all of the digests it requires are transitively reachable. But the local cache and the filesystem store use two different LMDB stores, which means that we cannot transactionally update them in a way that would rule out a cache entry existing even though its file content had been garbage collected... and it's not clear that merging those stores is desirable. Fixes pantsbuild#10719. In addition to the test, I lowered the lease time and garbage collection times and validated that the case described on pantsbuild#10719 is no longer reproducible.
|
Cherrypicking to |
stuhood
added a commit
that referenced
this issue
Sep 30, 2020
…ng (cherrypick of #10789) (#10879) ### Problem #10719 likely describes two different variants of "we hit the local process cache, but then failed to actually use the result because it had been garbage collected". In one of the cases it is crystal clear that the result was collected, because it is the stdout of the process that is missing. In the other case, the failure occurs while attempting to merge directories produced by process runs. ### Solution When hitting the local process cache, ensure that all of the process outputs exist (and as a sideffect, that they are downloaded locally if a remote cache is configured). Added and fixed a test for this case. An alternative implementation would have been to guarantee that a cache entry must exist only if all of the digests it requires are transitively reachable. But the local cache and the filesystem store use two different LMDB stores, which means that we cannot transactionally update them in a way that would rule out a cache entry existing even though its file content had been garbage collected... and it's not clear that merging those stores is desirable. ### Result Fixes #10719. In addition to the test, I lowered the lease time and garbage collection times and validated that the case described on #10719 is no longer reproducible.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The text was updated successfully, but these errors were encountered: