-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: all written artifacts should be saved and garbage collected #13678
Conversation
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Hey @agilgur5 - I know you mentioned saving artifacts in parallel would be a good improvement. My thought was that with what I did it's clearly just a "fix" (which could theoretically be patched), and that saving in parallel would be more of a feature. I was also paranoid there could be some new risk of doing it in parallel, that some artifact type may not be able to handle the concurrent saves, although maybe that's not likely. If you feel this is silly, I can revise. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Signed-off-by: Julie Vogelman <julievogelman0@gmail.com>
Fixes #13583
Motivation
Two issues:
WorkflowTaskResult
, which was causing Artifact GC to try to delete them (and fail to do so)Modifications
To address # 1 above, only write to
WorkflowTaskResult
the artifacts which have been written, whether or not they're marked "Optional". This is what Artifact GC will attempt to delete.To address # 2 above, don't prematurely return if an artifact fails to be saved.
Verification
e2e test for Artifact GC now includes tests for these cases.
To test # 1 above, see artgc-optional-artifact-not-written.yaml, which includes an Optional artifact that isn't present, which we confirm does not fail ArtifactGC. We also confirm that the Workflow succeeds in this case.
To test # 2 above, see artgc-non-optional-artifact-not-written.yaml , which includes a non-Optional artifact that isn't present, plus another artifact which is present and we confirm does get saved. We also confirm that the Workflow succeeds in this case, since the Artifact that wasn't present was non-optional.