-
Notifications
You must be signed in to change notification settings - Fork 361
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WX-1595 GCP Batch backend refactor to include the PAPI request manager #7412
Commits on Apr 18, 2024
-
Refactor GCP Batch request manager
BatchApiRequestWorkerSpec works! The goal is to port the papiv2 request manager behavior into GCP batch. We just need to rename the methods to remove the PAPI names.
Configuration menu - View commit details
-
Copy full SHA for ff8543c - Browse repository at this point
Copy the full SHA ff8543cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5cca13a - Browse repository at this point
Copy the full SHA 5cca13aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0823351 - Browse repository at this point
Copy the full SHA 0823351View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f5ab24 - Browse repository at this point
Copy the full SHA 0f5ab24View commit details -
Configuration menu - View commit details
-
Copy full SHA for 94d1aaf - Browse repository at this point
Copy the full SHA 94d1aafView commit details -
Configuration menu - View commit details
-
Copy full SHA for af6b42d - Browse repository at this point
Copy the full SHA af6b42dView commit details -
Configuration menu - View commit details
-
Copy full SHA for fdb5c24 - Browse repository at this point
Copy the full SHA fdb5c24View commit details -
Configuration menu - View commit details
-
Copy full SHA for ff23056 - Browse repository at this point
Copy the full SHA ff23056View commit details -
Configuration menu - View commit details
-
Copy full SHA for ecc28d1 - Browse repository at this point
Copy the full SHA ecc28d1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2afef28 - Browse repository at this point
Copy the full SHA 2afef28View commit details -
Configuration menu - View commit details
-
Copy full SHA for a249835 - Browse repository at this point
Copy the full SHA a249835View commit details -
Now, we just need to fix the runtime errors, wiring the correct messages, etc.
Configuration menu - View commit details
-
Copy full SHA for 7ef43da - Browse repository at this point
Copy the full SHA 7ef43daView commit details -
Configuration menu - View commit details
-
Copy full SHA for 71daebe - Browse repository at this point
Copy the full SHA 71daebeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1e8bcef - Browse repository at this point
Copy the full SHA 1e8bcefView commit details -
The queue must be cleared after executing the requests.
Configuration menu - View commit details
-
Copy full SHA for ad51153 - Browse repository at this point
Copy the full SHA ad51153View commit details -
Configuration menu - View commit details
-
Copy full SHA for 476a923 - Browse repository at this point
Copy the full SHA 476a923View commit details -
Clean up unnecessary code + batch abort request bugfix
Abort request handler was returning the wrong message, also, keep the old behavior where abort request is handled once only.
Configuration menu - View commit details
-
Copy full SHA for f92974b - Browse repository at this point
Copy the full SHA f92974bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0bf7a90 - Browse repository at this point
Copy the full SHA 0bf7a90View commit details -
Configuration menu - View commit details
-
Copy full SHA for 94ce5cd - Browse repository at this point
Copy the full SHA 94ce5cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 76bb726 - Browse repository at this point
Copy the full SHA 76bb726View commit details -
Configuration menu - View commit details
-
Copy full SHA for dbd6d24 - Browse repository at this point
Copy the full SHA dbd6d24View commit details -
Configuration menu - View commit details
-
Copy full SHA for e619ef0 - Browse repository at this point
Copy the full SHA e619ef0View commit details -
Huge refactor on batch RunStatus
This grabs many details from PAPIv2, work still pending on the tests. NOTE: This could break the current integration.
Configuration menu - View commit details
-
Copy full SHA for 057ee14 - Browse repository at this point
Copy the full SHA 057ee14View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6fe1518 - Browse repository at this point
Copy the full SHA 6fe1518View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0412a9d - Browse repository at this point
Copy the full SHA 0412a9dView commit details -
Configuration menu - View commit details
-
Copy full SHA for d71751d - Browse repository at this point
Copy the full SHA d71751dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3ccc34d - Browse repository at this point
Copy the full SHA 3ccc34dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 810a061 - Browse repository at this point
Copy the full SHA 810a061View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9acc240 - Browse repository at this point
Copy the full SHA 9acc240View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7a4db8c - Browse repository at this point
Copy the full SHA 7a4db8cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2db7ce0 - Browse repository at this point
Copy the full SHA 2db7ce0View commit details -
Query the job status to not delete a terminal job.
Configuration menu - View commit details
-
Copy full SHA for 62129c9 - Browse repository at this point
Copy the full SHA 62129c9View commit details -
Refactor GcpBatchGroupedRequests to be a data holder only
The execution is now handled by BatchRequestExecutor.
Configuration menu - View commit details
-
Copy full SHA for 26603c8 - Browse repository at this point
Copy the full SHA 26603c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 29f3c4f - Browse repository at this point
Copy the full SHA 29f3c4fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 778c83a - Browse repository at this point
Copy the full SHA 778c83aView commit details -
Clean up RunStatus from unnecessary data
This removes the custom error mappers that detect preemptible errors because these are supposed to be handled natively by GCP.
Configuration menu - View commit details
-
Copy full SHA for 7becc27 - Browse repository at this point
Copy the full SHA 7becc27View commit details -
Configuration menu - View commit details
-
Copy full SHA for 38eb051 - Browse repository at this point
Copy the full SHA 38eb051View commit details -
Fix minor bug in BatchRequestExecutor
The requests were being reversed unnecessarily.
Configuration menu - View commit details
-
Copy full SHA for 64510cf - Browse repository at this point
Copy the full SHA 64510cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for be118f5 - Browse repository at this point
Copy the full SHA be118f5View commit details -
Configuration menu - View commit details
-
Copy full SHA for b4a1b30 - Browse repository at this point
Copy the full SHA b4a1b30View commit details -
Configuration menu - View commit details
-
Copy full SHA for e8a61f1 - Browse repository at this point
Copy the full SHA e8a61f1View commit details -
Run centaurGcpBatch tests only
There is a failed test which we need to fix, there is no need to run the whole suite for now.
Configuration menu - View commit details
-
Copy full SHA for d9c362a - Browse repository at this point
Copy the full SHA d9c362aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 11a02bb - Browse repository at this point
Copy the full SHA 11a02bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1e2661b - Browse repository at this point
Copy the full SHA 1e2661bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 290f065 - Browse repository at this point
Copy the full SHA 290f065View commit details -
Configuration menu - View commit details
-
Copy full SHA for 50517e4 - Browse repository at this point
Copy the full SHA 50517e4View commit details
Commits on Apr 23, 2024
-
Remove unnecessary comments from GcpBatchAsyncBackendJobExecutionActo…
…rSpec There are a few tests that were marked with TODO notes, still, we found out that the behavior from batch is the same than the behavior from papiv2, both of these backends behave differently than papi-common.
Configuration menu - View commit details
-
Copy full SHA for a333f65 - Browse repository at this point
Copy the full SHA a333f65View commit details
Commits on Apr 26, 2024
-
Configuration menu - View commit details
-
Copy full SHA for bfd9dd1 - Browse repository at this point
Copy the full SHA bfd9dd1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 54c152d - Browse repository at this point
Copy the full SHA 54c152dView commit details
Commits on May 2, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 59c9675 - Browse repository at this point
Copy the full SHA 59c9675View commit details
Commits on May 3, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 83891cf - Browse repository at this point
Copy the full SHA 83891cfView commit details
Commits on May 7, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0cf2ba0 - Browse repository at this point
Copy the full SHA 0cf2ba0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 526447d - Browse repository at this point
Copy the full SHA 526447dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8d7c618 - Browse repository at this point
Copy the full SHA 8d7c618View commit details -
Configuration menu - View commit details
-
Copy full SHA for fa18a3c - Browse repository at this point
Copy the full SHA fa18a3cView commit details -
Configuration menu - View commit details
-
Copy full SHA for f872fee - Browse repository at this point
Copy the full SHA f872feeView commit details
Commits on May 9, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0e30684 - Browse repository at this point
Copy the full SHA 0e30684View commit details
Commits on May 10, 2024
-
Configuration menu - View commit details
-
Copy full SHA for c08b09b - Browse repository at this point
Copy the full SHA c08b09bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 23650c0 - Browse repository at this point
Copy the full SHA 23650c0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 35dce53 - Browse repository at this point
Copy the full SHA 35dce53View commit details -
Configuration menu - View commit details
-
Copy full SHA for cee36d9 - Browse repository at this point
Copy the full SHA cee36d9View commit details
Commits on May 14, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 6119047 - Browse repository at this point
Copy the full SHA 6119047View commit details -
Configuration menu - View commit details
-
Copy full SHA for f4511b9 - Browse repository at this point
Copy the full SHA f4511b9View commit details -
Configuration menu - View commit details
-
Copy full SHA for b30be69 - Browse repository at this point
Copy the full SHA b30be69View commit details
Commits on May 15, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8f16dbc - Browse repository at this point
Copy the full SHA 8f16dbcView commit details
Commits on May 16, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 28a62ef - Browse repository at this point
Copy the full SHA 28a62efView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0e4ddcb - Browse repository at this point
Copy the full SHA 0e4ddcbView commit details
Commits on May 21, 2024
-
Configuration menu - View commit details
-
Copy full SHA for f1a83ff - Browse repository at this point
Copy the full SHA f1a83ffView commit details -
Set requestsAbortAndDiesImmediately=false
This allow us handling the abort result instead of blindly marking the job as aborted.
Configuration menu - View commit details
-
Copy full SHA for dca1420 - Browse repository at this point
Copy the full SHA dca1420View commit details -
Now, we look for the submit requests before sending an abort request, canceling jobs that were not submitted to GCP. Turns out that this is now unnecessary because we are already mapping query errors to a RunStatus.
Configuration menu - View commit details
-
Copy full SHA for eca754f - Browse repository at this point
Copy the full SHA eca754fView commit details -
Remove customPollStatusFailure
Turns out that this is now unnecessary because we are already mapping query errors to a RunStatus.
Configuration menu - View commit details
-
Copy full SHA for 0095a87 - Browse repository at this point
Copy the full SHA 0095a87View commit details -
Configuration menu - View commit details
-
Copy full SHA for 837f86e - Browse repository at this point
Copy the full SHA 837f86eView commit details -
Fix abort from BatchApiRequestManager
When aborting an individual job, only that job can be aborted instead of all the jobs from that workflow.
Configuration menu - View commit details
-
Copy full SHA for dcd0efe - Browse repository at this point
Copy the full SHA dcd0efeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3c9d020 - Browse repository at this point
Copy the full SHA 3c9d020View commit details
Commits on May 24, 2024
-
Turns out that this is not the correct fix.
Configuration menu - View commit details
-
Copy full SHA for e261cb9 - Browse repository at this point
Copy the full SHA e261cb9View commit details
Commits on May 25, 2024
-
- Switch the noisy logs to debug level. - Remove status codes ported from PAPI because they do not have any usefulness in batch. - Remove all the test cases involved the PAPI codes. - Clean RunStatus from the unused args. - Rename JES occurrences to Batch.
Configuration menu - View commit details
-
Copy full SHA for 1050850 - Browse repository at this point
Copy the full SHA 1050850View commit details -
Configuration menu - View commit details
-
Copy full SHA for 75acee0 - Browse repository at this point
Copy the full SHA 75acee0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 387e399 - Browse repository at this point
Copy the full SHA 387e399View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5f01bc9 - Browse repository at this point
Copy the full SHA 5f01bc9View commit details
Commits on May 29, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 37e5fde - Browse repository at this point
Copy the full SHA 37e5fdeView commit details