dag pausing should pause queued tasks as well #1065
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -570,8 +570,8 @@ def prioritize_queued(self, session, executor, dagbag): | |||
| self.logger.info("Pickling DAG {}".format(dag)) | |||
| pickle_id = dag.pickle(session).id | |||
|
|
|||
| if dag.dag_id in overloaded_dags or dag.concurrency_reached: | |||
| overloaded_dags.add(dag.dag_id) | |||
| if dag.dag_id in dag_blacklist or dag.concurrency_reached: | |||
There was a problem hiding this comment.
Don't need the first part of the if since they are already in the blacklist
There was a problem hiding this comment.
that's for the or concurrency_reached, I can break it down as an elif, though dag_blacklist is a set so it won't grow anyways
|
LGTM |
|
The previous setup would allow to queue to empty out even when the DAG was paused. No new tasks would get queued, but the ones in the queue were allowed to drain... |
mistercrunch
added a commit
that referenced
this pull request
Feb 24, 2016
dag pausing should pause queued tasks as well
mobuchowski
pushed a commit
to mobuchowski/airflow
that referenced
this pull request
Jan 4, 2022
…sources (apache#1065) * Refactor spark plan traversal to find input/output datasets from Datasources, including added BigQuery and JDBC relations Signed-off-by: Michael Collado <mike@datakin.com> * Add support for URIs in namespaces (add : and / as valid characters) Signed-off-by: Michael Collado <mike@datakin.com> * Refactored open lineage URIs to reference constant values Signed-off-by: Michael Collado <mike@datakin.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.