Skip to content

Commit

Permalink
Fix dag file processor hertbeat to run only if necessary (#30899)
Browse files Browse the repository at this point in the history
The standalone file processor as of #30278 introduced accidentally
an artifficial delay between dag processing by adding heartbeat
but missing to set "only_if_necessary" flag to True.

If your dag file processing has been fast (faster than the
scheduler job_heartbeat_sec) this introduced unnecessary pause
between the next dag file processor loop (up until the time
passed), it also introduced inflation of the
dag_processing_last_duration metrics (it would always show minimum
job_heartbeat_sec)

Adding "only_if_necessary" flag fixes the problem.

Fixes: #30593
Fixes: #30884
(cherry picked from commit 00ab45f)
  • Loading branch information
potiuk authored and ephraimbuddy committed Apr 27, 2023
1 parent 3702656 commit 568ca2b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion airflow/jobs/dag_processor_job_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ def __init__(
self.processor.heartbeat = lambda: perform_heartbeat(
job=self.job,
heartbeat_callback=empty_callback,
only_if_necessary=False,
only_if_necessary=True,
)

def _execute(self) -> int | None:
Expand Down

0 comments on commit 568ca2b

Please sign in to comment.