Exponential Backoff Not Functioning in BaseSensorOperator Reschedule Mode

[sanket2000]

Fix: Increment try_number/poke_count in BaseSensorOperator for correct exponential backoff in reschedule mode.

What happened?

• The BaseSensorOperator does not correctly implement exponential backoff when in reschedule mode.
• The sensor reschedules at a constant interval instead of increasing the interval exponentially.

What you think should happen instead?

• The try_number (or poke_count) should increment on each reschedule too, ensuring the backoff interval increases exponentially.

How to reproduce?

• Set up a BaseSensorOperator with mode="reschedule" and exponential_backoff=True.
• Observe the interval between reschedules; it remains constant instead of increasing exponentially.

#6654

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[sanket2000]

To reproduce:

from airflow.models.dag import DAG
from airflow.sensors.base import BaseSensorOperator


class DummySensor(BaseSensorOperator):
    def __init__(self, return_value=False, **kwargs):
        super().__init__(**kwargs)
        self.return_value = return_value

    def poke(self, context):
        return self.return_value


with DAG(dag_id="base_sensor_test") as d:

    s = DummySensor(
        task_id="sensor",
        mode="reschedule",
        exponential_backoff=True,
    )

Task Log: attempt=1.log

[sanket2000]

How can I request Reviewers?

[dirrao]

Can you add the test case for the bug?

[sanket2000]

Thank you for the review, this is my first PR, I wanted some guidance on the testing.
This should be the file I need to add my test tests/sensors/test_base.py right?

[sanket2000]

I looked into tests/sensors/test_base.py and couldn't find any reference tests that specifically test the execute function of the sensor. To test the increment of poke_count (or try_number for older versions), such a test might be necessary. Any guidance or assistance in implementing this test would be greatly appreciated.

[sanket2000]

I have added a change, if it looks good poke_count can potentially be remove altogether. We can estimate the pokes based on start time and run duration.

[sanket2000]

I would also encourage authors and participants of PR #30669 to take a look as this approach doesn't involve DB column changes proposed in the above mentioned PR

[eladkal]

I would also encourage authors and participants of PR #30669 to take a look as this approach doesn't involve DB column changes proposed in the above mentioned PR

cc @hussein-awala

[RNHTTR]

Thank you for the review, this is my first PR, I wanted some guidance on the testing. This should be the file I need to add my test tests/sensors/test_base.py right?

yes

[sanket2000]

I feel this PR is ready for review along with tests.
It really is a roller-coaster to work on the test files :)

[potiuk]

This one looks good. One thing to add is that you should add newsfragment describing the change (see newsfragment directory). This is a change in behaviour (a good one - fixing unintended behaviour) - but it also should be noted in release notes when this one gets merged.

[sanket2000]

This one looks good. One thing to add is that you should add newsfragment describing the change (see newsfragment directory). This is a change in behaviour (a good one - fixing unintended behaviour) - but it also should be noted in release notes when this one gets merged.

Thanks for the review, I have added the necessary file.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.