Skip to content

Conversation

@steveahnahn
Copy link
Contributor

Problem

ti.get_first_reschedule_date() returns a string instead of a datetime object when running due to TaskRescheduleStartDate.model_construct() bypasses Pydantic validation. The corresponding unit test also tested against a string.

Solution

Use the standard Pydantic constructor instead of model_construct() to ensure it parses the ISO string into a timezone-aware datetime object and fixes the associated test.

Related Issue

Closes #58777

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't correct. I ran your test dag with airflow 3.1.3 and added few logs:

Index: task-sdk/src/airflow/sdk/bases/sensor.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/task-sdk/src/airflow/sdk/bases/sensor.py b/task-sdk/src/airflow/sdk/bases/sensor.py
--- a/task-sdk/src/airflow/sdk/bases/sensor.py	(revision 08e871458153cbac24a9ef3dba31286934f603e1)
+++ b/task-sdk/src/airflow/sdk/bases/sensor.py	(date 1764310309945)
@@ -184,7 +184,9 @@
         if self.reschedule:
             ti = context["ti"]
             first_reschedule_date = ti.get_first_reschedule_date(context)
+            print("first reschedule date:", first_reschedule_date, type(first_reschedule_date))
             started_at = start_date = first_reschedule_date or timezone.utcnow()
+            print("start date:", start_date, type(start_date))
 
             def run_duration() -> float:
                 # If we are in reschedule mode, then we have to compute diff

And it doesn't return string anywhere. It's a datetime object:

[2025-11-28 11:43:12] INFO - __file__='/files/plugins/triggera.py' loaded source=task.stdout
[2025-11-28 11:43:12] INFO - __file__='/files/plugins/triggera_comprehensive.py' loaded source=task.stdout
[2025-11-28 11:43:12] INFO - DAG bundles loaded: dags-folder source=airflow.dag_processing.bundles.manager.DagBundlesManager loc=manager.py:209
[2025-11-28 11:43:12] INFO - Filling up the DagBag from /files/test_dags/poke-bug.py source=airflow.dag_processing.dagbag.DagBag loc=dagbag.py:627
[2025-11-28 11:43:12] INFO - first reschedule date: None <class 'NoneType'> source=task.stdout
[2025-11-28 11:43:12] INFO - start date: 2025-11-28 06:13:12.540845+00:00 <class 'datetime.datetime'> source=task.stdout
[2025-11-28 11:43:12] INFO - Poking callable: <function poke_dag.<locals>.sensor_task at 0xffff84709750> source=airflow.task.operators.airflow.providers.standard.decorators.sensor.DecoratedSensorOperator loc=python.py:75
[2025-11-28 11:43:12] INFO - Rescheduling task, marking task as UP_FOR_RESCHEDULE source=task loc=task_runner.py:1011
[2025-11-28 11:43:14] INFO - __file__='/files/plugins/triggera.py' loaded source=task.stdout
[2025-11-28 11:43:14] INFO - __file__='/files/plugins/triggera_comprehensive.py' loaded source=task.stdout
[2025-11-28 11:43:14] INFO - DAG bundles loaded: dags-folder source=airflow.dag_processing.bundles.manager.DagBundlesManager loc=manager.py:209
[2025-11-28 11:43:14] INFO - Filling up the DagBag from /files/test_dags/poke-bug.py source=airflow.dag_processing.dagbag.DagBag loc=dagbag.py:627
[2025-11-28 11:43:14] INFO - Poking callable: <function poke_dag.<locals>.sensor_task at 0xffff8470d990> source=airflow.task.operators.airflow.providers.standard.decorators.sensor.DecoratedSensorOperator loc=python.py:75
[2025-11-28 11:43:14] INFO - Rescheduling task, marking task as UP_FOR_RESCHEDULE source=task loc=task_runner.py:1011
[2025-11-28 11:43:14] INFO - first reschedule date: 2025-11-28 06:13:12.546843+00:00 <class 'datetime.datetime'> source=task.stdout
[2025-11-28 11:43:14] INFO - start date: 2025-11-28 06:13:12.546843+00:00 <class 'datetime.datetime'> source=task.stdout
[2025-11-28 11:43:16] INFO - __file__='/files/plugins/triggera.py' loaded source=task.stdout
[2025-11-28 11:43:16] INFO - __file__='/files/plugins/triggera_comprehensive.py' loaded source=task.stdout
[2025-11-28 11:43:16] INFO - DAG bundles loaded: dags-folder source=airflow.dag_processing.bundles.manager.DagBundlesManager loc=manager.py:209
[2025-11-28 11:43:16] INFO - Filling up the DagBag from /files/test_dags/poke-bug.py source=airflow.dag_processing.dagbag.DagBag loc=dagbag.py:627
[2025-11-28 11:43:16] INFO - Poking callable: <function poke_dag.<locals>.sensor_task at 0xffff843feef0> source=airflow.task.operators.airflow.providers.standard.decorators.sensor.DecoratedSensorOperator loc=python.py:75
[2025-11-28 11:43:16] INFO - Skipping task. reason=Sensor has timed out; run duration of 4.354058 seconds exceeds the specified timeout of 3.0. source=task loc=task_runner.py:1003
[2025-11-28 11:43:16] INFO - first reschedule date: 2025-11-28 06:13:12.546843+00:00 <class 'datetime.datetime'> source=task.stdout
[2025-11-28 11:43:16] INFO - start date: 2025-11-28 06:13:12.546843+00:00 <class 'datetime.datetime'> source=task.stdout

@steveahnahn
Copy link
Contributor Author

@amoghrajesh You're right that at runtime it returns datetime. This is because when the response travels from supervisor to task runner, it gets deserialized where Pydantic validation converts the string to datetime. As for client.py, it was returning a string with model_construct() which bypasses validation, and the test itself was testing against a string and passing. This fix ensures the client returns the correct type rather than relying on downstream

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for clarifying.

@amoghrajesh
Copy link
Contributor

@prdai could you change the title of the PR to reflect the changes better?

@prdai
Copy link
Contributor

prdai commented Dec 1, 2025

@amoghrajesh i didn't create this pr, so i can't really change the title...

@amoghrajesh
Copy link
Contributor

My bad. Wrong tag

@steveahnahn steveahnahn changed the title fix string to datetime pydantic and test Fix client bypassed pydantic validation to enforce datetime & fix corresponding unittest Dec 1, 2025
@steveahnahn steveahnahn requested a review from prdai December 1, 2025 20:13
@steveahnahn
Copy link
Contributor Author

Title updated

@potiuk potiuk added this to the Airflow 3.1.4 milestone Dec 2, 2025
@potiuk potiuk added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Dec 2, 2025
@potiuk potiuk merged commit 4360616 into apache:main Dec 2, 2025
91 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 2, 2025
(cherry picked from commit 4360616)

Co-authored-by: Steve Ahn <steveahnahn@g.ucla.edu>
@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Backport successfully created: v3-1-test

Status Branch Result
v3-1-test PR Link

potiuk pushed a commit that referenced this pull request Dec 2, 2025
(cherry picked from commit 4360616)

Co-authored-by: Steve Ahn <steveahnahn@g.ucla.edu>
ephraimbuddy pushed a commit that referenced this pull request Dec 3, 2025
(cherry picked from commit 4360616)

Co-authored-by: Steve Ahn <steveahnahn@g.ucla.edu>
RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Dec 3, 2025
Copilot AI pushed a commit to jason810496/airflow that referenced this pull request Dec 5, 2025
itayweb pushed a commit to itayweb/airflow that referenced this pull request Dec 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:task-sdk backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ti.get_first_reschedule_date method returns a string instead of a datetime object

5 participants