Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix SFTPSensor.newer_than not working with jinja logical ds/ts expression #39056

Merged
merged 39 commits into from
May 6, 2024
Merged

Fix SFTPSensor.newer_than not working with jinja logical ds/ts expression #39056

merged 39 commits into from
May 6, 2024

Conversation

grrolland
Copy link
Contributor

closes: #36629

Add str to time transformation in case the newer_than parameter is templated.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Apr 16, 2024

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@grrolland grrolland mentioned this pull request Apr 16, 2024
Closed
2 tasks
@eladkal eladkal changed the title Fixes https://github.com/apache/airflow/issues/36629 Fix SFTPSensor.newer_than not working with jinja logical ds/ts expression Apr 16, 2024
Taragolis
Taragolis reviewed Apr 17, 2024
View reviewed changes
tests/providers/sftp/sensors/test_sftp.py Outdated Show resolved Hide resolved
airflow/providers/sftp/sensors/sftp.py Outdated Show resolved Hide resolved
Grégoire Rolland and others added 13 commits April 19, 2024 09:21
Fixes apache#36629
3401b6e
Fixes PR failed test
f350253
Remove an parametrize duplicate tests
79e3f7b
@hussein-awala
update simple-salesforce type hints to support 1.12.6 (apache#39047)
0f0a37f
Fix formatting
4ced567
@ephraimbuddy
Add changelog for airflow python client 2.9.0 (apache#39060)
d976f58
@potiuk
Upgrade to latest hatchling as build dependency (apache#39044)
8b7dcc4
@eladkal
Prepare docs 1st wave (RC3) + ad hoc April 2024 (apache#38995) (apach…
6dc8c05 
…e#39054)

* Prepare docs 1st wave (RC3) + ad hoc April 2024 (apache#38995)

* update databricks
@rawwar
[docs] update DagBag class docstring to include all params (apache#…
a06349e 
…38814)

* update docstring for DagBag class

* break long line

* fix space

Signed-off-by: kalyanr <kalyan.ben10@live.com>

---------

Signed-off-by: kalyanr <kalyan.ben10@live.com>
@lzdanski
Data aware scheduling docs edits (apache#38687)
4616ced
@potiuk
Moves airflow import in deprecated pod_generator to local (apache#39062)
d8b4f23 
The import might be invoked when K8S executor starts with sentry on
and it might lead to circular imports

Related: apache#31442
@jedcunningham
KPO xcom sidecar PodDefault usage (apache#38951)
d1d2fca 
We should use the same, non deprecated, version of PodDefaults for the
xcom sidecar when creating and reading xcom.
Fix formatting
e8ba0d7
Grégoire Rolland and others added 9 commits April 19, 2024 09:21
Change date/time parsing method for newer_than parameter un SFTPSensor
1cd9e2f
@vincbeck
Add examples in AWS auth manager documentation (apache#39040)
7fdb2bf
@humit0
update document (apache#39068)
ee81403
@ephraimbuddy
Update hatchling to version 1.24.0 (apache#39072)
2d47202
@bbovenzi
Check that the dataset<>task exists before trying to render graph (ap…
e55e09c 
…ache#39069)
Merge remote-tracking branch 'origin/fix-36629-sftp-sensor-newer-than…
ca700e0 
…-templating' into fix-36629-sftp-sensor-newer-than-templating

# Conflicts:
#	airflow/providers/sftp/sensors/sftp.py
#	tests/providers/sftp/sensors/test_sftp.py
Change date/time parsing method for newer_than parameter un SFTPSensor
6ba4b7c
@grrolland
Merge branch 'main' into fix-36629-sftp-sensor-newer-than-templating
fe9b6a9
@grrolland
Merge branch 'main' into fix-36629-sftp-sensor-newer-than-templating
1daa5c2
@Taragolis Taragolis merged commit 8965f2e into apache:main May 6, 2024
40 checks passed
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented May 6, 2024

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

RodrigoGanancia pushed a commit to RodrigoGanancia/airflow that referenced this pull request May 10, 2024
@grrolland @hussein-awala
@ephraimbuddy @potiuk @eladkal @rawwar @lzdanski @jedcunningham @vincbeck @humit0 @bbovenzi @RodrigoGanancia
Fix SFTPSensor.newer_than not working with jinja logical ds/ts expres…
6888036 
…sion (apache#39056)

* Fixes apache#36629

* Fixes PR failed test

* Remove an parametrize duplicate tests

* Fix formatting

* Fix formatting

* Fixes apache#36629

* Fixes PR failed test

* Remove an parametrize duplicate tests

* update simple-salesforce type hints to support 1.12.6 (apache#39047)

* Fix formatting

* Add changelog for airflow python client 2.9.0 (apache#39060)

* Upgrade to latest hatchling as build dependency (apache#39044)

* Prepare docs 1st wave (RC3) + ad hoc April 2024 (apache#38995) (apache#39054)

* Prepare docs 1st wave (RC3) + ad hoc April 2024 (apache#38995)

* update databricks

* [docs] update `DagBag` class docstring to include all params (apache#38814)

* update docstring for DagBag class

* break long line

* fix space

Signed-off-by: kalyanr <kalyan.ben10@live.com>

---------

Signed-off-by: kalyanr <kalyan.ben10@live.com>

* Data aware scheduling docs edits (apache#38687)

* Moves airflow import in deprecated pod_generator to local (apache#39062)

The import might be invoked when K8S executor starts with sentry on
and it might lead to circular imports

Related: apache#31442

* KPO xcom sidecar PodDefault usage (apache#38951)

We should use the same, non deprecated, version of PodDefaults for the
xcom sidecar when creating and reading xcom.

* Fix formatting

* Change date/time parsing method for newer_than parameter un SFTPSensor

* Add examples in AWS auth manager documentation (apache#39040)

* update document (apache#39068)

* Update hatchling to version 1.24.0 (apache#39072)

* Check that the dataset<>task exists before trying to render graph (apache#39069)

* Change date/time parsing method for newer_than parameter un SFTPSensor

* Fix utc timezone in unit tests

* Fix utc timezone in unit tests

---------

Signed-off-by: kalyanr <kalyan.ben10@live.com>
Co-authored-by: Grégoire Rolland <gregoire.rolland@mymoneybank.com>
Co-authored-by: Hussein Awala <hussein@awala.fr>
Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
Co-authored-by: Kalyan <kalyan.ben10@live.com>
Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: humit <jhjang1005@naver.com>
Co-authored-by: Brent Bovenzi <brent@astronomer.io>
pateash pushed a commit to pateash/airflow that referenced this pull request May 13, 2024
@grrolland @hussein-awala
@ephraimbuddy @potiuk @eladkal @rawwar @lzdanski @jedcunningham @vincbeck @humit0 @bbovenzi @pateash
Fix SFTPSensor.newer_than not working with jinja logical ds/ts expres…
105979e 
…sion (apache#39056)

* Fixes apache#36629

* Fixes PR failed test

* Remove an parametrize duplicate tests

* Fix formatting

* Fix formatting

* Fixes apache#36629

* Fixes PR failed test

* Remove an parametrize duplicate tests

* update simple-salesforce type hints to support 1.12.6 (apache#39047)

* Fix formatting

* Add changelog for airflow python client 2.9.0 (apache#39060)

* Upgrade to latest hatchling as build dependency (apache#39044)

* Prepare docs 1st wave (RC3) + ad hoc April 2024 (apache#38995) (apache#39054)

* Prepare docs 1st wave (RC3) + ad hoc April 2024 (apache#38995)

* update databricks

* [docs] update `DagBag` class docstring to include all params (apache#38814)

* update docstring for DagBag class

* break long line

* fix space

Signed-off-by: kalyanr <kalyan.ben10@live.com>

---------

Signed-off-by: kalyanr <kalyan.ben10@live.com>

* Data aware scheduling docs edits (apache#38687)

* Moves airflow import in deprecated pod_generator to local (apache#39062)

The import might be invoked when K8S executor starts with sentry on
and it might lead to circular imports

Related: apache#31442

* KPO xcom sidecar PodDefault usage (apache#38951)

We should use the same, non deprecated, version of PodDefaults for the
xcom sidecar when creating and reading xcom.

* Fix formatting

* Change date/time parsing method for newer_than parameter un SFTPSensor

* Add examples in AWS auth manager documentation (apache#39040)

* update document (apache#39068)

* Update hatchling to version 1.24.0 (apache#39072)

* Check that the dataset<>task exists before trying to render graph (apache#39069)

* Change date/time parsing method for newer_than parameter un SFTPSensor

* Fix utc timezone in unit tests

* Fix utc timezone in unit tests

---------

Signed-off-by: kalyanr <kalyan.ben10@live.com>
Co-authored-by: Grégoire Rolland <gregoire.rolland@mymoneybank.com>
Co-authored-by: Hussein Awala <hussein@awala.fr>
Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
Co-authored-by: Kalyan <kalyan.ben10@live.com>
Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: humit <jhjang1005@naver.com>
Co-authored-by: Brent Bovenzi <brent@astronomer.io>
@eladkal eladkal mentioned this pull request May 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dirrao dirrao dirrao left review comments

@Taragolis Taragolis Taragolis approved these changes

Assignees
No one assigned
Labels
area:providers provider:sftp
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SFTPSensor.newer_than not working with jinja logical ds/ts expression