Skip to content

Conversation

@nguy4130
Copy link
Contributor

@nguy4130 nguy4130 commented Oct 9, 2025

Fixes: #41472

Adding fail_on_file_not_exist param to SFTPToGCSOperator so that user can configure the parameter and operator will not fail in case of sftp file not exist.

Note that this is only added to the non-wildcard source file path as that code already kind of handles when the file does not exist.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Tested with a local DAG

from airflow import DAG
from airflow.providers.google.cloud.transfers.sftp_to_gcs import SFTPToGCSOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "airflow",
    "retries": 1,
    "retry_delay": timedelta(minutes=1),
}

with DAG(
    dag_id="sftp_to_gcs_example",
    default_args=default_args,
    description="Transfer a file from SFTP to GCS",
    start_date=datetime(2023, 1, 1),
    catchup=False,
    tags=["example", "sftp", "gcs"],
) as dag:
    transfer_file = SFTPToGCSOperator(
        task_id="transfer_file",
        source_path="test.mv.dba",
        sftp_conn_id="sftp_default",
        gcp_conn_id="google_cloud_default",
        destination_bucket="airflow",
        destination_path="test/test.mv.db",
        fail_on_file_not_exist=False,
    )

when fail_on_file_not_exist=False, the DAG exited gracefully
Screenshot 2025-10-09 at 4 52 23 PM

when fail_on_file_not_exist=True or not set, the DAG failed with error (existing behavior)
Screenshot 2025-10-09 at 4 53 00 PM

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Oct 9, 2025
@nguy4130 nguy4130 marked this pull request as ready for review October 9, 2025 23:54
@potiuk
Copy link
Member

potiuk commented Oct 14, 2025

Nice - but CI does not agree :)

@potiuk
Copy link
Member

potiuk commented Oct 16, 2025

Still :(

@nguy4130
Copy link
Contributor Author

@potiuk CI should be happy now!

@potiuk
Copy link
Member

potiuk commented Oct 19, 2025

@potiuk CI should be happy now!

🤞

@potiuk potiuk merged commit 2e1422b into apache:main Oct 19, 2025
76 checks passed
@nguy4130 nguy4130 deleted the fail_on_sftp_file_not_exist_gcs branch October 19, 2025 20:39
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Oct 20, 2025
* Add fail_on_file_not_exist to SFTPToGCSOperator

* working with local DAG

* working with local DAG

* Trigger Build

* fix test

* fix unit test case

---------

Co-authored-by: John Nguyen <john.nguyen@creditkarma.com>
TyrellHaywood pushed a commit to TyrellHaywood/airflow that referenced this pull request Oct 22, 2025
* Add fail_on_file_not_exist to SFTPToGCSOperator

* working with local DAG

* working with local DAG

* Trigger Build

* fix test

* fix unit test case

---------

Co-authored-by: John Nguyen <john.nguyen@creditkarma.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airflow SFTPToGCSOperator sftp file exist check

4 participants