Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions providers/sftp/docs/sensors/sftp_sensor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,26 @@ To get more information about this sensor visit :class:`~airflow.providers.sftp.

We can also use TaskFlow API. It takes the same arguments as the :class:`~airflow.providers.sftp.sensors.sftp.SFTPSensor` along with -

python_callable (optional)
A callable that will be executed after files matching the sensor criteria are found.
This allows you to process the found files with custom logic. The callable receives:

- Positional arguments from ``op_args``
- Keyword arguments from ``op_kwargs``, with ``files_found`` automatically added
(if ``op_kwargs`` is provided and not empty) containing the list of files that matched
the sensor criteria

The return value of the callable is stored in XCom along with the ``files_found`` list,
accessible via ``{"files_found": [...], "decorator_return_value": <callable_return_value>}``.

op_args (optional)
A list of positional arguments that will get unpacked when
calling your callable (templated)
op_kwargs (optional)
A dictionary of keyword arguments that will get unpacked
in your function (templated)
A list of positional arguments that will get unpacked when calling your callable (templated).
Only used when ``python_callable`` is provided.

Whatever returned by the python callable is put into XCom.
op_kwargs (optional)
A dictionary of keyword arguments that will get unpacked in your function (templated).
If provided and not empty, the ``files_found`` list is automatically added to this dictionary
when the callable is invoked. Only used when ``python_callable`` is provided.

.. exampleinclude:: /../../sftp/tests/system/sftp/example_sftp_sensor.py
:language: python
Expand Down
9 changes: 9 additions & 0 deletions providers/sftp/src/airflow/providers/sftp/sensors/sftp.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@ class SFTPSensor(BaseSensorOperator):
:param file_pattern: The pattern that will be used to match the file (fnmatch format)
:param sftp_conn_id: The connection to run the sensor against
:param newer_than: DateTime for which the file or file path should be newer than, comparison is inclusive
:param python_callable: Optional callable that will be called after files are found. The callable
will receive the found files list in ``op_kwargs['files_found']`` if ``op_kwargs`` is provided
and not empty. The return value of the callable will be stored in XCom along with the
files_found list.
:param op_args: A list of positional arguments that will get unpacked when calling your callable
(templated). Only used when ``python_callable`` is provided.
:param op_kwargs: A dictionary of keyword arguments that will get unpacked in your callable
(templated). If provided and not empty, the ``files_found`` list will be automatically added
to this dictionary. Only used when ``python_callable`` is provided.
:param deferrable: If waiting for completion, whether to defer the task until done, default is ``False``.
"""

Expand Down