Add rows processor to GenericTransfer#61143
Merged
potiuk merged 37 commits intoapache:mainfrom Feb 10, 2026
Merged
Conversation
Nataneljpwd
reviewed
Jan 30, 2026
Contributor
Nataneljpwd
left a comment
There was a problem hiding this comment.
Looks great, I have left a few comments and questions, mainly nitpicks
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Outdated
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Outdated
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Outdated
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/triggers/sql.py
Outdated
Show resolved
Hide resolved
…onstructor of GenericTransfer
…onstructor of SQLInsertRowsOperator
Nataneljpwd
approved these changes
Jan 31, 2026
Contributor
Nataneljpwd
left a comment
There was a problem hiding this comment.
Looks great!
Only 1 question regarding the python interface file
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.pyi
Outdated
Show resolved
Hide resolved
Contributor
Author
Thx for to review @Nataneljpwd, really appreciated, well done! I think I've fixed all issues, awaiting CI/CD now. |
Nataneljpwd
reviewed
Feb 1, 2026
Contributor
Nataneljpwd
left a comment
There was a problem hiding this comment.
Looks great, 1 minor nitpick for type hinting, other than that, looks great
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Outdated
Show resolved
Hide resolved
Nataneljpwd
approved these changes
Feb 2, 2026
Contributor
Nataneljpwd
left a comment
There was a problem hiding this comment.
Looks great! just need to make the CI pass
9718eac to
9d3b2ef
Compare
… removed override of render_template_fields
Nataneljpwd
reviewed
Feb 2, 2026
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Show resolved
Hide resolved
Contributor
Author
|
@Nataneljpwd thx for the review and helping fixing the mypy issues! |
Nataneljpwd
approved these changes
Feb 4, 2026
Contributor
Nataneljpwd
left a comment
There was a problem hiding this comment.
Everything looks amazing, nice work!
shahar1
reviewed
Feb 6, 2026
providers/common/sql/src/airflow/providers/common/sql/triggers/sql.pyi
Outdated
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py
Show resolved
Hide resolved
providers/common/sql/src/airflow/providers/common/sql/operators/sql.py
Outdated
Show resolved
Hide resolved
…s/sql.py Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
potiuk
approved these changes
Feb 10, 2026
81 tasks
Alok-kumar-priyadarshi
pushed a commit
to Alok-kumar-priyadarshi/airflow
that referenced
this pull request
Feb 11, 2026
* refactor: Fixed docstring for rows_processor parameter in SQLInsertRowsOperator * refactor: Added rows_processor parameter in GenericTransfer * refactor: Make SQLExecuteQueryTrigger non-blocking from Airflow 3.2+ * refactor: Make inserting of rows in sync and deferred mode more DRY * refactor: Updated GenericTransfer type * refactor: Ignore mypy for _rows_processor * refactor: Removed _process_rows and made rows_processor optional in constructor of GenericTransfer * refactor: Removed _process_rows and made rows_processor optional in constructor of SQLInsertRowsOperator * refactor: Made _get_records in trigger protected * refactor: Updated typing of rows_processor in GenericTransfer interface * refactor: Fixed typing of rows parameter in _insert_rows method * refactor: Refactored unit test for SQLInsertRowsOperator * refactor: Fixted typing of _insert_rows method * refactor: Updated docstring for rows_processor parameter * refactor: Changed typing of context parameter from Context to Any in rows_processor parameter * refactor: Changed typing of rows_processor * refactor: Refactored _insert_rows method in SQLInsertRowsOperator and removed override of render_template_fields * refactor: Removed unused import of jinja2 * docs: Added comment to explain why rows_processor is typed like this * Update providers/common/sql/src/airflow/providers/common/sql/operators/sql.py Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com> * refactor: Removed get_records * refactor: Evaluate XComArgs first before evaluating rows * refactor: Fixed check on rows in SQLInsertRowsOperator --------- Co-authored-by: David Blain <david.blain@b-holding.be> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
Ratasa143
pushed a commit
to Ratasa143/airflow
that referenced
this pull request
Feb 15, 2026
* refactor: Fixed docstring for rows_processor parameter in SQLInsertRowsOperator * refactor: Added rows_processor parameter in GenericTransfer * refactor: Make SQLExecuteQueryTrigger non-blocking from Airflow 3.2+ * refactor: Make inserting of rows in sync and deferred mode more DRY * refactor: Updated GenericTransfer type * refactor: Ignore mypy for _rows_processor * refactor: Removed _process_rows and made rows_processor optional in constructor of GenericTransfer * refactor: Removed _process_rows and made rows_processor optional in constructor of SQLInsertRowsOperator * refactor: Made _get_records in trigger protected * refactor: Updated typing of rows_processor in GenericTransfer interface * refactor: Fixed typing of rows parameter in _insert_rows method * refactor: Refactored unit test for SQLInsertRowsOperator * refactor: Fixted typing of _insert_rows method * refactor: Updated docstring for rows_processor parameter * refactor: Changed typing of context parameter from Context to Any in rows_processor parameter * refactor: Changed typing of rows_processor * refactor: Refactored _insert_rows method in SQLInsertRowsOperator and removed override of render_template_fields * refactor: Removed unused import of jinja2 * docs: Added comment to explain why rows_processor is typed like this * Update providers/common/sql/src/airflow/providers/common/sql/operators/sql.py Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com> * refactor: Removed get_records * refactor: Evaluate XComArgs first before evaluating rows * refactor: Fixed check on rows in SQLInsertRowsOperator --------- Co-authored-by: David Blain <david.blain@b-holding.be> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
choo121600
pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
* refactor: Fixed docstring for rows_processor parameter in SQLInsertRowsOperator * refactor: Added rows_processor parameter in GenericTransfer * refactor: Make SQLExecuteQueryTrigger non-blocking from Airflow 3.2+ * refactor: Make inserting of rows in sync and deferred mode more DRY * refactor: Updated GenericTransfer type * refactor: Ignore mypy for _rows_processor * refactor: Removed _process_rows and made rows_processor optional in constructor of GenericTransfer * refactor: Removed _process_rows and made rows_processor optional in constructor of SQLInsertRowsOperator * refactor: Made _get_records in trigger protected * refactor: Updated typing of rows_processor in GenericTransfer interface * refactor: Fixed typing of rows parameter in _insert_rows method * refactor: Refactored unit test for SQLInsertRowsOperator * refactor: Fixted typing of _insert_rows method * refactor: Updated docstring for rows_processor parameter * refactor: Changed typing of context parameter from Context to Any in rows_processor parameter * refactor: Changed typing of rows_processor * refactor: Refactored _insert_rows method in SQLInsertRowsOperator and removed override of render_template_fields * refactor: Removed unused import of jinja2 * docs: Added comment to explain why rows_processor is typed like this * Update providers/common/sql/src/airflow/providers/common/sql/operators/sql.py Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com> * refactor: Removed get_records * refactor: Evaluate XComArgs first before evaluating rows * refactor: Fixed check on rows in SQLInsertRowsOperator --------- Co-authored-by: David Blain <david.blain@b-holding.be> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Was generative AI tooling used to co-author this PR?
This PR extends the GenericTransfer operator with a rows_processor parameter, similar to SQLInsertRowsOperator.
The rows_processor allows users to post-process rows returned by the source hook before they are inserted into the destination hook.
This enables lightweight transformations (filtering, normalization, enrichment, etc.) without introducing an intermediate task, while keeping database-specific type handling encapsulated in the hook itself.
In addition, this PR makes SQLExecuteQueryTrigger non-blocking when running on Airflow 3.2 or higher.
Previously, the trigger had to remain blocking due to a bug in the CommsDecoder that could raise a “Response read out of order” error when handling asynchronous trigger events. With this issue fixed in Airflow 3.2, the trigger can now safely run in a non-blocking mode, improving triggerer scalability and resource usage.
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.