Skip to content

Conversation

@dabla
Copy link
Contributor

@dabla dabla commented Aug 28, 2025

As already discussed previously on the devlist this PR adds the SQLInsertRowsOperator in the common sql provider. Main advantage of having such operators is that you don't need a PythonOperator or a task decorated python method in your DAG to allow inserting data from an XCom, you just call the operator, which allows you to keep your DAG clean and readable.

An example on how to use the operator:

insert_rows = SQLInsertRowsOperator(
        task_id="insert_rows",
        table_name="actors",
        columns=[
            "name",
            "firstname",
            "age",
        ],
        rows=[
            ("Stallone", "Sylvester", 78),
            ("Statham", "Jason", 57),
            ("Li", "Jet", 61),
            ("Lundgren", "Dolph", 66),
            ("Norris", "Chuck", 84),
        ],
        preoperator=[
            """
            CREATE TABLE IF NOT EXISTS actors (
                index BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
                name TEXT NOT NULL,
                firstname TEXT NOT NULL,
                age BIGINT NOT NULL
            );
            """,
            "TRUNCATE TABLE actors;",
        ],
        postoperator="DROP TABLE IF EXISTS actors;",
        insert_args={
            "commit_every": 1000,
            "autocommit": False,
            "executemany": True,
            "fast_executemany": True,
        },
    )

It also offers some convienience methods like rows_processor, preoperator (like with GenericTransfer) and postoperator. It accepts the same parameters as the insert_rows method from the DbApiHook.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@potiuk
Copy link
Member

potiuk commented Aug 29, 2025

Tests failing :)

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just static check should be fixed.

@dabla
Copy link
Contributor Author

dabla commented Aug 30, 2025

Looks good. Just static check should be fixed.

Fixed

@dabla dabla requested a review from potiuk August 30, 2025 19:28
@potiuk potiuk merged commit ff7d3e0 into apache:main Aug 31, 2025
77 checks passed
nothingmin pushed a commit to nothingmin/airflow that referenced this pull request Sep 2, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Sep 30, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 1, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 2, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 3, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 4, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 7, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 8, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 9, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 10, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 11, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 12, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 14, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 15, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 17, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
* refactor: Added SQLInsertRowsOperator

* docs: Added documentation for the SQLInsertRowsOperator

* docs: Added example file for the SQLInsertRowsOperator

* refactor: Fixed doc GenericTransfer:

* refactor: Fixed static checks SQLInsertRowsOperator

* refactor: Made conn_id optional in SQLInsertRowsOperator

* refactor: Made rows optional in SQLInsertRowsOperator to fix PEP 484 issue

* refactor: Removed duplicate blank line in SQLInsertRowsOperator docstring

* refactor: Moved jinja to type checking block

* refactor: Put summary on one line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants