-
Notifications
You must be signed in to change notification settings - Fork 14.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add RedshiftSQLHook, RedshiftSQLOperator #18447
Add RedshiftSQLHook, RedshiftSQLOperator #18447
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small things... looking good though 👍
One bigger thing is ... should we take this opportunity to replace the existing RedshiftHook (which is about managing clusters) with the one added in this PR (since this by far the more useful and needed one, and more deserving of the name RedshiftHook)
1efe5a9
to
7205a8c
Compare
ddd9d49
to
a5e8786
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need a whole new hook for this -- the existing Postgres hook will already connect to Redshift
_If we decide we do want a hook for this anyway then it should probably subclass that postgres hook.
Geometry and Super are returned as strings -- isn't that the default behaviour for pscopg already? |
@ashb, PostgresHook does not support authenticating via identity provider. That functionality is a big part of #16355. IMO It would make more sense to have a separate hook for Redshift as it's a different product than Postgres. PostgresHook is using a driver designed for Postgres. RedshiftSQLHook would use a driver designed for Redshift.
Geometry can also be hex encoded under OID 3999, and requires manipulation by the driver before being returned to the user. |
…> RedshiftSQLOperator
…shift_conn_id value
…ault redshift_conn_id value" This reverts commit a02cdbc14871416c1449acd8c49103d518c4f279.
…onnection examples for IAM, IDP
This reverts commit 76a6e93083a5a9ca8cb6f7f57c5a1a7c46f40da7.
32832d3
to
c3b4e48
Compare
Awesome work, congrats on your first merged pull request! |
closes: #16355 #16544
Adds
RedshiftSQLHook
andRedshiftSQLOperator
, which allow Airflow users to execute Redshift operations.RedshiftSQLHook
is supported by theredshift_connector
database driver which supports authenticating via IAM or Identity Provider. Integration withsqlalchemy
is supported viasqlalchemy-redshift
.Unit tests have been added to:
redshift_connector
as expectedredshift_connector.execute
method as expectedDocs have been added to:
redshift_connector
docs for details about supported connection optionsUse case / motivation
There is now a redshift_connector library which we can use instead of psycopg2. The redshift connector library provides support for IAM & Identity provider authentication as well as Redshift specific datatype support (e.g. GEOMETRY, SUPER, etc.).
See #16355 for more info.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.