Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding autocommit property for Redshift queries for Vacuums, etc. #2242

Merged
merged 5 commits into from
Sep 27, 2017
Merged

Adding autocommit property for Redshift queries for Vacuums, etc. #2242

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0d9f227
Add autocommit property to Postgres and Redshift query classes, so we…
jamesmcm Sep 27, 2017
09231c8
Added autocommit property to Postgres and Redshift queries.
jamesmcm Sep 27, 2017
4ba32e1
Defined autocommit in rdbms.py and added test
jamesmcm Sep 27, 2017
2883e71
Change abstractproperty to property for autocommit
jamesmcm Sep 27, 2017
c423f6d
Fixing flake problems
jamesmcm Sep 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion luigi/contrib/postgres.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,8 @@ class PostgresQuery(rdbms.Query):
Template task for querying a Postgres compatible database

Usage:
Subclass and override the required `host`, `database`, `user`, `password`, `table`, and `query` attributes.
Subclass and override the required `host`, `database`, `user`, `password`, `table`, and `query` attributes.
Optionally one can override the `autocommit` attribute to put the connection for the query in autocommit mode.

Override the `run` method if your use case requires some action with the query result.

Expand All @@ -359,6 +360,8 @@ class PostgresQuery(rdbms.Query):

def run(self):
connection = self.output().connect()
if self.autocommit:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll want to initialize this in rdbms.Query. Set its default to False?

connection.autocommit=self.autocommit
cursor = connection.cursor()
sql = self.query

Expand Down
3 changes: 2 additions & 1 deletion luigi/contrib/redshift.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -639,7 +639,8 @@ class RedshiftUnloadTask(postgres.PostgresQuery, _CredentialsMixin):
Template task for running UNLOAD on an Amazon Redshift database

Usage:
Subclass and override the required `host`, `database`, `user`, `password`, `table`, and `query` attributes.
Subclass and override the required `host`, `database`, `user`, `password`, `table`, and `query` attributes.
Optionally, override the `autocommit` atribute to run the query in autocommit mode - this is necessary to run VACUUM for example.
Override the `run` method if your use case requires some action with the query result.
Task instances require a dynamic `update_id`, e.g. via parameter(s), otherwise the query will only execute once
To customize the query signature as recorded in the database marker table, override the `update_id` property.
Expand Down