Adding autocommit property for Redshift queries for Vacuums, etc. #2242
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… can run VACUUMs in Redshift
Necessary to run VACUUM in Redshift.
dlstadther
reviewed
Sep 27, 2017
luigi/contrib/postgres.py
Outdated
| @@ -359,6 +360,8 @@ class PostgresQuery(rdbms.Query): | |||
|
|
|||
| def run(self): | |||
| connection = self.output().connect() | |||
| if self.autocommit: | |||
There was a problem hiding this comment.
You'll want to initialize this in rdbms.Query. Set its default to False?
dlstadther
requested changes
Sep 27, 2017
|
Okay, please check the new commit. I couldn't find an easy way of getting the autocommit status from the postgres server, so for now the test just uses the attribute on the connection. |
dlstadther
reviewed
Sep 27, 2017
luigi/contrib/rdbms.py
Outdated
| @@ -178,6 +182,10 @@ def table(self): | |||
| def query(self): | |||
| return None | |||
|
|
|||
| @abc.abstractproperty | |||
There was a problem hiding this comment.
I don't think this needs to be an abstractproperty. You shouldn't require the user to override it. Just a plain @property should suffice.
There was a problem hiding this comment.
Oops, I fixed this now. Thanks.
dlstadther
requested changes
Sep 27, 2017
|
Thanks, it looks good now :) |
dlstadther
approved these changes
Sep 27, 2017
|
Thanks @jamesmcm for contributing! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This solves #2047
I've added the ability to override the autocommit property in Postgres Queries (used by Redshift Queries) - this will allow the user to make queries in autocommit mode as some operations require this.
The user only needs to set:
autocommit = Truein their RedshiftQuery subclass to make use of it.
Specifically this was done so that we could run VACUUMs in Redshift from Luigi.
I've tested it locally against our database, but I couldn't get the tox tests to run on my machine.
(Sorry for the 2 separate commits, it seems my attempt at squashing them failed)