Refactored setup of google api clients

[freider]

Description

• Adds out-of-the-box support for google cli authentication auto-discovery (get_application_default)
• Reuses more code between gcs and bigquery implementations

Motivation and Context

Works with the current version of the cli tools and api client (google-api-python-client==1.6.1), which was previously not working (http and oauth could not both be provided as was the default/required behavior by the previous code).

##Tests
I ran tasks both for gcs (cloud storage) uploads and big query integration and both work after this patch. I don't see how it would break anything for people who use manual configuration of the authentication process, but it would of course be nice if someone who has a GCS/BigQuery pipeline running tested this out as well :)

[Tarrasch]

Oh, are you back at Spotify now? :)

There's also luigi tests for gcs which I recall are disabled on Travis. Have you run them too?

[freider]

Hi Arash! :-)
Not back at Spotify but back in Luigi land at least part-time as we're setting up some simple log processing at EdQu using BigQuery - so I can motivate spending more some time on luigi again. It's cool to see all the new stuff that's has been contributed.
Ah, I'll have a look and run the gcs tests locally to verify that it still works.

[freider]

I ran the tests gcs and bigquery tests with success (pre-patch they would not run using the current version of google-api-python-client (1.6.2)).
Noteworthy is that the tests won't run using python3 as there seem to be some issues there unrelated to this patch - guessing that code has never been tested on py3.

Any idea what is causing the travis fail for this PR? It just looks like jibberish to me :-)

[Tarrasch]

Yea the hadoop failures are jibberish. But the flake8 issue you certainly should be able to address. Here's you log, https://travis-ci.org/spotify/luigi/jobs/198885463. There's some docs in CONTRIBUTING.md too nowadays. :)

[freider]

Ah, should have seen that, thanks! :)

[Tarrasch]

LGTM. Maybe someone using gcp wants to review too?

[freider]

I wonder if many other use the gcp integration actively, considering the code prior to this patch doesn't work with the current google-api-python-client version... (so if they use it they probably use an old version and wouldn't upgrade luigi either ;) )

[Tarrasch]

@freider, I'm pretty sure there's others using it. Even Spotify is contributing to it still and I've seen various parties commenting to PRs, that said, nobody have commented for a while so I'll just merge this. :)