Storage: TypeError: request() takes at least 3 arguments (3 given) #3666
Comments
|
@kipi78 The issue is your transport: http = urllib3.PoolManager()
#authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=http)Until #1998 is closed, you must use an
Is there any reason you passed an explicit |
|
Thank you for your response. I wrote _http according to this doc: Okay, I read that it was not an httplib2 object but an urllib3 object, but I tried with a httplib2.Http. httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
#httpStorage = httplib2.Http()
authed_http = AuthorizedHttp(credentials, httpStorage)
storage_client = storage.Client(credentials=credentials, _http=httpStorage)How am I supposed to use that |
|
It seems to me that you should be passing I am pre-emptively closing this out, it seems you've figure out how to configure against your proxy? While waiting on #3674 / #1998, you may also want to check out https://github.com/GoogleCloudPlatform/httplib2shim |
|
Let me know if there are still concerns and we can re-open and discuss if need be. |
|
No, my comments meant what I tried to simplify the case. I already tried with credentials = service_account.Credentials.from_service_account_file(json)
httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
authed_http = AuthorizedHttp(credentials, httpStorage)
storage_client = storage.Client(credentials=credentials, _http=authed_http)
bucket = storage_client.get_bucket('mybucket')I will have a try with httplib2shim, thank you. |
|
So, with httplib2shim, with the sample code from the doc which uses GoogleCredentials from oauth2client.client, the problem is that google-cloud-python Client only supports credentials from google-auth-library-python. But authorize is not a method of credentials from google-cloud-python Client. My code: credentials = GoogleCredentials.get_application_default()
httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
httpStorage = credentials.authorize(httpStorage)
storage_client = storage.Client(credentials=credentials, _http=httpStorage)
bucket = storage_client.get_bucket('mybucket')As I need to use ca_certs, I tried to implement this:
with the bellow code: credentials = service_account.Credentials.from_service_account_file(json)
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat)
http = httplib2shim.Http(pool=http)
authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=authed_http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')Which still leads to the same error, whether I try with a simple PoolManager or with a ProxyManager. |
|
@jonparrott Can you weigh in here? |
|
I'm not able to reproduce, here's the code I used: >>> from google.cloud import storage
>>> import google.auth
>>> import google.auth.transport.urllib3
>>> import urllib3
>>> import httplib2shim
>>>
>>> pool = urllib3.PoolManager()
>>> credentials, project = google.auth.default()
>>> authed_pool = google.auth.transport.urllib3.AuthorizedHttp(credentials, http=pool)
>>> http = httplib2shim.Http(pool=authed_pool)
>>>
>>> client = storage.Client(project=project, _http=http)
>>> list(client.list_buckets())
[<Bucket: ...>, ...]
>>>I would strongly encourage waiting until #1998 is closed as this will all be significantly easier then. |
|
Okay, the thing is that you passed an AuthorizedHttp to httplib2shim.Http(), while I passed the ProxyManager and applied the AuthorizedHttp on httplib2shim.Http(). So, I don't have this error anymore with your code :) The thing is now to use the ProxyManager, it looks like I don't manage to do it. I need to configure a proxy with user/password and a certificate. My code bellow leads to this error:
proxyUrl="http://host:80"
scopes=["https://www.googleapis.com/auth/devstorage.read_write"]
credentials = service_account.Credentials.from_service_account_file(json, scopes=scopes)
default_headers = make_headers(proxy_basic_auth='user:password')
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, headers=default_headers)
authed_http = AuthorizedHttp(credentials, http)
http = httplib2shim.Http(pool=authed_http)
storage_client = storage.Client(_http=http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')I will certainly wait, many thanks. |
|
Can you make any requests using the proxymanager alone? Like just manager.get('http://httpbin.org')?
|
|
Yes it looks like the ProxyManager alone is working: http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, headers=default_headers)
a = http.request('GET','http://httpbin.org')
print(a.data)The expected HTML is printed. |
|
I wonder if the headers are being overridden. Why happens if you make a request and specify some random headers? |
|
As soon as I insert an error in the login/password in the headers, this small request does not work. Same thing if I try with random headers. So, I guess the headers are not overridden. |
|
Are you saying that adding any headers causes it to fail? Can I see the code and traceback? |
|
Here is the code: credentials = service_account.Credentials.from_service_account_file(json, scopes=scopes)
default_headers = make_headers(proxy_basic_auth='user:pass')
#default_headers = {'foo': 'bar'}
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, headers=default_headers)
a = http.request('GET','http://httpbin.org')
print(a.data)
authed_http = AuthorizedHttp(credentials, http)
http = httplib2shim.Http(pool=authed_http)
storage_client = storage.Client(_http=http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')With this version, the request to httpbin.org is printed but i got this traceback:
If I uncomment the second line for default_headers or type an uncorect user/password, I got the same error and traceback + the html page of proxy error of my organization. Do you need something else ? |
|
So with a bit of testing using this script and import google.auth.transport.urllib3
from google.cloud import storage
import httplib2shim
from urllib3 import ProxyManager
from urllib3.util import make_headers
proxy_url = "http://localhost:8080"
proxy_headers = make_headers(proxy_basic_auth='user:pass')
http = ProxyManager(proxy_url=proxy_url, proxy_headers=proxy_headers)
# Test a direct request to httpbin
a = http.request('GET', 'http://httpbin.org/headers')
print(a.data)
# Test a direct request to httpbin using google credentials
credentials, project = google.auth.default()
authed_http = google.auth.transport.urllib3.AuthorizedHttp(
credentials, http=http)
a = authed_http.request('GET', 'http://httpbin.org/headers')
print(a.data)
# Test a direct request using httplib2shim
shimmed_http = httplib2shim.Http(pool=authed_http)
response, content = shimmed_http.request('http://httpbin.org/headers')
print(response, content)
# Test a request using the client library
client = storage.Client(project=project, _http=shimmed_http)
print(list(client.list_buckets()))It seems the following is true:
I'm going to investigate some more, but it seems to confirm that the issue is that the |
|
Actually - I noticed something that makes a lot more sense: the I noticed your code is passing in If not, I would just wait until our next release where you can just configure requests globally to respect your proxy settings. |
|
Yes, it looks like it's better when passing credentials = service_account.Credentials.from_service_account_file(json, scopes=scopes)
default_headers = make_headers(proxy_basic_auth='user:pass')
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, proxy_headers=default_headers)
authed_http = AuthorizedHttp(credentials, http)
http = httplib2shim.Http(pool=authed_http)
storage_client = storage.Client(_http=http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')
blobs = bucket.list_blobs()
for blob in blobs:
print(blob.name)
#Download
blob.download_to_filename("/path/"+blob.name)
#Upload
b = bucket.blob(blob_name="testblob.txt")
b.upload_from_filename("/path/testblob.txt")Traceback is (for download):
|
|
That's because |
|
So, will it be possible to configure my own transport with proxy configuration or not ? |
|
Soon |
|
In the meantime, requests should honor the |
|
Uploading and downloading is now working using the |
|
Good deal. Glad we could help. :) |
|
I have installed the new release (pip install --upgrade google-cloud) so that I now have google-cloud==0.27.0, google-cloud-core==0.26.0 and google-cloud-storage==1.3.1. I don't need httplib2shim anymore, right ? I can't perform get_bucket. My code: credentials = service_account.Credentials.from_service_account_file(json, scopes=scopes)
default_headers = make_headers(proxy_basic_auth='user:pass')
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, proxy_headers=default_headers)
authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=authed_http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')I got the following traceback:
|
|
@kipi78 That's correct. Now the |
|
So, I don't understand. Am I doing something wrong here ? |
|
Yes / no. You no longer need to use |
|
I agree with that but I'm no longer using it as you can see in the code. |
|
Sorry I'm not familiar, where does |
|
|
|
@kipi78 now that requests is our default transport you don't need to use httplib2 or urllib3. You should be able to just set the |
|
Hi, Alright, I was wondering how to specify the certificate but it is a bundled feature of Thank you ! |
|
Great to hear!
…On Wed, Aug 9, 2017, 1:10 AM kipi78 ***@***.***> wrote:
Hi,
Alright, I was wondering how to specify the certificate but it is a
bundled feature of requests, we just need to export REQUESTS_CA_BUNDLE.
It is now working fine with a very simple storage_client =
storage.Client() :)
Thank you !
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3666 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAPUc8xcp8mktOll_IDV55nC3X2ooACnks5sWWllgaJpZM4OiXHB>
.
|
…eCloudPlatform/python-docs-samples#3666) Co-authored-by: Takashi Matsuo <tmatsuo@google.com>
Hi,
I'm having trouble while using google.cloud for Storage. I start by trying to list blobs in a bucket.
I'm trying to get threw the proxy of my organization. So I try to create my own http connection in order to specify the proxy, but even with a simple urllib3.PoolManager(), I keep getting the error and stacktrace bellow:
Error:
TypeError: request() takes at least 3 arguments (3 given)
Stacktrace:
My code is the following:
Python is 2.7.11.
I'm using Jupyter (notebook server is 5.0.0).
pip freeze:Click to expand
Please feel free to reformat this issue if that could help.
Many thanks.
The text was updated successfully, but these errors were encountered: