support Google App Engine's URL fetch service

[djc]

From acmo...@gmail.com on August 30, 2010 22:51:50

What steps will reproduce the problem? 1. use Google's App Engine SDK
2. install couchdb-python into your new project
3. import couchdb
2. attempt to connect to an external couchdb server with couch = couchdb.Server(' https://couchdb.example.com/' )
3. exception is raised What is the expected output? What do you see instead? Google has their own url fetch service in their app engine. < https://code.google.com/appengine/docs/python/urlfetch/overview.html >. Other than that, they appear to prevent applications from making external socket connections. It appears to me that since this code uses couchdb/http.py, it won't work on Google's app engine. Here's the stacktrace:

Traceback (most recent call last):
File "/home/amoore/dev/google_appengine/google/appengine/ext/webapp/init.py", line 511, in call
handler.get(_groups)
File "/home/amoore/dev/emradmin/main.py", line 13, in get
userdb = couch['user']
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 137, in getitem
db.resource.head() # actually make a request to the database
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 377, in head
return self._request('HEAD', path, headers=headers, *_params)
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 419, in _request
credentials=self.credentials)
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 239, in request
resp = _try_request_with_retries(iter(self.retry_delays))
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 196, in _try_request_with_retries
return _try_request()
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 211, in _try_request
if conn.sock is None:
AttributeError: HTTPConnection instance has no attribute 'sock'

If urllib, urllib2 or httplib were used instead, I think this would work. What version of the product are you using? On what operating system? python 2.5.2, Google App Engine, linux

Original issue: http://code.google.com/p/couchdb-python/issues/detail?id=147

[djc]

From matt.goo...@gmail.com on August 30, 2010 14:52:29

couchdb.http is actually just a layer over httplib so, in theory, it should work. Unfortunately, couchdb.http uses an undocumented 'sock' attribute of httplib.HTTPConnection that the Google API doesn't seem to provide.

'sock' is primarily used to re-open dead, cached connections but I don't see any other obvious, i.e. documented, way to test if a connection is open. I guess we could just rely on the retry code picking it up.

Note: couchdb.http also uses conn.sock.sendall but I suspect that could easily be replaced with conn.send.

Labels: -Type-Defect Type-Enhancement

[djc]

From acmo...@gmail.com on August 30, 2010 14:56:10

Aha. That makes sense. If you come up with a potential fix, and you don't want to jump through the hoops of getting an App Engine account, I can try it out for you. In the meantime, I'll try to hack my way into a "fix".

Thanks!

[djc]

From acmo...@gmail.com on August 31, 2010 09:02:10

Thanks for the pointers. I've attached a patch that removes the calls to 'sock'. The test suite seems to do about the same (there are 3 failures before and after the patch), and this patch works in Google's App Engine. I don't really know how to test the effects on timeouts.

Attachment: couchdb_google.diff

[djc]

From acmo...@gmail.com on August 31, 2010 15:00:12

hrm. It looks like they don't implement a 'isclosed' method, either:

File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 292, in iter
return iter([item.id for item in self.view('_all_docs')])
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 984, in iter
return iter(self.rows)
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 1003, in rows
self._fetch()
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 990, in _fetch
data = self.view._exec(self.options)
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 880, in _exec
_, _, data = self.resource.get_json(**self._encode_options(options))
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 394, in get_json
data = json.decode(data.read())
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 96, in read
self.close()
File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 100, in close
while not self.resp.isclosed():
File "/home/amoore/dev/google_appengine/google/appengine/dist/httplib.py", line 231, in getattr
return getattr(self.fp, attr)
AttributeError: StringIO instance has no attribute 'isclosed'

This means you can't iterate over a database.

I'll see if I can work around this, too.

[djc]

From polestn...@gmail.com on September 12, 2010 20:35:22

So, what about GAE support?

[djc]

From djc.ochtman on September 19, 2010 06:15:11

acmoore: that patch looks solid. Can you give me your name and an email address so we can credit you on the changeset?

[djc]

From amo...@mooresystems.com on September 19, 2010 07:08:34

You can credit it to: Andrew Moore amoore@mooresystems.com. Thank you!
I'm also willing to assign copyright to one of you, in case that helps with future maintenance.

I have a patch for the 'isclosed' problem, too. It's necessary in order to iterate over databases inside the GAE. I'll find it, clean it up, and submit it, too.

[djc]

From djc.ochtman on September 19, 2010 08:01:27

Pushed as r25df81c5d704 . Would be great if you come up with the other patch! Can you also show us what kind of test suite failures we get on GAE? Also, apparently we don't have iterating over databases covered in the test suite? Would be nice to fix that, too.

[djc]

From amo...@mooresystems.com on September 19, 2010 09:12:44

Thanks for pushing out that previous patch.

Here's the patch I'm using to get around the problem with .isclosed(). Now that I look at it, I'm not sure that it is only working because I always happen to get all of the data in the first read or not. Works for me, though.

Attachment: isclosed.diff

[vl4dt]

Hello, I don't know if this issue is being watched but I am having a similar issue only when trying to access a view's results, like this:

def count_users(client_id, collection=COLLECTION_CLIENT):
    """Counts the users for a given client: <client_id>."""
    couch = couchdb.Server(COUCHDB_URL)

    db = couch[collection]

    view = db.view('client/users')

    results = view[client_id]

    return len(results.rows)

The error I get is the following (excerpt):

  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/client.py", line 1196, in rows
    self._fetch()
  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/client.py", line 1182, in _fetch
    data = self.view._exec(self.options)
  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/client.py", line 1048, in _exec
    _, _, data = _call_viewlike(self.resource, options)
  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/client.py", line 1107, in _call_viewlike
    return resource.get_json(**_encode_view_options(options))
  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/http.py", line 510, in get_json
    return self._request_json('GET', path, headers=headers, **params)
  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/http.py", line 535, in _request_json
    data = json.decode(data.read().decode('utf-8'))
  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/http.py", line 153, in read
    self.close()
  File "/base/data/home/apps/s~liu-passport-api-qa/1.390979779811766718/lib/couchdb/http.py", line 162, in close
    self.resp.read(CHUNK_SIZE)
DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded.

I'd prefer not to query the database via urlfetch, is easy as well but, why bother? I just like the function as it is, simple, readable and clean 👍

[djc]

I don't really have access to GAE, so I can't work on a fix. If you come up with a fix that is sufficiently low-complexity/well-contained, I wouldn't mind merging it.

[pv-g]

@vl4dt I just bumped into the same issue.
Did you get to fix it? Or have a clue about the problem and which part should be fixed?

[pv-g]

@vl4dt @djc it seems the problem described here is a duplicate of #224.
And alexan...@codjovi.fr pointed out the right problem : an infinite loop within http:ResponseBody::close. I'll continue on their issue.