Add validate parameter to gcloud.storage.connection.get_bucket()

[jgeewax]

But make it default to False (http://www.appneta.com/blog/s3-list-get-bucket-default/)

[tseaver]

Hmm, I'm not seeing the point here. bucket = get_bucket("name") currently triggers a GET request to /b/name. That API doesn't support any "return all keys" option, so we cannot save the additional HTTP request triggered by bucket.get_all_keys().

Is this complication just for API compatibility with boto?

[jgeewax]

I think the intent here was to allow people to get all keys and the bucket in one go. (Note that S3's API doesn't support a "get all keys" option either, it's really doing a second HTTP call if validate=True and pulling in all the keys at "get time".)

I don't think this is super important, but the argument could be made for having it...

It means that code might look like:

# This does two HTTP requests.
bucket = get_bucket('name', fetch_all_keys=True)  # Or validate=True
print bucket.get_all_keys()  # Ideally fast because it's cached..?

Again, not married to this proposal, but worth discussing.

[tseaver]

The net time won't be any faster, as the first line will still consume the time for the second HTTP request.

Caching the keys takes us into the rabbit-hole of invalidation: we would need to add watch support too.

[jgeewax]

This is definitely not a blocker for a 1.0 release of gcloud.storage

/cc @tseaver

[dhermes]

I made an argument in #357 that this should be closed:

Yes the "get all keys" / "get all objects" concept is not something that really applies
to a mature application (one with more than a few thousand objects). As for the
people "just getting started", the two HTTP requests won't be such a big deal. What's
more, we shouldn't be encouraging people to get a list of "all objects" locally; they should
only use what they need (e.g. for serving a request for a signed-in user).

I'm closing it out with that in mind. @jgeewax @tseaver if you disagree please re-open and let's discuss.

[jgeewax]

I'm fine with closing this out -- but want to clarify that I'm -1 to removing a get_all_objects method (or a way to iterate over the keyspace).

[dhermes]

Sure thing, we want to keep it around (via _BlobIterator).