storage: API proposal

[ryanseys]

New

http://stephenplusplus.github.io/gcloud-node/#/docs/master/storage

Old

Moved to docs: https://docs.google.com/document/d/1SG7DYKEjzX8MiYcPtrKunprT6fgLXIFDsdoVA8ETdow

I took some time to go through what calls the current storage API supports and tried to redesign them to be easier to understand and encapsulate what most developers would like to do with the api. This covers buckets, files, and ACL on them both. The API I propose is below. Let me know if I missed anything or you have questions or a better idea 👍

Initialization

var gcloud = require('gcloud')({ /* optional stuff like credentials */ });
var storage = gcloud.storage;

Buckets

Reference to a bucket (local only, no network request)

var myBucket = storage.bucket('ryan-files');

Create a bucket

storage.createBucket('ryan-files', cb);

Delete a bucket

storage.deleteBucket('ryan-files', cb);

List all buckets

storage.listBuckets(cb);

Get metadata about a bucket

storage.getBucket('bucket-name', cb);
// or (pick one)
storage.bucket('ryan-files').getInfo(cb);

Files/Objects

Reference to a file/object in a bucket

myBucket.file('stephen.png');
// or (pick one)
myBucket.object('stephen.png');

Upload an object to a bucket

myBucket.upload('local_filename.png', { name: 'remote_filename.png' /*, other opts too */ }, cb);
// or more simply
myBucket.upload('local_filename.png', 'remote_filename.png', cb);

Remove an object from a bucket

myBucket.remove('remote_filename.png', cb);
// or (pick one)
myBucket.delete('remote_filename.png', cb);

Copy an object from one bucket to another

myBucket.copy('ryan.png', storage.bucket('stephen-files').file('stephen.png'), cb);

ACLs

Create an ACL?

List of scope/permission pairs (may be overkill?)

var myACL = storage.acl(['entity', 'permission', 'entity2', 'permission2']);

Set default object ACL for all objects added to bucket

myBucket.setDefaultACL(myACL, cb);

Clear default ACL for all objects added to bucket

myBucket.clearDefaultACL(myACL, cb);

Get default ACL for all objects added to bucket

myBucket.getDefaultACL(myACL, cb);

Set ACL of bucket

myBucket.setACL(myACL, cb);

Set ACL of file/object

myBucket.file('myfile.png').setACL(myACL, cb);

Clear custom ACL for file/object

myBucket.file('myfile.png').clearACL(cb);

Clear custom ACL for bucket

myBucket.clearACL(cb);

Get ACL of a file/object

myBucket.file('myfile.png').getACL(cb);

Get ACL of a bucket

myBucket.getACL(cb);

Channels

Stop watching resources through this channel

This is provided by storage.channels.stop() in google-api-nodejs-client

storage.stopChannels();

[stephenplusplus]

I like this proposal 💯

Get metadata about a bucket

storage.getBucket('bucket-name', cb);
// or (pick one)
storage.bucket('ryan-files').getInfo(cb);

storage.bucket()

Though I prefer getMetadata.

Reference to a file/object in a bucket

myBucket.file('stephen.png');
// or (pick one)
myBucket.object('stephen.png');

Thanks for the shout out! :)

I really like the file object. It allows us another child in the hierarchy, allowing logical separation of actions. I think they should be full featured objects, removing a lot of functionality from bucket. Consider if all file objects are duplex streams:

var ryan = myBucket.file('ryan.gif');

// upload a file to your bucket's ryan.gif file
fs.createReadStream('local_file.png').pipe(ryan);

// pipe a readable stream of your bucket's ryan.gif file to a destination
ryan.pipe(fs.createWriteStream('local_ryan.gif'));

// copy to another bucket - replace myBucket.copy
ryan.pipe(anotherBucket.file('ryan-clone.gif'));

// get file metadata - replace myBucket.stat
ryan.stat(cb);

// set file metadata - replace myBucket.write's metadata write only functionality
ryan.setMetadata({});

// delete file - replace myBucket.delete
ryan.delete(cb); // (sorry)

Footnote: let's do a little better housekeeping. This is the result of a lengthy discussion at #168 based around supporting the creation of buckets. I agree with flushing that convo and starting anew, but we should make sure these discussions are linked, and old ones are closed.

[ryanseys]

If ryan.delete(cb) deletes the file then why doesn't myBucket.delete(cb) delete the bucket? (we had this whole delegation conversation earlier).

If we provide ryan.pipe(anotherBucket.file('ryan-clone.gif')); then there's no reason why we shouldn't or couldn't also provide myBucket.copy() (for developers who don't want to use the streaming interface and just want plain methods).

[stephenplusplus]

If ryan.delete(cb) deletes the file then why doesn't myBucket.delete(cb) delete the bucket? (we had this whole delegation conversation earlier).

Fair. But, the other part of my point in regards to myBucket.delete was it would be confused with remove, which removes a file. If we have a file object to handle file things, I have less of an issue with myBucket.delete and myFile.delete. Did you agree with me from the other conversation? How do you look at it/which do you prefer?

Sure, we can provide quick methods as well, and we can internally use the stream api:

myBucket.copy('local_file.png', anotherBucket.file('ryan-clone.gif'));
// where...
Bucket.prototype.copy = function(localFilepath, destinationFileObject, callback) {
  fs.createReadStream(localFilepath).pipe(destinationFileObject)
    .on('error', callback)
    .on('complete', callback);
};

[ryanseys]

Yes I like myBucket.delete(cb) delete a bucket and myBucket.getMetadata(cb) get bucket metadata etc. The one thing you didn't like from the previous discussion was myBucket.create(cb) and storage.listBuckets() would still have to remain.

[stephenplusplus]

The one thing you didn't like from the previous discussion was myBucket.create(cb) and storage.listBuckets() would still have to remain.

I may not understand you, but isn't it now storage.createBucket? I like creating a bucket from the storage object, not from an already created Bucket instance. When I have a Bucket, I think "this refers to an existing remote endpoint," so creating one from that object breaks that model. That's not the same for deleting, as it makes sense to delete an already existing bucket from a Bucket instance.

And storage.listBuckets is cool with me. I think I suggested getBuckets or listBuckets, and I'm still +1 to either.

[ryanseys]

Yeah, this makes sense. storage.createBucket() is better than myBucket.create() and either listBuckets or getBuckets, I'm indifferent.

[silvolu]

Thanks guys, this is a great discussion!
Could we move it to a public Google Doc? That would allow us to keep an updated snapshot of the proposed API at the top of the doc and a track of considered alternatives, and I think it would make it clearer for newcomers to follow the discussion and understand where are we at.

[rakyll]

Could we move this to Docs? I have comments,

[stephenplusplus]

Moved to docs: https://docs.google.com/document/d/1SG7DYKEjzX8MiYcPtrKunprT6fgLXIFDsdoVA8ETdow

[stephenplusplus]

@silvolu - should I jump on putting this into code next?

[silvolu]

I cannot comment ON the doc :/
@rakyll did you manage to add your comments on this proposal?

[ryanseys]

@silvolu added you to the doc

[stephenplusplus]

@ryanseys @silvolu http://stephenplusplus.github.io/gcloud-node/#/docs/master/storage

[ryanseys]

I don't think it's recommended convention to name your buckets with a
capital and especially not with the word Bucket in the name. It's
redundant. We should keep with a default project name styled naming
convention, using all lower case words and numbers separated by dashes.
Developers will most likely adopt our conventions if they are clear and
consistent enough for them to follow.

string and options.name are backward in the bucket function docs table :)

[stephenplusplus]

makes-sense. Updated :)

[ryanseys]

string and options.name still backward :)

[ryanseys]

stroage.getBuckets({ should be storage.getBuckets({

[ryanseys]

Initial example of uploading and downloading a file should be more prominent. Also, can we include that example in the bucket.upload example as an alternative.

Also, if we have an upload function, can we also have a download function? Or is that kinda silly?

Edit: Be more prominent by perhaps updating the header from "Example" to something more descriptive like "Example uploading and downloading a file"

[stephenplusplus]

Initial example of uploading and downloading a file should be more prominent.

Which example?

string and options.name still backward :)

What should it look like?

[ryanseys]

The main example in the File description.

They should swap cells. Should be options.name then string. See the row above it, it has options then string,object