Support "Requester-Pays" billing in google-cloud-nio

[droazen]

No description provided.

[droazen]

@jean-philippe-martin It's looking like @Horneth won't have time to implement this in the short-term future -- since you mentioned that the change was fairly simple, would you be able to submit a PR yourself? We'd be happy to help review.

[Horneth]

A few thoughts on this:

One important thing to be aware of about RP is that specifying the billing project in the request to GCS incurs cost to the requester even if the bucket does not have requester pays enabled.
Because of that, I think NIO should only set the billing project if the caller explicitly asked for it. It would be the responsibility of the caller to determine whether or not they want to set the billing project.
They could either

1. never set it at first, and then re-issue the NIO operation with a billing project if the first attempt failed due to the bucket having RP enabled
2. check first separately if RP is enabled and pass the billing project to NIO accordingly

Or, NIO could implement one of those 2 options itself so that it's transparent to the user but they have less control (in which case I would favor option 1 over 2).

[droazen]

@Horneth Ideally we'd just check up-front whether the bucket has requester pays enabled, and specify the user's default project as the billing project if it is.

It would also be good, I think, if we included a toggle that allows the client to tell the library to throw and refuse to proceed if an attempt is made to access requester-pays data, so that users who don't want to incur GCS access charges can get a hard guarantee that they won't.

[lbergelson]

@jean-philippe-martin has added support for requester pays to gcloud.
See googleapis/google-cloud-java#3406

I've set up a new fork of the project at https://github.com/broadinstitute/google-cloud-java. I have a branch https://github.com/broadinstitute/google-cloud-java/tree/lb_update_pom_to_publish_to_orgbroad which I believe should make the changes necessary to run on dataproc and avoid googleapis/google-cloud-java#2453.

However, if you rollback the dependencies the project no longer compiles. You can compile the nio-subproject, but the parent project can no longer build against the old dependencies. That makes me very nervous because it seems likely that we will encounter runtime errors if we substitute them.

JP created a small test case to reproduce the error and it seems like the dataproc team is looking at it. Hopefully they can resolve the issue and we can switch to the base library instead of needing to publish an additional sketchy version of it.

[droazen]

This was done -- closing.