Storage: Consider externalising http configuration

[ravindranathakila]

https://cloud.google.com/storage/quotas

self.storage_client: Client = storage.Client.from_service_account_json(config_json_path)
adapter: HTTPAdapter = self.storage_client._connection.http.adapters["https://"]
self.storage_client._connection.http.adapters["https://"] = HTTPAdapter(pool_connections=1000, pool_maxsize=1000, pool_block=adapter._pool_block)

[dhermes]

@ravindranathakila I'm unclear what the action item is for us here?

[ravindranathakila]

Do please consider properly externalising the http configuration so that it can be tweaked for high throughput systems. Especially when using storage.Client.from_service_account_json()

Right now, having to set _connection implies it isn't externalised. Unless the library is able to "auto-detect" the ideal configuration given large-files/low-throughput small-files/high-throughput average-files/average-throughput, I suggest you don't hide this.

[sangramql]

@tseaver please suggest the expected implementation and approach.

[tseaver]

@sangramql There isn't an "obvious" design for this one.

@ravindranathakila I'm not clear on what you are thinking here. The _connection here doesn't hold any configuration which would affect how blob instances are uploaded / downloaded: it wraps an instance of an object supporting the requests.Session interface (normally a google.auth.transport.requests.AuthorizedSession instance).

The parameter which most affects "througput" is chunk_size, which can be passed to a Blob constructor (or set on it afterwards).