Support for google cloud storage for nzarr

[mortenwp]

The current implementation of nzarr is limited to S3 amazon storage.

Are there any plan on extending this to also cover google cloud storage.
Some of the same concept and limitations are fairly equal on the two systems:
I.e. reading a part of an object is possible, but you can only write whole objects.

I'm sure azure also could be considered, but that does not cover my current needs :)

[DennisHeimbigner]

We would like to add support for as many cloud storage systems as we can.
But it is relatively low priority at the moment. We just do not have the resources
to do this yet.

[mortenwp]

OK, thanks for the quick reply, then at least I know that nothing is in the pipeline at the moment. I guess I with this post maybe will find out if someone else has the same needs as me. I should also famliarize me with the current S3 implementation and see how easy/difficult it would be to implement the support myself. If most of the concepts/APIs are equivalent it might not be to bad

[DennisHeimbigner]

The issues to be looked at:

1. Does google cloud have a REST api, and if so, how close is it to the S3 REST API.
2. How does the google cloud authorization mechanism compare with the S3 mechanism

[mortenwp]

They have a c++ client API which I suspect wraps REST

https://github.com/googleapis/google-cloud-cpp#quickstart

[mortenwp]

And they actually also support an s3 interface to their storage:

https://cloud.google.com/storage/docs/interoperability

[DennisHeimbigner]

Have you tried using the Amazon library to access the google cloud?

[mortenwp]

I have now :-) But did not get very far.
I have created HMAC keys from google and stored them in .aws/config file
I then tried using
nccopy my_file https://storage.googleapis.com/long_bucket_name/myfile
nccopy my_file https://storage.googleapis.com/long_bucket_name/myfile#mode=nczarr
nccopy my_file https://storage.googleapis.com/long_bucket_name/myfile#mode=nczarr,file
nccopy` my_file https://storage.googleapis.com/long_bucket_name/myfile#mode=nczarr,s3

First one prints error unknown file format
the next two outputs errors about malformed url.
third one seems to be doing something, but fall over with

I have also tried to use file:// based uri for the output (to output local nczarr files), but then I get errors about object not found. I have never tried using nccopy with uri's before so likely I'm doing silly...

[mortenwp]

Moving the HMAC keys to ~/.aws/credentials seems to wonders. Now the dataset is being created in the google bucket. in chunkcs of 3.1 Mb

[mortenwp]

With 3.1 Mb chunks the upload time for 2.1 Gb file is 13 minutes, if I rechunck the file so 133 Mb chunks it is 2.0 minutes
Raw copy with gcloud storage cp or gsutil cp is 65 seconds (at least gcloud storage seems to use parallel upload)
In all cases the files are copied from europe to the us

[mortenwp]

Not yet, but I thought I should leave a comment here in case someone else was on the search for info.
I have a a couple of other alternative routes to deal with the netcdf files to investigate first. But nzarr is still on my list, just not at the top :) The main thing seems to be get the authentication keys from google, and put them the in the .aws/credentials file

[DennisHeimbigner]

I was looking at the build process for google-cloud-cpp and it looks
like it has a very complicated set of precursor libraries.
This going to make it hard for others to use.
Have you successfully installed google-cloud-cpp?

[mortenwp]

I did build v1.42.1 on Centos7 with gcc9.3 (devtoolset-9).
I had to provide an alternative installation for openssl 1.0.2k, and use a newer cmake. I got a build script from a colleague, that had done it with older versions as well. I have not tried any of the ones from 2.0.x series

With the suggested s3 route you only rely on the S3 SDK/CDK

[DennisHeimbigner]

Can you post that build script?

[mortenwp]

I had a look at this in more detail last night, in addition to the build script there is a bunch of CMakeFile files that pulls down the dependencies and build them, and they are somewhat mixed in with our local build system, so will be a bit hard to share. :-/

It is essentially the same dependencies as they explain how to include on several platforms here https://github.com/googleapis/google-cloud-cpp/blob/main/doc/packaging.md (platform specific instructions are at the bottom).

There is a open issue on providing binary packages at googleapis/google-cloud-cpp#10039 but they foresee issues for older platforms.

Also we pull a specific version of the code with
wget https://github.com/googleapis/google-cloud-cpp/archive/refs/tags/v1.42.1.tar.gz -O v1.42.1.tar.gz in case you don't want to juggle with the entire git tree

Hope this is of any help :-)

[mortenwp]

There is a docker image at https://github.com/dockage/google-cloud-cpp (ubuntu based), that provides an image with v1.22 of the google cloud cpp.

I guess the Dockerfile can be updated with a never library version? (and probably also check if the dependencies needs newer version).
Having it in docker image would leave your standard build system untouched. Do you think this will be a way forward for looking at native google-cloud-cpp build.

I'm still a bit uncertain about if you would be able to dedicate any time to this? I know my use case is a bit limited since the s3 solution provides a solution also for gcs access. My request for a native google cloud cpp integration is basically to avoid linking in libraries for several cloud providers, and switching completely to s3 will be challenging to me since i have other libraries that are google cloud cpp based.

[WardF]

This is all great information that we will definitely use as we move towards supporting more architectures. That said, we definitely still have work to do to really nail down the build process, documentation and testing for Amazon S3 support, in order to make sure we have it as airtight as possible. The good news is that once we have that nailed down, we can use it as a roadmap/template for how to add support for other storage, in a way that makes it not just possible to use, but (ideally) easy to use :).

[czender]

Agree that support in NCZarr in Google Cloud would be great. However, there is a (potentially large, or even dominant) segment of the community that will balk at using NCZarr until/unless it interoperates smoothly with record (unlimited) dimensions. Nearly all models I know of make time an unlimited dimension, and none of that data can currently be converted to NCZarr, or natively produced as NCZarr through the netCDF API. I might be missing something, though to me the supporting the record dimension would be the highest priority for NCZarr development.

[mortenwp]

My use cases are a bit of mix. Some of our internal formats uses the unlimited dimensions, and some don't. I think I can change our application code to use fixed number of records for most if not all my use cases. But that is just me.

I still need to do some tests where I'm a bit closer to my google cloud region, so find out how small chunks I can get away with, and still get acceptable performance (still just toying around with the S3 versions on GCS).

I agree with @WardF 's comment that focus to iron our issues with the S3 interface before spreading to use the google toolkit is a good idea. On my wish list is to be able to build with a static version of aws (and static version of netcdf), and that currently requires a bit of fiddling of what libraries has to be specified where. From what I can see I need

./configure --prefix=/usr/local/netcdf-4.9.2 --enable-shared=no --enable-static=yes LDFLAGS=-L/usr/local/lib -L/usr/local/aws-cdk-cpp-static/lib64 -laws-cpp-sdk-s3 -laws-cpp-sdk-core -laws-crt-cpp -laws-c-s3  -laws-c-event-stream -laws-c-auth -laws-c-http -laws-c-compression -laws-checksums -laws-c-mqtt  -laws-c-cal -laws-c-io -laws-c-common -laws-c-sdkutils -ls2n -lcrypto -lstdc++ LIBS=-laws-c-common -lpthread CPPFLAGS=-I/usr/local/include -I/usr/local/aws-cdk-cpp-static/include --enable-netcdf-4 --enable-s3 --disable-libxml2  CC=icc CXX=icpc

I think it would be a good idea the installer would just have to specify where his aws toolkit is along the lines
-with-aws=/usr/local/aws-cdk-cpp and not mess with the LDFLAGS, and then internally the link string could be build up depend on if a static or shared build was requested (assuming static build means static netcdf, static aws). I don't know how often the aws people change their library layout that will make such an approach less attractive...

A side rant: I understand that when I request a pure static build I also disable plugins. In theory it should be possible to call plugins from an otherwise static library, but I also understand that the hdf5 mechanism allow some call back mechanism to the main library from the plugin. But it seems like they have some discussion around this in their Issue 1009

I think it would be useful if the documentation mentioned the few steps that it takes to get the S3 implementation to be used on GCS.

Generate HMAC keys for your google project on console.cloud.google.com.
Insert the HMAC keys into ~/.aws/credentials as

[default]
aws_access_key_id=...
aws_secret_access_key=...

Specify your netcdf object as https://storage.googleapis.com/<bucket_name>/<netcdf_object_name>
Maybe the gs://<bucket_name>/<netcdf_object_name> should be a valid way of specifying gcs paths for the current S3/aws implementation, maybe with an option to specify the endpointurl through an environment variable

[mortenwp]

Just discovered this https://github.com/aws/aws-sdk-cpp/wiki/Improving-S3-Throughput-with-AWS-SDK-for-CPP-v1.9#working-with-the-new-s3-crt-client which should provide parallel uploads of large chuncks with minimal code change

[WardF]

Interesting, taking a look at it now.