Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for GCS emulators #177

Merged

Conversation

manuteleco
Copy link
Contributor

Allow configuring a Google Cloud Storage emulator to be used instead of the real service. This can be achieved by providing a STORAGE_EMULATOR_HOST environment variable with the scheme, host and port of the emulator server (e.g.: http://localhost:8080).

Besides using a different API endpoint, configuring an emulator also has the effect of lifting the requirement of providing valid authentication credentials. Emulators don't require or even implement authentication.

This behavior is similar to the one implemented in official Google Cloud Storage client libraries developed and maintained by Google.


Closes #176

It also has a weak dependency with MarkEdmondson1234/googleAuthR#228. Without that PR in googleAuthR, using an emulator will still work in googleCloudStorageR but it will unfortunately require valid credentials for the real GCS.


Just like in #175 (comment), I wonder whether I have to build the pkgdown documentation here or not. I also wonder what else should I include in this PR (e.g., an entry in NEWS.md, etc.).

Also, be aware that I'm new to R, so feel free to point out any unidiomatic or plain odd code that needs to be fixed.


Finally, for testing I have used https://github.com/fsouza/fake-gcs-server, but it currently requires a few adjustments for it to work with googleCloudStorageR: fsouza/fake-gcs-server#1258 and fsouza/fake-gcs-server#1252. With some luck, they will be accepted and merged eventually.

I did not test every single GCS operation; it is possible some are still not working with an emulator yet. But I believe this PR already brings value even if it doesn't happen to provide 100% support. Any operation that is still not working could be refined later, once encountered.

@@ -18,7 +18,7 @@ gcs_upload(
upload_type = c("simple", "resumable")
)

gcs_upload_set_limit(upload_limit = 5000000L)
gcs_upload_set_limit(upload_limit = 5e+06)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this annoying. Not sure why roxygen2 decided to make this change.

@MarkEdmondson1234
Copy link
Collaborator

MarkEdmondson1234 commented Jul 26, 2023

Please do add a note to the NEWS.md and feel free to add as a contributor to DESCRIPTION - thank you! Looks like a useful addition. The rebuild of the documentation is usually done before CRAN launch

Allow configuring a Google Cloud Storage emulator to be used instead of
the real service. This can be achieved by providing a
`STORAGE_EMULATOR_HOST` environment variable with the scheme, host and
port of the emulator server (e.g.: http://localhost:8080).

Besides using a different API endpoint, configuring an emulator also has
the effect of lifting the requirement of providing valid authentication
credentials. Emulators don't require or even implement authentication.

This behavior is similar to the one implemented in official Google Cloud
Storage client libraries developed and maintained by Google.
@manuteleco
Copy link
Contributor Author

Just in case you didn't get a notification from my last change, I have adjusted NEWS.md and DESCRIPTION. Let me know if there is anything else I should do 👍 .

@MarkEdmondson1234
Copy link
Collaborator

@manuteleco great thank you! I was on holiday

@MarkEdmondson1234 MarkEdmondson1234 merged commit 6d7e9ad into cloudyr:master Aug 24, 2023
@manuteleco manuteleco deleted the support_storage_emulator branch September 2, 2023 08:49
@MarkEdmondson1234
Copy link
Collaborator

@manuteleco I am getting some bug reports since this was merged, is it possible this is related?

#183

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support using GCS emulators
2 participants