Cloud storage - How to emulate it locally?

[david-gang]

I wanted to know what your recommended way is to emulate google cloud storage locally, if at all.
In stackoverflow there is a question regarding this: https://stackoverflow.com/questions/37542530/is-there-google-cloud-storage-emulator
There are two answers:

• Use the in memory emulator - This is just available with the java library
• Use minio - I am not sure how to specify google cloud to use minio

A third option would to mock classes from google cloud storage but which classes should be mocked?

[chemelnucfin]

Hi David, we are going to use #4840 to keep track of emulator issues as it seems like we would need a broader discussion. Please feel free to follow there.

[selfiebot]

Hi folks, I've had a similar question from the BBC and looking at the github issue (https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) I don't see any updates to this. Have there been any new discussions around providing an emulator for GCS? The BBC comment was similar to the stackoverflow one, but here's their detail: **** We noticed there is an official GCP PubSub emulator, but we couldn’t find an official Cloud Storage emulator. We read this StackOverflow discussion about the topic which advises either the Java in-memory stategy or https://www.minio.io/. Since we’re using Python, it seems the in-memory strategy is not an option, whereas the Python client source code doesn’t seem to implement a similar feature. Since we’re not using AppEngine, it seems the local dev-server isn’t an option as well. What is the recommended way to emulate Cloud Storage locally (or in a container)? **** Cheers, Nabeel

…

On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote: Hi David, we are going to use #4840 to keep track of emulator issues as it seems like we would need a broader discussion. Please feel free to follow there. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[selfiebot]

Hi folks, Just wondering if anyone had any response to this thread? Customer is still pending an answer... Cheers, Nabeel

…

On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad ***@***.***> wrote: Hi folks, I've had a similar question from the BBC and looking at the github issue ( https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) I don't see any updates to this. Have there been any new discussions around providing an emulator for GCS? The BBC comment was similar to the stackoverflow one, but here's their detail: **** We noticed there is an official GCP PubSub emulator, but we couldn’t find an official Cloud Storage emulator. We read this StackOverflow discussion about the topic which advises either the Java in-memory stategy or https://www.minio.io/. Since we’re using Python, it seems the in-memory strategy is not an option, whereas the Python client source code doesn’t seem to implement a similar feature. Since we’re not using AppEngine, it seems the local dev-server isn’t an option as well. What is the recommended way to emulate Cloud Storage locally (or in a container)? **** Cheers, Nabeel On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote: > Hi David, we are going to use #4840 to keep track of emulator issues as it seems like we would need a broader discussion. Please feel free to follow there. > > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub, or mute the thread.

-- Nabeel Saad | Solutions Engineer | nsaad@google.com | +44 (0)7 525611473 This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.

[selfiebot]

+Dave Nettleton <davnet@google.com> is this something you can help with or point in the right direction?

On Wed, Aug 8, 2018 at 10:29 AM Nabeel Saad ***@***.***> wrote: Hi folks, Just wondering if anyone had any response to this thread? Customer is still pending an answer... Cheers, Nabeel On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad ***@***.***> wrote: > Hi folks, > > I've had a similar question from the BBC and looking at the github issue ( > https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) > I don't see any updates to this. > > Have there been any new discussions around providing an emulator for GCS? > > The BBC comment was similar to the stackoverflow one, but here's their > detail: > > > **** > We noticed there is an official GCP PubSub emulator, but we couldn’t find > an official Cloud Storage emulator. We read this StackOverflow discussion > about the topic which advises either the Java in-memory stategy or > https://www.minio.io/. Since we’re using Python, it seems the in-memory > strategy is not an option, whereas the Python client source code doesn’t > seem to implement a similar feature. Since we’re not using AppEngine, it > seems the local dev-server isn’t an option as well. What is the recommended > way to emulate Cloud Storage locally (or in a container)? > **** > > Cheers, > Nabeel > > > On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote: > > Hi David, we are going to use #4840 to keep track of emulator issues as > it seems like we would need a broader discussion. Please feel free to > follow there. > > > > > > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub, or mute the thread. -- Nabeel Saad | Solutions Engineer | ***@***.*** | +44 (0)7 525611473 This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.

-- Nabeel Saad | Solutions Engineer | nsaad@google.com | +44 (0)7 525611473 This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.

[selfiebot]

+Vaibhav who's owns GCS programmability Thanks Dave

…

On Wed, Aug 8, 2018 at 2:34 AM Nabeel Saad ***@***.***> wrote: +Dave Nettleton ***@***.***> is this something you can help with or point in the right direction? On Wed, Aug 8, 2018 at 10:29 AM Nabeel Saad ***@***.***> wrote: > Hi folks, > > Just wondering if anyone had any response to this thread? Customer is > still pending an answer... > > Cheers, > Nabeel > > On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad ***@***.***> wrote: > >> Hi folks, >> >> I've had a similar question from the BBC and looking at the github issue >> (https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) >> I don't see any updates to this. >> >> Have there been any new discussions around providing an emulator for GCS? >> >> The BBC comment was similar to the stackoverflow one, but here's their >> detail: >> >> >> **** >> We noticed there is an official GCP PubSub emulator, but we couldn’t >> find an official Cloud Storage emulator. We read this StackOverflow >> discussion about the topic which advises either the Java in-memory stategy >> or https://www.minio.io/. Since we’re using Python, it seems the >> in-memory strategy is not an option, whereas the Python client source code >> doesn’t seem to implement a similar feature. Since we’re not using >> AppEngine, it seems the local dev-server isn’t an option as well. What is >> the recommended way to emulate Cloud Storage locally (or in a container)? >> **** >> >> Cheers, >> Nabeel >> >> >> On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote: >> > Hi David, we are going to use #4840 to keep track of emulator issues >> as it seems like we would need a broader discussion. Please feel free to >> follow there. >> > >> > >> > — >> > You are receiving this because you are subscribed to this thread. >> > Reply to this email directly, view it on GitHub, or mute the thread. > > -- > > > Nabeel Saad | Solutions Engineer | ***@***.*** | +44 (0)7 > 525611473 > > This email may be confidential and privileged. If you received this > communication by mistake, please don't forward it to anyone else, please > erase all copies and attachments, and please let me know that it has gone > to the wrong person. > > The above terms reflect a potential business arrangement, are provided > solely as a basis for further discussion, and are not intended to be and do > not constitute a legally binding obligation. No legally binding obligations > will be created, implied, or inferred until an agreement in final form is > executed in writing by all parties involved. > -- Nabeel Saad | Solutions Engineer | ***@***.*** | +44 (0)7 525611473 This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.

[selfiebot]

Hi Vaibhav, Any response on status of GCS emulator solutions? More detail in the below: I've had a similar question from the BBC and looking at the github issue ( https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) I don't see any updates to this. Have there been any new discussions around providing an emulator for GCS? The BBC comment was similar to the stackoverflow one, but here's their detail: **** We noticed there is an official GCP PubSub emulator, but we couldn’t find an official Cloud Storage emulator. We read this StackOverflow discussion about the topic which advises either the Java in-memory stategy or https://www.minio.io/. Since we’re using Python, it seems the in-memory strategy is not an option, whereas the Python client source code doesn’t seem to implement a similar feature. Since we’re not using AppEngine, it seems the local dev-server isn’t an option as well. What is the recommended way to emulate Cloud Storage locally (or in a container)? **** Cheers, Nabeel

On Wed, Aug 8, 2018 at 10:41 AM Dave Nettleton ***@***.***> wrote: +Vaibhav who's owns GCS programmability Thanks Dave On Wed, Aug 8, 2018 at 2:34 AM Nabeel Saad ***@***.***> wrote: > +Dave Nettleton ***@***.***> is this something you can help with > or point in the right direction? > > On Wed, Aug 8, 2018 at 10:29 AM Nabeel Saad ***@***.***> wrote: > >> Hi folks, >> >> Just wondering if anyone had any response to this thread? Customer is >> still pending an answer... >> >> Cheers, >> Nabeel >> >> On Thu, Jul 5, 2018 at 7:17 PM Nabeel Saad ***@***.***> wrote: >> >>> Hi folks, >>> >>> I've had a similar question from the BBC and looking at the github >>> issue ( >>> https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4840) >>> I don't see any updates to this. >>> >>> Have there been any new discussions around providing an emulator for >>> GCS? >>> >>> The BBC comment was similar to the stackoverflow one, but here's their >>> detail: >>> >>> >>> **** >>> We noticed there is an official GCP PubSub emulator, but we couldn’t >>> find an official Cloud Storage emulator. We read this StackOverflow >>> discussion about the topic which advises either the Java in-memory stategy >>> or https://www.minio.io/. Since we’re using Python, it seems the >>> in-memory strategy is not an option, whereas the Python client source code >>> doesn’t seem to implement a similar feature. Since we’re not using >>> AppEngine, it seems the local dev-server isn’t an option as well. What is >>> the recommended way to emulate Cloud Storage locally (or in a container)? >>> **** >>> >>> Cheers, >>> Nabeel >>> >>> >>> On Tuesday, February 20, 2018 at 8:19:04 PM UTC, chemelnucfin wrote: >>> > Hi David, we are going to use #4840 to keep track of emulator issues >>> as it seems like we would need a broader discussion. Please feel free to >>> follow there. >>> > >>> > >>> > — >>> > You are receiving this because you are subscribed to this thread. >>> > Reply to this email directly, view it on GitHub, or mute the thread. >> >> -- >> >> >> Nabeel Saad | Solutions Engineer | ***@***.*** | +44 (0)7 >> 525611473 >> >> This email may be confidential and privileged. If you received this >> communication by mistake, please don't forward it to anyone else, please >> erase all copies and attachments, and please let me know that it has gone >> to the wrong person. >> >> The above terms reflect a potential business arrangement, are provided >> solely as a basis for further discussion, and are not intended to be and do >> not constitute a legally binding obligation. No legally binding obligations >> will be created, implied, or inferred until an agreement in final form is >> executed in writing by all parties involved. >> > -- > > > Nabeel Saad | Solutions Engineer | ***@***.*** | +44 (0)7 > 525611473 > > This email may be confidential and privileged. If you received this > communication by mistake, please don't forward it to anyone else, please > erase all copies and attachments, and please let me know that it has gone > to the wrong person. > > The above terms reflect a potential business arrangement, are provided > solely as a basis for further discussion, and are not intended to be and do > not constitute a legally binding obligation. No legally binding obligations > will be created, implied, or inferred until an agreement in final form is > executed in writing by all parties involved. > --

Nabeel Saad | Solutions Engineer | nsaad@google.com | +44 (0)7 525611473 This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.

[vbrvk]

You can try this one https://github.com/fsouza/fake-gcs-server

[eweitz]

Maintainers, how suitable is googleapis/python-storage/tests/perf/benchwrapper.py for locally emulating GCS? Its README seems to suggest that is possible at some level.

Having a relatively Python-native way to emulate Google Cloud Storage would be quite valuable.

[joejean]

For those of you still looking for a solution for this. I just tried https://gitlab.com/potato-oss/google-cloud/gcloud-storage-emulator/ and it seems to be working fine for me :).

[oittaa]

For those of you still looking for a solution for this. I just tried https://gitlab.com/potato-oss/google-cloud/gcloud-storage-emulator/ and it seems to be working fine for me :).

Thanks! Unfortunately it's a dead project or something. I tried to submit pull requests, but never got any comments or anything back. So I forked it and now the version in my repository has these additional features:

• PATCH requests
• object metadata
• checksums
• correct datetime formats

https://github.com/oittaa/gcp-storage-emulator and pre-built Docker images are located at https://hub.docker.com/r/oittaa/gcp-storage-emulator

[leonardbinet]

for those interested in this topic, I found this repo as well which seems well maintained: https://github.com/fullstorydev/emulators

cf https://www.fullstory.com/blog/open-sourcing-our-google-cloud-emulators/