Creating Directories in a fuse mount from Jupyter container

[kaipak]

Touching files works fine and indeed, files I copy/create in the terminal show up in the console. However, if I try to create directories, I get the following:

[jovyan@jupyter-kaipak /gcs/storage-benchmarks
 :) ]$ mkdir foo
mkdir: cannot create directory ‘foo’: Bad address
[jovyan@jupyter-kaipak /gcs/storage-benchmarks
 :( ]$

Stranger still are the timestamps:

 :) ]$ ls -la
total 0
drwxrwxrwx 0 jovyan 1000 0 Jan  1  1970 .
drwxrwxrwx 0 jovyan 1000 0 Jan  1  1970 ..
drwxrwxrwx 0 jovyan 1000 0 Jan  1  1970 test_fuse
drwxrwxrwx 0 jovyan 1000 0 Jan  1  1970 test_zarr

Finally, if I delete a directory in the console, it doesn't register in my mounted filesystem until much later, as if it's not updating frequently. I don't get any of these problems from my local laptop. I tried giving my account full ownership access to a different bucket, but this doesn't seem to work either.

[mrocklin]

The FUSE system currently attached to pangeo.pydata.org is based on a pretty experimental system in gcsfs that was made very quickly. I would not expect particularly smooth functioning. If you wanted to raise this issues upstream at https://github.com/dask/gcsfs that would be welcome. PRs moreso :)

…

On Sat, Apr 7, 2018 at 7:33 PM, Kai Pak ***@***.***> wrote: Touching files works fine and indeed, files I copy/create in the terminal show up in the console. However, if I try to create directories, I get the following: ***@***.*** /gcs/storage-benchmarks :) ]$ mkdir foo mkdir: cannot create directory ‘foo’: Bad address ***@***.*** /gcs/storage-benchmarks :( ]$ Stranger still are the timestamps: :) ]$ ls -la total 0 drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 . drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 .. drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 test_fuse drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 test_zarr Finally, if I delete a directory in the console, it doesn't register in my mounted filesystem until much later, as if it's not updating frequently. I don't get any of these problems from my local laptop. I tried giving my account full ownership access to a different bucket, but this doesn't seem to work either. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#202>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszDZiRjyi7c8CiZa0BHZbnk4p_nXJks5tmUyvgaJpZM4TLTKb> .

[mrocklin]

Relevant code is here: https://github.com/dask/gcsfs/blob/master/gcsfs/gcsfuse.py On Sat, Apr 7, 2018 at 7:53 PM, Matthew Rocklin <mrocklin@anaconda.com> wrote:

…

The FUSE system currently attached to pangeo.pydata.org is based on a pretty experimental system in gcsfs that was made very quickly. I would not expect particularly smooth functioning. If you wanted to raise this issues upstream at https://github.com/dask/gcsfs that would be welcome. PRs moreso :) On Sat, Apr 7, 2018 at 7:33 PM, Kai Pak ***@***.***> wrote: > Touching files works fine and indeed, files I copy/create in the terminal > show up in the console. However, if I try to create directories, I get the > following: > > ***@***.*** /gcs/storage-benchmarks > :) ]$ mkdir foo > mkdir: cannot create directory ‘foo’: Bad address > ***@***.*** /gcs/storage-benchmarks > :( ]$ > > Stranger still are the timestamps: > > :) ]$ ls -la > total 0 > drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 . > drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 .. > drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 test_fuse > drwxrwxrwx 0 jovyan 1000 0 Jan 1 1970 test_zarr > > Finally, if I delete a directory in the console, it doesn't register in > my mounted filesystem until much later, as if it's not updating frequently. > I don't get any of these problems from my local laptop. I tried giving my > account full ownership access to a different bucket, but this doesn't seem > to work either. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#202>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/AASszDZiRjyi7c8CiZa0BHZbnk4p_nXJks5tmUyvgaJpZM4TLTKb> > . >

[kaipak]

I take it there isn't a huge issue with installing Google's gcsfuse for now?

I mean, I'd be happy to also take a swag at a PR :).

[jgerardsimcock]

I had similar issues yesterday. I can use gcsfs from my local machine but on a pangeo-like deployment on k8s, I get this behavior.

fsspec/gcsfs#89

I'm inclined to discuss a path forward on this.

[jgerardsimcock]

@kaipak please see fsspec/gcsfs#96

Did you pass values to project-id and token?

[kaipak]

@jgerardsimcock yes, and this seems to fix the issue on my local Jupyter container, but I still get errors from workers. gcsfs works great with the hack noted by #209

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.