gcs.makedir raises exception: TypeError: _mkdir() got an unexpected keyword argument 'create_parents'

[JackKelly]

Hi! Thanks again for all your work on gcsfs; it's a great bit of software! I think I've bumped into a small bug...

Minimal Complete Verifiable Example:

import gcsfs

gcs = gcsfs.GCSFileSystem()
gcs.makedir('gs://<bucket>/<path>')

What happened:

In [48]: gcs.makedir(os.path.join(PATH, 'v1'))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-48-63d29a261feb> in <module>
----> 1 gcs.makedir(os.path.join(PATH, 'v1'))

~/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/fsspec/spec.py in makedir(self, path, create_parents, **kwargs)
   1151     def makedir(self, path, create_parents=True, **kwargs):
   1152         """Alias of :ref:`FilesystemSpec.mkdir`."""
-> 1153         return self.mkdir(path, create_parents=create_parents, **kwargs)
   1154 
   1155     def mkdirs(self, path, exist_ok=False):

~/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/fsspec/asyn.py in wrapper(*args, **kwargs)
     86     def wrapper(*args, **kwargs):
     87         self = obj or args[0]
---> 88         return sync(self.loop, func, *args, **kwargs)
     89 
     90     return wrapper

~/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/fsspec/asyn.py in sync(loop, func, timeout, *args, **kwargs)
     51     except RuntimeError:
     52         pass
---> 53     coro = func(*args, **kwargs)
     54     result = [None]
     55     event = threading.Event()

TypeError: _mkdir() got an unexpected keyword argument 'create_parents'

Environment:

• GCSFS version: 2021.07.0
• Python version: 3.9
• Operating System: Ubuntu Linux 21.04 (running on a VM in Google Cloud)
• Install method: conda

[martindurant]

Sorry for the slow reply here.

This appears to be a bad application of an alias. If you called "mkdir" instead of "makedir", it would have worked. mkdir in gcsfs does not support creating parents automatically, and this should probably be fixed. Indeed, GCS does not support directories really, so you should not expect mkdir('gs:///') to actually have an effect.

[martindurant]

Added label good first issue, both for:

• the alias in fsspec.AbstractFileSystem , which should just pass on kwargs
• allow mdir in gcsfs to accept create_parents, which would create the bucket if it doesn't already exist.

[mwbyeon]

To solve this problem, I used the following class:

import gcsfs
from fsspec.asyn import sync_wrapper


class GCSFileSystem(gcsfs.GCSFileSystem):
    # https://github.com/fsspec/gcsfs/issues/404
    def _mkdir(self, bucket, acl="projectPrivate", default_acl="bucketOwnerFullControl", **kwargs):
        return super()._mkdir(bucket, acl, default_acl)
    mkdir = sync_wrapper(_mkdir)

[sidphbot]

actually mkdir and rmdir or their respective async versions do not work for me, the command passes without any action as being subdirectories they are bound to contain "/" and it is forbidden in both the functions

bucket: str
            bucket name. If contains '/' (i.e., looks like subdir), will have no effect because GCS doesn't have real directories.

I think this is handled in a different way in gsutils by creating a fake hierarchy

[martindurant]

Note that GCS does not have any directories below buckets. The online console and gsutils emulate buckets by using zero-length files, but they are not really directories. On the other hand, you can create any key without first making directories, and the intervening implied directories will be implicitly inferred.

[martindurant]

@mwbyeon , did you intend to submit a fix?