BUG? MemoryFileSystem started to raise error when creating existing dir

[jorisvandenbossche]

In the latest release, the MemoryFileSystem started to raise an error in mkdir if the target directory already exists as a directory. I think this changed in #654, and it might also have been a deliberate change. But it is inconsistent how the local filesystem works.

LocalFileSystem: works to create existing directory but fails if it already exists as a file:

from fsspec.implementations.local import LocalFileSystem as FSSpecLocalFileSystem
fs = FSSpecLocalFileSystem()
fs.mkdir("test_dir")
fs.touch("test_file")

# creating an existing dir is fine
>>> fs.mkdir("test_dir")
# creating a dir that already exists as a file errors
>>> fs.mkdir("test_file")
...
FileExistsError: [Errno 17] File exists: '/home/joris/scipy/test_file'

MemoryFileSystem: the latest release started to raise in both cases:

from fsspec.implementations.memory import MemoryFileSystem
memfs = MemoryFileSystem()
memfs.mkdir("/test")
memfs.mkdir("/test/subdir")
memfs.touch("/test/file")

In [17]: memfs.mkdir("/test/subdir")
---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
<ipython-input-17-24458f166c60> in <module>
----> 1 memfs.mkdir("/test/subdir")

~/miniconda3/envs/arrow-dev/lib/python3.8/site-packages/fsspec/implementations/memory.py in mkdir(self, path, create_parents, **kwargs)
     97         path = self._strip_protocol(path)
     98         if path in self.store or path in self.pseudo_dirs:
---> 99             raise FileExistsError
    100         if self._parent(path).strip("/") and self.isfile(self._parent(path)):
    101             raise NotADirectoryError(self._parent(path))

FileExistsError: 

In [18]: memfs.mkdir("/test/file")
---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
<ipython-input-18-783aa803549c> in <module>
----> 1 memfs.mkdir("/test/file")

~/miniconda3/envs/arrow-dev/lib/python3.8/site-packages/fsspec/implementations/memory.py in mkdir(self, path, create_parents, **kwargs)
     97         path = self._strip_protocol(path)
     98         if path in self.store or path in self.pseudo_dirs:
---> 99             raise FileExistsError
    100         if self._parent(path).strip("/") and self.isfile(self._parent(path)):
    101             raise NotADirectoryError(self._parent(path))

FileExistsError: 

I don't know what the specified behaviour should be (since the specification isn't clear about this: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.mkdir).

[martindurant]

Hm, I would say local should error too, since doing os.mkdir() twice with the same path raises FileExistsError.

[jorisvandenbossche]

OK, if that's used as the reference, that's indeed clear.
I personally find the FileExistsError a bit confusing since it's not a file but a directory that exists, but yeah os.mkdir does that as well.