[KED-2315] Versioning datasets fails with NotADirectoryError

[clstaudt]

Description

Adding the versioned: true attribute to a dataset results in pipeline failure.

Context

The dataset versioning feature is not usable.

Steps to Reproduce

add versioned: true to a dataset in the catalog

Expected Result

A directory named like the dataset should have been created, containing versions of the dataset.

Actual Result

fails with NotADirectoryError

2020-11-28 18:48:01,270 - kedro.io.data_catalog - INFO - Saving data to `report_mass` (ExcelDataSet)...
2020-11-28 18:48:30,671 - kedro.runner.sequential_runner - WARNING - There are 1 nodes that have not run.
You can resume the pipeline run by adding the following argument to your previous command:
  --from-nodes "calculate_mass_sold"
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/kedro/io/core.py", line 240, in save
    self._save(data)
  File "/opt/anaconda3/lib/python3.7/site-packages/kedro/extras/datasets/pandas/excel_dataset.py", line 177, in _save
    with self._fs.open(save_path, **self._fs_open_args_save) as fs_file:
  File "/opt/anaconda3/lib/python3.7/site-packages/fsspec/spec.py", line 771, in open
    **kwargs
  File "/opt/anaconda3/lib/python3.7/site-packages/fsspec/implementations/local.py", line 116, in _open
    self.makedirs(self._parent(path), exist_ok=True)
  File "/opt/anaconda3/lib/python3.7/site-packages/fsspec/implementations/local.py", line 47, in makedirs
    os.makedirs(path, exist_ok=exist_ok)
  File "/opt/anaconda3/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
NotADirectoryError: [Errno 20] Not a directory: '/ProjectDirectory/data/08_reporting/report_mass.xlsx/2020-11-28T17.37.57.113Z'

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version: 0.16.2
• Python version used (python -V): Python 3.7.8
• Operating system and version: macOS 10.15.7

[clstaudt]

The cause seems to be that previously the dataset was unversioned and a file already existed. After clearing the output folders manually it works.

I still leave the ticket, because being able to switch a dataset from versioned to unversioned and back by just editing the catalog file could be something Kedro wants to support.

[deepyaman]

I still leave the ticket, because being able to switch a dataset from versioned to unversioned and back by just editing the catalog file could be something Kedro wants to support.

Given the way versioning is implemented, this isn't possible (in a straightforward, intuitive manner, at least). That being said, this is a common issue users face, and it could be nice to check for an unversioned dataset and return a more meaningful error message in that case.

[yetudada]

@deepyaman Is right on this one. We do recognise that this is a poor user experience though. We'll look into it. I'll create a ticket for our backlog to try revisit this one again.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.