Kedro compatibility with fsspec

[selalamiTF]

Description

Kedro cannot be installed with DVC because of a dependency conflict over fsspec. I need to install Kedro and DVC on the same environment as they are required by the same app.

Context

I can't find a way today to install these two tools together which is quite inconvenient since I find Data Science projects might benefit a lot from them (similar to what Kedro did with Mlflow).

Possible Implementation

Upgrade fsspec or support a lager version.

Possible Alternatives

I can't find any today. Please feel free to suggest any alternatives

[datajoely]

Hi @selalamiTF fsspec has recently switched from the SemVer to CalVer system - we're aware of the problem and looking into how to resolve it.

[selalamiTF]

Hello @datajoely Thank you for you quick reply ! That's great news ! Do you plan to release the fix in the next release ? Just to get an idea on when to expect this as I hit a dead end in my project because of this problem. Thanks

[ignacioparicio]

Hi @selalamiTF, while we work on this, is downgrading DVC an option for you? DVC upgraded their fsspec requirement out of current Kedro bounds on April 9, so any version released before that (e.g. 2.0.17, released on April 2) should work.

[selalamiTF]

Hello @ignacioparicio. Thanks for the tip but I tried 2.0.17 but I still have the same problem (with Kedro 0.16.6) :/

[ignacioparicio]

Unfortunately back in 0.16.6 Kedro's requirement for fsspec was fsspec>=0.5.1, <0.8 (see this), you would also have to bump your Kedro version :/

DVC==2.0.17 and kedro>=0.17 should work

[selalamiTF]

Great ! I guess I can do that since we are still in a prototyping stage (luckily) and trying to figure out how all this can work together ;). Thank you very much for the help, it's great that you guys are available to support your community !

[datajoely]

Tangentially related - the team at Dolt (another data versioning solution) have recently built a Kedro plugin, perhaps this is something that would be helpful?

https://www.dolthub.com/blog/2021-06-16-kedro-dolt-plugin/

[selalamiTF]

Thanks, I will definitely take a look at this !

[ignacioparicio]

We have upgraded our fsspec requirement from fsspec>=0.5.1, <0.9 to fsspec>=2021.04, <2022.01 (the upper bound to be revised past that date). The change will come into effect with the next Kedro release, be it 0.17.5 or 0.18.0.

I'm closing this issue for now, but please shout if you want it reopen.