Help needed: How to set up access to a Partitioned Pandas.CSVDataset using SFTP/SSH?

[drdebian]

Hi there,

new Kedro user here. I'm trying to set up a data pipeline where the first step is to retrieve data from a Linux host which is only accessible via SSH. On it, there is a directory full of temporally partitioned CSV files, which I would like to read in via a Kedro pipeline for further analysis.

Now, I've read through the (otherwise great) tutorials and documentation numerous times, but I'm still at a loss on how to set up access to CSV files through SFTP and what the right fsspec-parameters are to put in base/catalog.yml or local/credentials.yml. Can anyone provide a working example, where one would put hostname, username/password and remote directory information properly?

Any help would be greatly appreciated!

[datajoely]

Hi @drdebian we have this on the docs, but only on the 'latest' branch under example 16:

[drdebian]

Thanks a lot for the quick reply, works like a charm!

[drdebian]

At first I didn't think it would work, because I get the wavy red line beneath my credentials, but it looks like that's something not properly specified in the kedro catalog JSON schema.

[datajoely]

Yeah that's a bug in the schema file - it's also not a GCSFilesystem path in this case. Need to look into that one.