Paths as URIs

[TomNicholas]

This PR closes #242 at the data model level - all paths are coerced to absolute URIs (i.e. file:///directory/test.nc or s3://bucket/test.nc) as they go into the Manifest.

As this forbids constructing manifests using relative paths, it requires minor changes to many tests (e.g. test.nc-> /test.nc). It also will require slightly more invasive changes to any tests that involve kerchunk references.

• Closes Forbid relative paths, and use file URI scheme internally? #242
• Tests added
• Tests passing
• Full type hint coverage
• Changes are documented in docs/releases.rst
• New functions/methods are listed in api.rst
• New functionality has documentation

Sub-tasks:

• Validate paths in .rename_paths method automatically
• Add fs_root option internally
• Add virtual_backend_kwargs to all backends (see Add virtual_backend_kwargs argument to open_virtual_dataset #315)
• Deprecation warning for reader_options (rename to fsspec_kwargs?) (also see Add virtual_backend_kwargs argument to open_virtual_dataset #315)
• Publicly expose fs_root for kerchunk and dmrpp readers as an option to virtual_backend_kwargs (requires Add virtual_backend_kwargs argument to open_virtual_dataset #315)
  • kerchunk
  • dmrpp EDIT: turns out I don't think it's necessary, at least not to get tests to pass
• Test using fs_root
  • For kerchunk reader (requires Refactor kerchunk reader tests to call open_virtual_dataset #317)
  • For dmrpp reader (requires Add dmrpp relative path integration test #318)
• Document how to use fs_root for kerchunk and dmrpp readers
• Use fs_root automatically in other readers
  • hdf5
  • hdf
  • netcdf3
  • fits
  • tiff
• Test this actually fixes issue of writing relative paths to icechunk ([Virtual Datasets] Local Storage ValueError: Filenames containing trailing '/#\d+/' are not supported:  earth-mover/icechunk#279 and ValueError when decoding virtual reference: Filenames containing trailing '/#\d+/' are not supported earth-mover/icechunk#402)

[TomNicholas]

What should we do with the paths in kerchunk references? Are they are always meant as absolute? I guess we should assume they are absolute, unless they have ./ or ../ at the start? Would fsspec ever produce relative paths like that?

cc @martindurant

[martindurant]

Are they are always meant as absolute?

They are always meant "as interpreted by the target filesystem". The nature of that filesystem might be implied by the protocol of a path alone, but commonly additional arguments are also required. This means, that relative paths do work if the target happens to be the local filesystem (file://), but I think of the other filesystems, only ssh supports this concept at all. I would not expect this to be meaningful for basically any practical case.

Note that the dir:// filesystem adds prefixes to URLs for any filesystem, if that's useful at all.

[martindurant]

(I am happy to require absolute paths even if it makes some tests slightly more verbose)

[TomNicholas]

Thanks @martindurant !

The nature of that filesystem might be implied by the protocol of a path alone, but commonly additional arguments are also required.

But is the nature of the filesystem explicitly recorded in the kerchunk references format anywhere? Obviously if the prefix is explicit (e.g. s3://) then you know but otherwise?

that relative paths do work

I guess we should assume they are absolute, unless they have ./ or ../ at the start?

Would this approach work then?

(I am happy to require absolute paths even if it makes some tests slightly more verbose)

This might be helpful if the above approach doesn't work.

[martindurant]

is the nature of the filesystem explicitly recorded in the kerchunk references format

No. The original intention was to have these in the "templates", but in practice, the remote_protocol, remote_options and fss arguments to ReferenceFileSystem are used (and often encoded in Intake prescriptions) in cases of ambiguity.

[TomNicholas]

On cloudpathlib, is upath any use?

I think that would have the same problem as cloudpathlib - I don't necessarily want these paths to have any connection to an actual storage layer, I just want to strictly validate and manipulate their forms.

Do you really need a wrapper?

Turns out I didn't need a wrapper, I can get away with just more if...else logic for each of the possible types of paths. It's slightly less neat but it avoids the dependency issues.

[TomNicholas]

This now works!

It now builds on #323 instead of #318, which allowed me test handling of relative paths for dmrpp.

The typing CI is failing but only with errors inside the hdf reader (cc @sharkinsspatial), in lines I didn't touch, which I don't understand, so I'm going to punt on that so I can merge this (because it blocks earth-mover/icechunk#402).