Reference-FS's target filesystem must have same value of asynchronous

[robertbridgart]

Using an older version of fsspec from last year (v2024.10.0), I used to run code which could, in a Python Notebook, create a reference file system (which points to a kerchunk file system stored in parquet format which is hosted on a HTTPS server with basic authentication requirements) and then use its map with xarray to create and use a lazy-loaded Dataset object to read the underlying data files. Both the parquet index file system files and the data files (NetCDFs) to which they point, are both hosted on the same HTTPS server with its authorization requirement (a header x-api-key key/value being passed in to authenticate all requests). I created and set the target and remote of this reference file system to https with the header containing the auth password as in the code below:

fs = fsspec.filesystem(
    "reference",
    fo="https://path/to/parquet/index/filesystem/dir",
    target_protocol="https",
    target_options={
        "headers": {
            "x-api-key": API_KEY
        },
        "timeout": 30
    },
    remote_protocol="https",
    remote_options={
        "headers": {
            "x-api-key": API_KEY
        },
        "timeout": 30
    }
)

ds = xr.open_dataset(
    fs.get_mapper(""),
    engine="zarr",
    backend_kwargs=dict(consolidated=False),
    chunks={'time':12, 'latitude':50, 'longitude':50}
)

This used to work fine. I believe that at some point a check was added to fsspec which ensures that all connected file systems are either synchronous or asynchronous and refuses to proceed unless they all match. Fair enough.

My issue is that since the filesystem asynchronous check came in, my original code has stopped working. I cannot figure out how to create a reference file system with both target https and remote https header options at the same time as trying to force either synchronous or asynchronous behavior on the entire system. No matter what I do, or how I try to tell the reference filesystem or the target /remote systems to be sync or async (i.e. matching), it either results in an exception about the mismatch, or an authentication error when attempting the read from the server being raised (presumably because somewhere internally it has either minted a new remote and lost my header option or ignores it?).

Can someone please explain to me how I can get past this issue and or what I'm doing wrong here? Thanks in advance for any insights.

[martindurant]

Have you moved to using zarr 3? That is, essentially, the big change that the async checks were designed to cope with.

[robertbridgart]

I assume you're asking if I am using the zarr 3.x package in my environment? If so, yes I am using 3.1.2.

[martindurant]

Yes, thank you.
In zarr3, the filesystem must be async, which means the child filesystem(s) of referenceFS must be async too AND ideally created in a coroutine running in the same event loop as zarr's backend. That's a complicated thing! There are now a whole load of similar issues around with kerchunk, fsspec and other components in the mix. fsspec/kerchunk#568 I think is the most directly related.

Although fsspec.filesystem will still be useful to you to in order to directly interact with the filesystem (e.g., verify that the right metadata and chunks show up in the right places), but you should generally let zarr set up the filesystems for you at the same time as it sets up its async storage too. So, you can either pass the URL ("reference://") and storage_options with engine="zarr" or use engine="kerchunk". If you don't specify asynchronous=, I expect it should be added.

[robertbridgart]

I've tried both the sync and async approaches and I'm afraid that I can't get anything to work. You stated that with zarr 3 you must use async, but I thought that although it is recommended, that sync was still optional, is this not the case?

Trying to use sync mode and letting zarr setup the file system for me as you suggested still fails because the header data is not passed all the way through and so attempts to read the data fail with:
ReferenceNotReachable: Reference "latitude/0" failed to fetch target https://path/to/data/file/datafile.nc
(which might be related to the issue you mentioned above kerchunk#568)

Sync code attempt:

import xarray as xr

API_KEY = "<MYKEY>"
CHUNKS = {"time": 1, "latitude": 50, "longitude": 50}
url = "https://data/to/parquet/index.parq"

ds = xr.open_dataset(
    "reference://",
    engine="zarr",
    backend_kwargs={
        "storage_options": {
            "fo": url,
            "fo_protocol": "https",
            "fo_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": False},
            "target_protocol": "https",
            "target_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": False},
            "remote_protocol": "https",
            "remote_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": False},
            "asynchronous": False
        },
        "consolidated": False,
    },
    chunks=CHUNKS,
)

print(ds)

Then when I try to use all async:

import asyncio
import xarray as xr

API_KEY = "<MYKEY>"
CHUNKS = {"time": 1, "latitude": 50, "longitude": 50}
url = "https://data/to/parquet/index.parq"

ds = xr.open_dataset(
        "reference://",
        engine="zarr",
        backend_kwargs={
            "storage_options": {
                "fo": url,
                "fo_protocol": "https",
                "fo_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": True},
                "target_protocol": "https",
                "target_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": True},
                "remote_protocol": "https",
                "remote_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": True},
                "asynchronous": True,
            },
            "consolidated": False,
        },
        chunks=CHUNKS,
)

print(ds)

Here I get a loop not running exception:
...
File "/env/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 701, in init
if ref_fs.isfile(fo2):
^^^^^^^^^^^^^^^^^^
File "/env/lib/python3.12/site-packages/fsspec/asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/env/lib/python3.12/site-packages/fsspec/asyn.py", line 76, in sync
raise RuntimeError("Loop is not running")
RuntimeError: Loop is not running

Any thoughts? I was hoping to be able to use sync mode and Dask for the parallel data file reads to avoid async loop type issues because we typically use Dask Clusters for these types of data reads. I feel like the LLM's I've been using to try and crack this are out of ideas and are now stuck in a loop themselves suggesting methods to help me get this working :(

Thanks.

FYI:
fsspec==2025.9.0
kerchunk==0.2.9
xarray==2025.9.0
zarr==3.1.2

[robertbridgart]

I had a quick look through the code and found why I think things were failing for my two cases.

For the sync attempt, an aiohttp ClientSession object had been created by fsspec (using a default loop for all non-async filesystems) which works fine to read the parquet index files, but when the same client session object (via the same HTTPFileSystem object) is used by zarr to read data files using a different event loop, a check inside aiohttp TimerContext detects the loop used to create it was different to the loop used to use perform an operation with it and it raises an exception - which causes the read attempt to fail.

For the async attempt, the call to HTTPFileSystem._cat_file from LazyReferenceMapper.setup (self.fs.cat_file(...)) fails because setup is not an async function/does not call self.fs.catfile in an event loop, and the call to self.fs.cat_file will be an async function b/c self.fs is our async HTTP guy and expects to be called in a loop and awaited (I think - still very much learning asyncio).

[martindurant]

I think I would try it like this - but also I had expected that the "kerchunk" backend is supposed to set things up correctly.

ds = xr.open_dataset(
        "reference://",
        engine="zarr",
        backend_kwargs={
            "storage_options": {
                "fo": url,
                "target_protocol": "https",
                "target_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": False},
                "remote_protocol": "https",
                "remote_options": {"headers": {"x-api-key": API_KEY}, "timeout": 120, "asynchronous": True},
                "asynchronous": True,
            },
            "consolidated": False,
        },
        chunks=CHUNKS,
)

[robertbridgart]

This works perfectly! Thanks a lot for your help, I appreciate it.

[martindurant]

Did you use my specific suggestion above, or the "kerchunk" engine?

Where do you think is the right place to put this information so that others don't need to go through this process?