bm_raster

[emilytheok]

I am trying to run the guided exploration of the BlackMarblePy package but I get an error during COLLATING RESULTS which I have not been able to solve. See the error message below and the screenshot of the code I am trying to run - many thanks for your support!

KeyError Traceback (most recent call last)
File ~/BlackmarbleDev/venv_blackmarble/lib/python3.13/site-packages/xarray/backends/file_manager.py:211, in CachingFileManager._acquire_with_cache_info(self, needs_lock)
210 try:
--> 211 file = self._cache[self._key]
212 except KeyError:

File ~/BlackmarbleDev/venv_blackmarble/lib/python3.13/site-packages/xarray/backends/lru_cache.py:56, in LRUCache.getitem(self, key)
55 with self._lock:
---> 56 value = self._cache[key]
57 self._cache.move_to_end(key)

KeyError: [<function open at 0x11e6e1c60>, ('/vsimem/23d9c2da-ba8c-4ffe-8dd4-166e82686b8b/23d9c2da-ba8c-4ffe-8dd4-166e82686b8b.tif',), 'r', (('sharing', False),), '52a1c5b2-f217-4dcf-8aeb-db7e9d848406']

During handling of the above exception, another exception occurred:

CPLE_OpenFailedError Traceback (most recent call last)
File rasterio/_base.pyx:310, in rasterio._base.DatasetBase.init()

File rasterio/_base.pyx:221, in rasterio._base.open_dataset()

File rasterio/_err.pyx:359, in rasterio._err.exc_wrap_pointer()

CPLE_OpenFailedError: No such file or directory
...
359 )

File rasterio/_base.pyx:312, in rasterio._base.DatasetBase.init()

RasterioIOError: No such file or directory

[russss]

I'm also seeing this issue - presumably this is the result of some dependency being updated, but it's tricky to work out what's at fault.

The issue is happening here:

blackmarblepy/src/blackmarble/raster.py

Lines 366 to 369 in ff9c83f

	ds = merge_arrays(da)
	clipped_dataset = ds.rio.clip(
	gdf.geometry.apply(mapping), gdf.crs, drop=True
	)

Rasterio is trying to open the in-memory result of merge_arrays as if it's a file on disk. The value of path is actually _UnparsedPath(path='/vsimem/8ca17fb7-3a3e-4946-90ed-78b8b1c798a8/8ca17fb7-3a3e-4946-90ed-78b8b1c798a8.tif').

[russss]

I managed to make it work by changing the code to clip before merging. Still no idea what went wrong.

--- a/src/blackmarble/raster.py
+++ b/src/blackmarble/raster.py
@@ -360,16 +360,12 @@ def bm_raster(
                             drop_values_by_quality_flag=drop_values_by_quality_flag,
                             output_directory=d,
                         ),
-                    )
+                    ).rio.clip(gdf.geometry.apply(mapping), gdf.crs, drop=True)
                     for f in filenames
                 ]
                 ds = merge_arrays(da)
-                clipped_dataset = ds.rio.clip(
-                    gdf.geometry.apply(mapping), gdf.crs, drop=True
-                )
-                clipped_dataset["time"] = pd.to_datetime(date)
-
-                datasets.append(clipped_dataset.squeeze())
+                ds["time"] = pd.to_datetime(date)
+                datasets.append(ds.squeeze())
             except TypeError:
                 continue

[iled]

I am also having this issue. There are a few open issues, here in github, where users show some variation of error being thrown during the "collating results" step.

I have also tried @russss workaround to clip before merging and still got an error in the "collating" step.

This is even reproducible in the example notebook:

# Daily data: raster for February 5, 2021
VNP46A2_20210205 = bm_raster(
    gdf, product_id="VNP46A2", date_range="2021-02-05", bearer=bearer
)
VNP46A2_20210205

Note that data files are downloaded, but then it fails in the collating step, which, I think, suggests there is no issue with the authentication/token.

@russss are you able to run the example above in the notebook with your suggested modification?