Unexpected results when trying to overwrite using .to_zarr with the same store name

[halestrm]

I am trying to read a zarr store into an xarray dataset, make a few modifications, and then save the resulting dataset back to the same-named zarr store by overwriting it. However, my data values change in the process of overwriting the zarr store. I have been able to reproduce this behavior without even changing any of the dataset values before trying to overwrite the store.

import xarray as xr

# Create very simple dataset
test_da = xr.DataArray([float(1), float(2)])
test_ds = test_da.to_dataset(name='test_data')
test_ds = test_ds.astype('int8')
test_ds

xarray.Dataset

    Dimensions:
        dim_0: 2
    Coordinates: (0)
    Data variables:
        test_data
        (dim_0)
        int8
        1 2
    Indexes: (0)
    Attributes: (0)

# Save created dataset as a zarr store
test_ds.to_zarr('test.zarr', mode='w')

# Open the zarr store and retrieve the data
test_retrieve = xr.open_dataset('test.zarr', engine='zarr')

# At this point, the data values are as expected
test_retrieve.compute()

xarray.Dataset

    Dimensions:
        dim_0: 2
    Coordinates: (0)
    Data variables:
        test_data
        (dim_0)
        int8
        1 2
    Indexes: (0)
    Attributes: (0)

# Save the retrieved data to the same zarr store
test_retrieve.to_zarr('test.zarr', mode='w')

# The values have changed, both in memory and in the zarr store
test_retrieve.compute()

xarray.Dataset

    Dimensions:
        dim_0: 2
    Coordinates: (0)
    Data variables:
        test_data
        (dim_0)
        int8
        48 97
    Indexes: (0)
    Attributes: (0)

Similar behavior occurs when the dtype is float, but then the values are usually (always?) replaced with nan.

An obvious workaround is to save to a differently named zarr store, delete the original, and then move the new store to the original path, but this creates a lot of overhead since I am working with thousands of zarr stores. Also, the fact that the values change to potentially valid results when using integer data types means that the data corruption could go undetected.