[Bug]: Cannot add electrodes to existing nwbfile in Zarr format

[alejoe91]

What happened?

We have a pipeline that adds LFP data after an NWB file with an electrodes table is created. The LFP electrodes should be appended to the electrodes table, but this is causing errors because hdmf cannot handle the zarr.Array object (see error in traceback)

Seems to be due to the fact that the hdmf.data_utils.append_data function is called by pynwb, which doesn't handle Zarr arrays.

@oruebel @rly @mavaylon1 could you take a look? Should the hdmf.data_utils.append_data handle the zarr.Array case?

Steps to Reproduce

# create mock NWB file
from pynwb.testing.mock.file import mock_NWBFile
from pynwb.file import Device
from pynwb.ecephys import ElectrodeGroup

nwbfile = mock_NWBFile()

# add device
device = Device(name="my_device", description="my device")
_ = nwbfile.add_device(device)
# add electrode group
eg = ElectrodeGroup(name="tetrode", description="my_tetrode", location="unknown", device=device)
_ = nwbfile.add_electrode_group(eg)

nwbfile.add_electrode_column(name="source", description="1st or 2nd round")

# add a few electrodes
rel_xs = [0., 10., 20., 30.]
rel_ys = [0., 0., 0., 0.]

for x, y in zip(rel_xs, rel_ys):
    for x, y in zip(rel_xs, rel_ys):
    nwbfile.add_electrode(
        rel_x=x, 
        rel_y=y, 
        enforce_unique_id=True, 
        source="first", 
        group=eg, 
        location="unknown"
    )


# save to Zarr (same error in "a" mode)
with NWBZarrIO("test.nwb", mode="w") as io:
    io.write(nwbfile)

# now reload and try to add some more electrodes
io = NWBZarrIO(str(results_folder / "test.nwb"), mode="r")
nwbfile_read = io.read()

rel_xs = [50., 60., 70., 80.]
rel_ys = [0., 0., 0., 0.]

for x, y in zip(rel_xs, rel_ys):
    nwbfile_read.add_electrode(
        rel_x=x,
        rel_y=y,
        enforce_unique_id=True,
        source="second",
        group=eg,
        location="unknown"
    )

Traceback

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[69], line 5
      2 rel_ys = [0., 0., 0., 0.]
      4 for x, y in zip(rel_xs, rel_ys):
----> 5     nwbfile_read.add_electrode(rel_x=x, rel_y=y, enforce_unique_id=True, source="second", group=eg, location="unknown")

File /opt/conda/lib/python3.9/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File /opt/conda/lib/python3.9/site-packages/pynwb/file.py:717, in NWBFile.add_electrode(self, **kwargs)
    714     else:
    715         d.pop(col_name)  # remove args from d if not set
--> 717 self.electrodes.add_row(**d)

File /opt/conda/lib/python3.9/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File /opt/conda/lib/python3.9/site-packages/hdmf/common/table.py:706, in DynamicTable.add_row(self, **kwargs)
    704     if row_id in self.id:
    705         raise ValueError("id %i already in the table" % row_id)
--> 706 self.id.append(row_id)
    708 for colname, colnum in self.__colids.items():
    709     if colname not in data:

File /opt/conda/lib/python3.9/site-packages/hdmf/container.py:876, in Data.append(self, arg)
    874 def append(self, arg):
    875     self._validate_new_data_element(arg)
--> 876     self.__data = append_data(self.__data, arg)

File /opt/conda/lib/python3.9/site-packages/hdmf/data_utils.py:32, in append_data(data, arg)
     30 else:
     31     msg = "Data cannot append to object of type '%s'" % type(data)
---> 32     raise ValueError(msg)

ValueError: Data cannot append to object of type '<class 'zarr.core.Array'>'

Operating System

Linux

Python Executable

Python

Python Version

3.9

Package Versions

pynwb 2.7.0
hdmf 3.13.0
hdmf-zarr 0.7.0

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this bug was not already reported?

[stephprince]

Thanks for the issue and example, I believe this will most likely need to be addressed in HDMF so @mavaylon1 will open up a related issue to work on it there.

[mavaylon1]

hdmf-dev/hdmf#1125