Pinning pynwb version

[t-b]

@njmei
The original idea was that ipfx and MIES use the same nwb schema version.
As there is currently no support for multiple schema versions in one NWB file. I also doubt that this can be ever done properly.

From the requirements.txt of pynwb

ipfx/requirements.txt

Line 13 in 8c9d00d

pynwb>=1.3.2,<2.0.0

and dandi-cli

https://github.com/dandi/dandi-cli/blob/39dc4f55caec0c92cdfed24db1bc604f988fc0e7/setup.cfg

one can work with older pynwb versions as well.

MIES supports NWB schema version 2.2.0, see https://github.com/AllenInstitute/IPNWB/blob/main/index.rst.

So in essence I think using the corresponding old pynwb version in ipfx, something like 1.2.1, would reduce the fragility introduced with #542.

[njmei]

@t-b Thanks for the comment! Since IPFX is intended to be used by end-users (at least based on my understanding), pinning the pynwb version to something old like 1.2.1 is probably not ideal since it will force end-users to create a separate python environment just for IPFX and will limit the ability for others to import IPFX modules/functionality into their own custom analyses...

[tmchartrand]

@t-b I also agree with @njmei that pinning to old versions isn't going to be a workable solution for IPFX. In fact, we're likely going to have some need to jump to using newer pynwb version at some point to use some metadata fields in newer versions of the schema.
This would be much less challenging if NWB/HDMF stuck to a policy of breaking changes on major versions only - this issue started because of a minor but breaking schema change that snuck through in hdmf (I think the field just didn't have a type specified before).
I think it would still be reasonable to assume that won't happen often, so we'll need to update MIES to match this schema change, but in general can probably let it lag behind and swap a file's cached schema for a newer version when IPFX modifies it. If we add in a validation call before saving it, then we can flag any issues early.

[t-b]

Thanks for your replies.

Yes (obviously) it is not nice to have to stick to an old version of pynwb.

In fact, we're likely going to have some need to jump to using newer pynwb version at some point to use some metadata fields in newer versions of the schema.

If we do that, we should coordinate that between MIES and ipfx so that both rely on a next fixed good version for a while. I've created AllenInstitute/IPNWB#37 to track that.

If we add in a validation call before saving it, then we can flag any issues early.

Yes, that should definitly be done and help in finding issues.

[tmchartrand]

@njmei just thinking maybe we should reopen this as a reminder of the need for further MIES/IPFX coordination, linked to Thomas's issue on the IPNWB end.

[t-b]

Yesterday I've chatted with @timjarsky about requiring a new fixed pynwb version.

TL;DR: We propose to move https://github.com/AllenInstitute/IPNWB and ipfx to the next patch release of pynwb, e.g. 3.1.2 and have a hard dependency on that version (pynwb==3.1.2 in requirements.txt).

Longer version:
Historically ipfx, as the data-downstream of IPNWB, always used a fixed pynwb version range. IPNWB itself still uses nwb-schema 2.2.4. Having a wide variety of potential pynwb versions reading IPNWB created and ipfx enhanced files will be a huge maintenance burden in the future, where we have little manpower now taking care of ipfx. By adopting a newer nwb-schema version in IPNWB and then requiring the exact latest pynwb version with all our fixes, we should be stable for the next ~5 years.

That newer ipfx version would then support python 3.10 to 3.14, 3.14 is EOL in Oct 2030. python 3.9 will be EOL in Oct 2025.

[t-b]

In the Allen Institute Ephys working group meeting last tuesday I presented the idea of requiring pynwb 3.1.2 for ipfx and switch MIES/IPNWB to the corresponding nwb-schema version (2.9.0).

Nobody objected, so we are moving forward.

• MIES PR: Upgrade IPNWB to nwb-schema 2.9.0 MIES#2477
• IPNWB PR: Upgrade NWB schema to 2.9.0 IPNWB#50