pip install netCDF4 for 1.6.3+ fails with TomlDecodeError using distro-default pip version

[PhilMiller]

It looks like the last couple patch releases after 1.6.2 created a dependency on newer versions of pip that weren't previously required up through 1.6.2. That seems surprising and maybe inappropriate. The upgrade itself isn't burdensome, but the error is wholly inscrutable and doesn't at all steer toward that as a solution. It took me over an hour to guess to try the upgrade based on pip appearing in the stack trace, and a few Google searches didn't surface anything that suggested it.

It seems to have been introduced with #1216. That states an increased dependency on setuptools 61, but that seems not to be strictly necessary, nor make a difference.

Details as follows:

This is all running in a VM/container of Ubuntu 20.04 setup under Rancher Desktop on an M1 Mac Mini.

Initial software versions/environment:

# cat /etc/issue
Ubuntu 20.04.6 LTS

# uname -a
Linux bb8e17fb54db 6.1.30-0-virt #1-Alpine SMP Fri, 26 May 2023 06:53:59 +0000 aarch64 aarch64 aarch64 GNU/Linux

# python -m venv /srv/venvs/2023-07-05-netcdftest/

# source /srv/venvs/2023-07-05-netcdftest/bin/activate

# which python
/srv/venvs/2023-07-05-netcdftest/bin/python

# python --version
Python 3.8.10

# pip --version
pip 20.0.2 from /srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip (python 3.8)

# pip list
Package       Version
------------- -------
pip           20.0.2 
pkg-resources 0.0.0  
setuptools    44.0.0 

# dpkg -l lib{netcdf,hdf5}-dev
||/ Name           Version                 Architecture Description
ii  libhdf5-dev    1.10.4+repack-11ubuntu1 arm64        Hierarchical Data Format 5 (HDF5) - development files - serial version
ii  libnetcdf-dev  1:4.7.3-1               arm64        creation, access, and sharing of scientific data

# gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

Expected behavior:

# pip install netCDF4==1.6.2
Collecting netCDF4==1.6.2
  Using cached netCDF4-1.6.2.tar.gz (777 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Requirement already satisfied: numpy>=1.9 in ./lib/python3.8/site-packages (from netCDF4==1.6.2) (1.24.4)
Requirement already satisfied: cftime in ./lib/python3.8/site-packages (from netCDF4==1.6.2) (1.6.2)
Building wheels for collected packages: netCDF4
  Building wheel for netCDF4 (PEP 517) ... done
  Created wheel for netCDF4: filename=netCDF4-1.6.2-cp38-cp38-linux_aarch64.whl size=2771544 sha256=ebf5705a4b23ba17de6ef84e20056f2754ad87444decab2035f454b5429e062b
  Stored in directory: /root/.cache/pip/wheels/a3/68/36/b5e406e05ece7ea5f55234b786a21e4c86cc7ca9f6c9650390
Successfully built netCDF4
Installing collected packages: netCDF4
Successfully installed netCDF4-1.6.2

Observed behavior:

# pip install netCDF4==1.6.3
Collecting netCDF4==1.6.3
  Using cached netCDF4-1.6.3.tar.gz (777 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/toml-0.10.0-py2.py3-none-any.whl/toml/decoder.py", line 455, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
  File "/usr/share/python-wheels/toml-0.10.0-py2.py3-none-any.whl/toml/decoder.py", line 725, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
  File "/usr/share/python-wheels/toml-0.10.0-py2.py3-none-any.whl/toml/decoder.py", line 772, in load_value
    raise ValueError("Stuff after closed string. WTF?")
ValueError: Stuff after closed string. WTF?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 357, in run
    resolver.resolve(requirement_set)
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 33, in prepare_distribution_metadata
    self.req.load_pyproject_toml()
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 512, in load_pyproject_toml
    pyproject_toml_data = load_pyproject_toml(
  File "/srv/venvs/2023-07-05-netcdftest/lib/python3.8/site-packages/pip/_internal/pyproject.py", line 75, in load_pyproject_toml
    pp_toml = toml.load(f)
  File "/usr/share/python-wheels/toml-0.10.0-py2.py3-none-any.whl/toml/decoder.py", line 134, in load
    return loads(f.read(), _dict, decoder)
  File "/usr/share/python-wheels/toml-0.10.0-py2.py3-none-any.whl/toml/decoder.py", line 458, in loads
    raise TomlDecodeError(str(err), original, pos)
toml.decoder.TomlDecodeError: Stuff after closed string. WTF? (line 19 column 1 char 729)

The exact same occurs for pip install netCDF4==1.6.4 (exact same line numbers in the stack trace)

If I upgrade pip, it succeeds:

# pip install --upgrade pip
Collecting pip
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-23.1.2

# pip install netCDF4==1.6.3
Collecting netCDF4==1.6.3
  Using cached netCDF4-1.6.3.tar.gz (777 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cftime in ./lib/python3.8/site-packages (from netCDF4==1.6.3) (1.6.2)
Requirement already satisfied: numpy in ./lib/python3.8/site-packages (from netCDF4==1.6.3) (1.24.4)
Building wheels for collected packages: netCDF4
  Building wheel for netCDF4 (pyproject.toml) ... done
  Created wheel for netCDF4: filename=netCDF4-1.6.3-cp38-cp38-linux_aarch64.whl size=2793162 sha256=6d1ec945a35417a401e23e6128933c98c7853fdb6eda2fe1754934ff73f528c1
  Stored in directory: /root/.cache/pip/wheels/ed/80/e0/fe93c4877211288a8aefe2a8fbf8ac565f6fd9b0f3ae00d9ba
Successfully built netCDF4
Installing collected packages: netCDF4
Successfully installed netCDF4-1.6.3

[jswhit]

@mwtoews do you have any insight into what could be causing this?

[jswhit]

Google tells me that this error most frequently occurs when there is a typo in the pipfile.

[mwtoews]

I have a few ideas to try, when I get a chance. Hopefully later today...

[mwtoews]

Apologies for the issue @PhilMiller , I can replicate this issue on a local Ubuntu 20.04 Docker VM. I have a fix with #1266 that re-arranges pyproject.toml to be compatible with the older TOML parser that was available at the time.

[PhilMiller]

Thank you for the very quick investigation and fix! I'll look forward to a 1.6.5 release.