NoneType error when attempting to write to netcdf

[matthew-shin]

Sorry, this is more of a question than a bug, and I don't really know how to remove the label!

Working from cf-python 3.11.0 and cfdm 1.9.0.1, when attempting to convert a pp file to a netcdf file, I get a "TypeError: NoneType is not subscriptable" when attempting to write.

Apologies if the error is reported to the wrong repo: I'll move it to cfdm if needed!

The actions I took were as follows:

Create a conda env with python3.9 and cf-python at 3.11.0
f = cf.read(SINGLE_PP_FILE)[0]
cf.write(f, "test2.nc", verbose=-1)

The verbose output is:

Writing to NETCDF4
Writing to NETCDF4
  Writing <CF Field: id%UM_m01s34i082_vn1105(atmosphere_hybrid_height_coordinate(22), latitude(144), longitude(192)) 1>:
    Compression = ''
    Writing <CF DimensionCoordinate: time() days since 2014-1-1 gregorian>
        to netCDF variable: time(('time', ''))

Here is the error traceback:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-61-5f25aa64323d> in <module>
----> 1 cf.write(f,"test2.nc",verbose=-1)

~/.conda/envs/cf-2/lib/python3.9/site-packages/cfdm/decorators.py in verbose_override_wrapper(*args, **kwargs)
    169         # enabling
    170         try:
--> 171             return method_with_verbose_kwarg(*args, **kwargs)
    172         except Exception:
    173             raise

~/.conda/envs/cf-2/lib/python3.9/site-packages/cf/read_write/write.py in write(fields, filename, fmt, mode, overwrite, global_attributes, file_descriptors, external, Conventions, datatype, least_significant_digit, endian, compress, fletcher32, shuffle, reference_datetime, verbose, cfa_options, single, double, variable_attributes, string, warn_valid, group, coordinates, HDF_chunksizes, no_shuffle, unlimited)
    716         # --- End: if
    717
--> 718         netcdf.write(
    719             fields,
    720             filename,

~/.conda/envs/cf-2/lib/python3.9/site-packages/cfdm/decorators.py in verbose_override_wrapper(*args, **kwargs)
    169         # enabling
    170         try:
--> 171             return method_with_verbose_kwarg(*args, **kwargs)
    172         except Exception:
    173             raise

~/.conda/envs/cf-2/lib/python3.9/site-packages/cfdm/read_write/netcdf/netcdfwrite.py in write(self, fields, filename, fmt, mode, overwrite, global_attributes, variable_attributes, file_descriptors, external, Conventions, datatype, least_significant_digit, endian, compress, fletcher32, shuffle, scalar, string, extra_write_vars, verbose, warn_valid, group, coordinates)
   4807                 )
   4808
-> 4809         self._file_io_iteration(
   4810             mode=effective_mode,
   4811             overwrite=overwrite,

~/.conda/envs/cf-2/lib/python3.9/site-packages/cfdm/read_write/netcdf/netcdfwrite.py in _file_io_iteration(self, mode, overwrite, fields, filename, fmt, global_attributes, variable_attributes, file_descriptors, external, Conventions, datatype, least_significant_digit, endian, compress, fletcher32, shuffle, scalar, string, extra_write_vars, warn_valid, group)
   5087         # ------------------------------------------------------------
   5088         for f in fields:
-> 5089             self._write_field_or_domain(f)
   5090
   5091         # ------------------------------------------------------------

~/.conda/envs/cf-2/lib/python3.9/site-packages/cfdm/read_write/netcdf/netcdfwrite.py in _write_field_or_domain(self, f, add_to_seen, allow_data_insert_dimension)
   3723                     owning_coord_key, _ = c[0]
   3724
-> 3725             z_axis = self.implementation.get_construct_data_axes(
   3726                 f, owning_coord_key
   3727             )[0]

TypeError: 'NoneType' object is not subscriptable

I received the same error when I attempted to use cfdm.write directly (though it was using the same cf-python field - I think these are equivalent?)

I am working from a barebones conda env, with effectively just cf-python (and dependencies) as well as ipython installed.

Here are the relevant packages:
Platform: Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-glibc2.17
HDF5 library: 1.10.6
netcdf library: 4.8.0
udunits2 library: /home/users/yms23/.conda/envs/cf-2/lib/libudunits2.so.0
Python: 3.9.7
netCDF4: 1.5.7
cftime: 1.5.1
numpy: 1.21.2
psutil: 5.8.0
scipy: not available
matplotlib: not available
ESMF: not available
cfdm: 1.9.0.1
cfunits: 3.3.4
cfplot: not available
cf: 3.11.0

For additional information, I backtracked through cf-python versions (and required dependencies) until I was able to write to netcdf successfully (but incredibly slowly...). Here is the environment for the successful version:

Platform: Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-centos-7.9.2009-Core
HDF5 library: 1.10.6
netcdf library: 4.8.0
udunits2 library: /home/users/yms23/.conda/envs/cf-python/lib/libudunits2.so.0
python: 3.7.11
netCDF4: 1.5.7
cftime: 1.5.0
numpy: 1.21.1
psutil: 5.8.0
scipy: not available
matplotlib: not available
ESMF: not available
cfdm: 1.8.3
cfunits: 3.3.4
cfplot: not available
cf: 3.4.0

[sadielbartholomew]

Hi Matthew,

First of all, sorry for the delay in getting back to you on your queries - we have been snowed under with deadlines and user support lately.

Sorry, this is more of a question than a bug, and I don't really know how to remove the label!

Thanks very much for providing so much detail in your issue report, that is always really helpful.

It could be a bug! It certainly looks like one when I read through your comment, especially if it worked in a previous version of cf-python. At best, the error message is not informative as to the underlying issue, so that's definitely a bug in my eyes!

I probably won't have time to investigate today, but next week myself or David (who is on leave this week) will get on this and put in a fix, though I am not sure how long it may be until we put out a new release to include the fix. It sounds like you have a workaround of sorts in using an earlier version of the library, though, at least...

able to write to netcdf successfully (but incredibly slowly...)

I should add that we are working in earnest on speeding up the library. We are aware in particular that our handling of PP files is relatively slow and hope to improve this in a major new release that includes use of Dask under-the-hood to parallelise data operations, amongst other performance enhancements. I hope the immediate slowness does not put you off using the library.

So, overall, thanks for your report and one of us will look into it early next week.

[matthew-shin]

Thanks for the reply! No worries on the speed - the reason I upgraded in the first place was because the speed is already better in the newer versions (before I realised this thing with pp files existed...). I'm excited to hear about the dask parallelisation - I was using the default multiprocessing package to do that up till now, and hopefully having dask integration will simplify/speed up the process even more!

Sadly, at this point, I don't really know what combination of package versions with which dependency versions works for this issue - so for now I can't really perform pp file conversion.

[sadielbartholomew]

OK, firstly sorry for getting back to you a bit later than stated.

I have just had a quick look and I think you are probably ultimately running into a simple issue whereby one statement had somehow acquired the wrong level of indentation, which was spotted and rectified a few days ago (by David after he spotted a bug that looked slightly different at first): see NCAS-CMS/cfdm@41b11ce if interested.

Though depending on the nature of the field you are attempting to write, it might not be the issue at hand (possibly masking it, or maybe we've run into another issue first). Let's see though... would you mind confirming whether you can successfully write out the field to netCDF when you use the current master branch of cfdm (and any compatible version of cf-python)? You can check that branch state out via cloning the repository locally and running pip install -e . in the root directory, for example.

Hopefully that sorts it, but if not, it will allow me to see where we are at. If there is still an issue, would you mind doing a f.dump() on your field before the attempted write, and sharing the output here, so I know more about how to reproduce any underlying bug that may remain?

I'm excited to hear about the dask parallelisation - I was using the default multiprocessing package to do that up till now, and hopefully having dask integration will simplify/speed up the process even more!

Glad to hear, and sorry if you've been applying such a workaround for speed. We already have a few "power users" who we know of (at least) who work with very large amounts of pp files, so are keen to speed this up for them and yourself, now. We think it will be around a few months before that release is ready, so watch this space. Thanks!

[davidhassell]

Hi Sadie - yes indeed, this looks like the same problem that was fixed over at NCAS-CMS/cfdm#164, thanks for spotting the match. Looking forward to closing this one (hopefully ...!)

[matthew-shin]

Sounds like a plan! To be sure, I'll set up a new conda env with the latest version of cf-python and the git version of cfdm to replace the conda-installed one. I will let you know how that goes.

[matthew-shin]

Hi All, sorry for taking so long on this!

I can confirm that this particular issue is solved using the 1.9.0.1 version of cfdm. (Installed into conda env via pip directly from git)

Thank you so much for your help (and patience!)

[sadielbartholomew]

Glad to hear, thanks @matthew-shin! And no worries. There will be a new release of cfdm at some point in the not-too-distant future (though no plans for a given date yet) so you won't have to use the workaround installation of master.