Problem with data access from PP file loaded cubes and numpy 1.16

[valeriupredoi]

I have some issues when loading PP files and accessing the cube data afterwords: in iris 2.2.0 with numpy 1.16:

Traceback (most recent call last):
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/ESMValTool-2.0a1-py3.7.egg/esmvaltool/diag_scripts/autoassess/autoassess_area_base.py", line 403, in <module>
    run_area(config)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/ESMValTool-2.0a1-py3.7.egg/esmvaltool/diag_scripts/autoassess/autoassess_area_base.py", line 376, in run_area
    metrics = metric_function(run_obj)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/ESMValTool-2.0a1-py3.7.egg/esmvaltool/diag_scripts/autoassess/land_surface_snow/snow.py", line 35, in land_swe_top
    swe_clim.data, mask=(np.array(swe_clim.data) == -1e20))
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/iris/cube.py", line 1692, in data
    return self._data_manager.data
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/iris/_data_manager.py", line 216, in data
    result = as_concrete_data(self._lazy_array)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/iris/_lazy_data.py", line 171, in as_concrete_data
    data, = _co_realise_lazy_arrays([data])
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/iris/_lazy_data.py", line 135, in _co_realise_lazy_arrays
    computed_arrays = da.compute(*arrays)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/dask/base.py", line 398, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/dask/threaded.py", line 76, in get
    pack_exception=pack_exception, **kwargs)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/dask/local.py", line 459, in get_async
    raise_exception(exc, tb)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/dask/compatibility.py", line 112, in reraise
    raise exc
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/dask/local.py", line 230, in execute_task
    result = _execute_task(task, data)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/dask/core.py", line 119, in _execute_task
    return func(*args2)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/dask/array/core.py", line 80, in getter
    c = a[b]
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/iris/fileformats/pp.py", line 652, in __getitem__
    self.mdi, self.mask)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_iris2_2/lib/python3.7/site-packages/iris/fileformats/pp.py", line 723, in _data_bytes_to_shaped_array
    data.setflags(write=True)
ValueError: cannot set WRITEABLE flag to True of this array

the offending block is:

        clim_file = os.path.join(run['climfiles_root'],
                                 'SWE_clm_{}.pp'.format(season))
        print(swe_clim.data)

even simpler case, using the PP file directly and just loading the cube from it:

import iris
c = iris.load_cube('/group_workspaces/jasmin4/esmeval/autoassess_specific_files/files/SWE_clm_djf.pp')
print(c.data)

results in the error above; note that the print statement is just for prrof of concept, of course that any operation on the data will not work either.

Copying the cube to a new one and operating on the copy doesnt help either.
The only thing that sets the behavior correct and allows me to access the data and operate on it is switching to numpy 1.15

Any ideas?
Apologies if this was raised elsewhere 😃

[pp-mo]

I have recently encountered this during work on #3255

It is specific to numpy >= 1.16, which has disallowed setting flags.write = True.
I fixed it on the 2.2.x branch in this PR.

That has since been merged back to master, so latest code should solve the problem.
Unfortunately not in a tagged release, yet.

[valeriupredoi]

awesome, cheers muchly! Can you guys pls update the conda and pypi sources so we can grab the latest code as well? Many thanks and have a good weekend 🍺

[pp-mo]

Hi @valeriupredoi
As you said

update the conda and pypi sources

But as I said

Unfortunately not in a tagged release, yet.

So, we can't update package sources until we have completed a new version release.

In the meantime, can you maybe install + use the latest code from the project ? ...

Identifying + fixing installation requirements can be a little awkward.
Hint : conda is your best friend here !
What I usually do is to roughly copy what the Travis CI testing script does, as coded in $INSTALLDIR/.travis.yml :

• First run something like, e.g.
  python $INSTALLDIR/requirements/gen_conda_requirements.py --groups core all >my_reqts.txt
  ( adding "--py2" if for python 2 )
• Then e.g.
  conda create -n iris_test python=3.6 --file my_reqts.txt
• Then
  cd $INSTALLDIR; pip install -e .
  : where "-e ." makes this dir an editable aka developer install -- note the "." !

[valeriupredoi]

@pp-mo much appreciated! The problem is that we (the ESMValTool devs and community) are using iris via the tagged release only (from the conda channel for now, but will switch to pypi: scitools-iris in the near future). We have our own environment recipe and work in production mode and only deal with source installs just in the most unfortunate case, which is definitely not how we release our tool. That's why I asked for a new release from you guys, but totally understand that such a bug would not warrant a new release 😁 In any case, the support for PP files in ESMValTool is minimal (ie close to none, just two out of many diagnostics get affected) and this will affect only a tiny bit of the tool until you guys tag a new release. BTW any indication when you guys will release a new version? Cheers muchly!

[valeriupredoi]

hi @pp-mo my apologies for getting back to this - I'd like to close it so we don't clutter you guys' issues space if you could confirm that iris 2.3.0 (or whatevs you gonna call the next release) will contain the fix (also bonus if you can tell us when to expect the new tagged release 😁 )

[pp-mo]

Hi @valeriupredoi apologies for silence, I was trying to work out the answer (!)

Not so sure about 2.3.0 just yet, but we do have a few things now that we'd like to issue in a bug release, which will be 2.2.1
As per this comment

Still hoping to include #3255 #3263, maybe , plus the already-done #3257 #3247 etc.
Expected in a week or two at most.

Will that help you, or do you need a new minor-version ?

[valeriupredoi]

hi @pp-mo - awesome, cheers for the heads up! Good luck with all those issues' inclusion, we will wait for you guys to release the new tagged version, no rush and no need for minor-version releases just for us 😁 I have just opened another issue (well, two days ago) #3269 - maybe you guys have time to look at it and get a fix in as well for 2.2.1 (sorry, am hoarding your attention while I can 😆 )

[lbdreyer]

@valeriupredoi
Iris v2.2.1 which includes the fix for this numpy 1.16 issue is now available from conda-forge

I'm going to close this issue so that we close of the v2.2.1 milestone.

[valeriupredoi]

awesome, cheers muchly @lbdreyer and @pp-mo 🍺