TypeError: cannot pickle '_thread.lock' object

[pradeeps31]

Discussed in https://github.com/corteva/rioxarray/discussions/613

^{Originally posted by pradeeps31 December 8, 2022}
Hi Guys,
I am trying to convert a netCDF file to GeoTIFF file using xarrray and rioxarray. But when I give the rio.to_raster command, it throws the TypeError: cannot pickle '_thread.lock' object. The NetCDF file (.nc4) is a precipitation dataset downloaded from NASA Earthdata. The code I have used is below:

>>> import xarray as xr
>>> import rioxarray as rio
>>> ncfile = xr.open_dataset('3B-DAY.MS.MRG.3IMERG.20210901-S000000-E235959.V06.nc4')
>>> pr = ncfile['precipitationCal']
>>> pr = pr.transpose('time', 'lat', 'lon')
>>> pr = pr.rio.set_spatial_dims('lon', 'lat')
>>> pr.rio.crs
>>> pr.rio.crs
>>> pr.rio.set_crs("epsg:4326")
<xarray.DataArray 'precipitationCal' (time: 1, lat: 1800, lon: 3600)>
[6480000 values with dtype=float32]
Coordinates:
  * lon      (lon) float32 -179.9 -179.8 -179.8 -179.6 ... 179.8 179.9 179.9
  * lat      (lat) float32 -89.95 -89.85 -89.75 -89.65 ... 89.75 89.85 89.95
  * time     (time) object 2021-09-01 00:00:00
Attributes:
    units:      mm
    long_name:  Daily accumulated precipitation (combined microwave-IR) estimate
>>> pr.rio.to_raster(r"trial1.tif")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\rioxarray\raster_array.py", line 1145, in to_raster
    return RasterioWriter(raster_path=raster_path).to_raster(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\rioxarray\raster_writer.py", line 232, in to_raster
    xarray_dataarray = xarray_dataarray.copy()
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\dataarray.py", line 1181, in copy
    return self._copy(deep=deep, data=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\dataarray.py", line 1189, in _copy
    variable = self.variable._copy(deep=deep, data=data, memo=memo)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\variable.py", line 1048, in _copy
    ndata = copy.deepcopy(ndata, memo)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_thread.lock' object
>>>

Could anyone help me to resolve this issue. I'm using python 3.11 version.

Thanks

Pradeep

[snowman2]

Mind filling out the issue template:

Environment Information

• python -c "import rioxarray; rioxarray.show_versions()"

• rioxarray version (python -c "import rioxarray; print(rioxarray.__version__)")
• rasterio version (rio --version)
• GDAL version (rio --gdal-version)
• Python version (python -c "import sys; print(sys.version.replace('\n', ' '))")
• Operation System Information (python -c "import platform; print(platform.platform())")

Installation method

• conda, pypi, from source, etc...

Conda environment information (if you installed with conda):

Environment (conda list):

$ conda list | grep -E "rasterio|xarray|gdal"

Details about conda and system ( conda info ):

$ conda info

[pradeeps31]

Environment Information:

rioxarray (0.13.1) deps:
  rasterio: 1.3.4
    xarray: 2022.12.0
      GDAL: 3.6.0
      GEOS: 3.11.1
      PROJ: 9.1.0
 PROJ DATA: C:\Users\Pradeep\anaconda3\envs\imerg_env\Library\share\proj
 GDAL DATA: C:\Users\Pradeep\anaconda3\envs\imerg_env\Library\share\gdal

Other python deps:
     scipy: 1.9.3
    pyproj: 3.4.0

System:
    python: 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:12:32) [MSC v.1929 64 bit (AMD64)]
executable: C:\Users\Pradeep\anaconda3\envs\imerg_env\python.exe
   machine: Windows-10-10.0.19045-SP0

Installation method: conda

Environment (conda list):

libgdal                   3.6.0               hc44c3d3_11    conda-forge
rasterio                  1.3.4           py311hd28ea1e_0    conda-forge
rioxarray                 0.13.1             pyhd8ed1ab_0    conda-forge
xarray                    2022.12.0          pyhd8ed1ab_0    conda-forge

Details about conda and system ( conda info ):

 active environment : imerg_env
  active env location : C:\Users\Pradeep\anaconda3\envs\imerg_env
          shell level : 2
     user config file : C:\Users\Pradeep\.condarc
populated config files : C:\Users\Pradeep\.condarc
        conda version : 22.9.0
  conda-build version : 3.21.8
       python version : 3.9.12.final.0
     virtual packages : __cuda=9.1=0
                        __win=0=0
                        __archspec=1=x86_64
     base environment : C:\Users\Pradeep\anaconda3  (writable)
    conda av data dir : C:\Users\Pradeep\anaconda3\etc\conda
conda av metadata url : None
         channel URLs : https://conda.anaconda.org/conda-forge/win-64
                        https://conda.anaconda.org/conda-forge/noarch
                        https://repo.anaconda.com/pkgs/main/win-64
                        https://repo.anaconda.com/pkgs/main/noarch
                        https://repo.anaconda.com/pkgs/r/win-64
                        https://repo.anaconda.com/pkgs/r/noarch
                        https://repo.anaconda.com/pkgs/msys2/win-64
                        https://repo.anaconda.com/pkgs/msys2/noarch
        package cache : C:\Users\Pradeep\anaconda3\pkgs
                        C:\Users\Pradeep\.conda\pkgs
                        C:\Users\Pradeep\AppData\Local\conda\conda\pkgs
     envs directories : C:\Users\Pradeep\anaconda3\envs
                        C:\Users\Pradeep\.conda\envs
                        C:\Users\Pradeep\AppData\Local\conda\conda\envs
             platform : win-64
           user-agent : conda/22.9.0 requests/2.27.1 CPython/3.9.12 Windows/10 Windows/10.0.19045
        administrator : False
           netrc file : None
         offline mode : False

[snowman2]

This one may be more difficult to track down. I don't have a Windows machine and the CICD runners don't show this issue. If someone else runs into this, more details are welcome.

[pradeeps31]

Hi @snowman2,

I was also unable to find exactly the cause of that error. But, I was able to find a workaround for this issue. I created an environment with the python 3.7.12 version and was able to run the code. But I would like to know what was the source of the error and the solution to the same.

[swesemeyer]

I have the same issue on Fedora Linux 38.
test_prec.nc.zip

Here is the code:

    xds = xarray.open_dataset("/tmp/test_prec.nc")
    precip = xds.precipitationCal.transpose('time', 'lat', 'lon')
    precip.rio.set_spatial_dims(x_dim="lon", y_dim="lat", inplace=True)
    precip.rio.write_crs("EPSG:4326", inplace=True)
    precip.rio.to_raster("/tmp/test_prec.tif") <-- this fails with the same error message

Error:
File "/usr/lib64/python3.10/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object

rioxarray.show_versions()
rioxarray (0.14.1) deps:
rasterio: 1.3.6
xarray: 2023.4.2
GDAL: 3.5.3
GEOS: 3.11.1
PROJ: 9.0.1
PROJ DATA: /home/user/venv_310/lib64/python3.10/site-packages/rasterio/proj_data
GDAL DATA: /home/user/venv_310/lib64/python3.10/site-packages/rasterio/gdal_data
Other python deps:
scipy: 1.10.1
pyproj: 3.4.1
System:
python: 3.10.11 (main, Apr 5 2023, 00:00:00) [GCC 13.0.1 20230404 (Red Hat 13.0.1-0)]
executable: /home/user/venv_310/bin/python
machine: Linux-6.2.15-300.fc38.x86_64-x86_64-with-glibc2.37

[snowman2]

pydata/xarray#8571 was included in v2024.01.0 and says it addresses the issue.