MemoryError with float64

[ZYFno996]

Describe your issue

I'm running w2w on my Hyper-V machine with 64GB memory/64 cores. I used a 31MB LCZ.tif slice (shape 44528×27830) and a 5.3MB geo_em.d01.nc (100×100). But It seems that the memory capacity is still insufficient. Python traceback is as follows, which says that w2w uses ndarray type float64 for calculation.
I noticed that someone has already posted this issue before and it is said there is not other methods but reduce the region. But I would like to suggest some small but (maybe) efficient optimizations:

1. Use float16 or float32 type instead of float64. Because both landuse type code and lcz.tif (8-bit depth data) are integers but not float decimals. Using float64 causes about 1~3 times memory wastes.
2. Use parallel processes or multiple tiles for calculation.

domain size:

w2w --version

w2w 0.6.0

nc-config --all

None

Installed Packages

as requirements

Traceback

Traceback (most recent call last):
  File "/home/core/InstalledSoftware/miniconda/envs/WRF/bin/w2w", line 8, in <module>
    sys.exit(main())
  File "/home/core/InstalledSoftware/miniconda/envs/WRF/lib/python3.8/site-packages/w2w/w2w.py", line 185, in main
    nbui_max = create_lcz_params_file(
  File "/home/core/InstalledSoftware/miniconda/envs/WRF/lib/python3.8/site-packages/w2w/w2w.py", line 1289, in create_lcz_params_file
    ucp_res = _hgt_resampler(
  File "/home/core/InstalledSoftware/miniconda/envs/WRF/lib/python3.8/site-packages/w2w/w2w.py", line 820, in _hgt_resampler
    ucp_2_wrf_nom = reproject(
  File "/home/core/InstalledSoftware/miniconda/envs/WRF/lib/python3.8/site-packages/rasterio/env.py", line 401, in wrapper
    return f(*args, **kwds)
  File "/home/core/InstalledSoftware/miniconda/envs/WRF/lib/python3.8/site-packages/rasterio/warp.py", line 344, in reproject
    _reproject(
  File "rasterio/_warp.pyx", line 404, in rasterio._warp._reproject
  File "rasterio/_warp.pyx", line 377, in rasterio._warp._reproject
  File "rasterio/_io.pyx", line 2120, in rasterio._io.MemoryDataset.__init__
numpy.core._exceptions.MemoryError: Unable to allocate 9.23 GiB for an array with shape (1, 27830, 44528) and data type float64

[jkittner]

Sounds good to me and makes sense. I think float64 is the default, since pyhton's builtin also has 64Bits.

I am, however, still wondering why it can't allocate 10G of RAM if your machine has 64G.

Do you want to look into that and send a PR with the necessary changes?

[ZYFno996]

Thanks for your cosideration. The traceback means that there is NO MORE 9.23 GiB memory. In fact, w2w has already used about 50 GB RAM. Then, I'm glad to take an inside view of source code and make it better.