Investigate failing tests on Linux and MacOS when upgrading MeshKernel to 2.0.0

[priscavdsluis]

What is the need for this task.
We are trying to upgrade MeshKernel from 1.0.0 to 2.0.0, see PR #442.
When doing that we notice that the CI is failing on Linux and MacOS:

What is the task?
Investigate why these tests are failing and whether it is on the hydrolib-core or meshkernel side.

Additional context
The stacktrace: stacktraces.txt

[backeb]

Describe bug:
failed to install dfm_tools on Mac OSX

OS
Darwin Bjorns-MacBook-Pro-2.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Dec 19 20:43:09 PST 2022; root:xnu-8020.240.18~2/RELEASE_ARM64_T6000 arm64

conda --version
conda 23.1.0

pip --version
pip 23.0.1 from /Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip (python 3.8)

Minimal example

conda create --name dfm_tools_env -c conda-forge python=3.8
conda activate dfm_tools_env
conda install -c conda-forge git shapely cartopy pyepsg geopandas contextily xarray dask netcdf4 bottleneck xugrid cdsapi pydap
python -m pip install git+https://github.com/Deltares/dfm_tools

Error message on exactuting python -m pip install git+https://github.com/Deltares/dfm_tools

Collecting pydantic<1.11,>=1.10
  Using cached pydantic-1.10.6-cp38-cp38-macosx_11_0_arm64.whl (2.5 MB)
Collecting meshkernel<3.0.0,>=2.0.0
  Using cached meshkernel-2.0.2.tar.gz (364 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Traceback (most recent call last):
        File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 484, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 222, in <module>
        File "<string>", line 70, in get_library_name
      OSError: Unsupported operating system: Darwin
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Additional context
Using terminal in Visual Studio Code

Version: 1.76.2 (Universal)
Commit: ee2b180d582a7f601fa6ecfdad8d9fd269ab1884
Date: 2023-03-14T17:54:09.061Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: 10.2.154.26-electron.0
OS: Darwin arm64 21.6.0
Sandboxed: No

[backeb]

Describe the issue:
failed to install dfm_tools on Linux

conda --version
conda 4.11.0

pip --version
pip 21.3.1 from /opt/conda/lib/python3.9/site-packages/pip (python 3.9)

Minimal example

conda create --name dfm_tools_env -c conda-forge python=3.8
conda activate dfm_tools_env
conda install -c conda-forge git shapely cartopy pyepsg geopandas contextily xarray dask netcdf4 bottleneck xugrid cdsapi pydap
python -m pip install git+https://github.com/Deltares/dfm_tools

Error message on executing python -m pip install git+https://github.com/Deltares/dfm_tools

Building wheels for collected packages: dfm-tools, meshkernel
  Building wheel for dfm-tools (setup.py) ... done
  Created wheel for dfm-tools: filename=dfm_tools-0.10.31-py3-none-any.whl size=70826 sha256=0aa11eeb1e435ae56b697bf96ad9eff595c537bba29fb7bb24f8f6a574624d9f
  Stored in directory: /tmp/pip-ephem-wheel-cache-22p4a3ww/wheels/e7/9f/34/3e224a6d2ec93d9c662c3188c7eda64aea08650ea083ad224d
  Building wheel for meshkernel (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for meshkernel (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-38
      creating build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/factories.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/meshkernel.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/py_structures.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/utils.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/c_structures.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/__init__.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/version.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      copying meshkernel/errors.py -> build/lib.linux-aarch64-cpython-38/meshkernel
      running build_ext
      git clone https://github.com/Deltares/MeshKernel
      Cloning into 'MeshKernel'...
      cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DADD_UNIT_TESTS_PROJECTS=OFF
      error: command 'cmake' failed: No such file or directory
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for meshkernel
Successfully built dfm-tools
Failed to build meshkernel
ERROR: Could not build wheels for meshkernel, which is required to install pyproject.toml-based projects

Additional context
Using terminal in JupyterLab
Version 3.2.6
OS
Linux e1dd4ba4d0ab 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

[lucacarniato]

Meshkernel is available as a compiled library in the wheel packages for both Windows and Linux. To install it, simply run the command 'pip install meshkernel'. There's no need to build Meshkernel from sources for these operating systems.

However, for Mac, there is currently no solution available as we don't have macOS agents. The Meshkernel 1.0.0 wheel package did include a compiled macOS library that was generated by one of our developers on their laptop.

[backeb]

Thanks @lucacarniato

So you're saying I can install dfm_tools by doing:

1. conda create --name dfm_tools_env -c conda-forge python=3.8
2. conda activate dfm_tools_env
3. conda install -c conda-forge git shapely cartopy pyepsg geopandas contextily xarray dask netcdf4 bottleneck xugrid cdsapi pydap
4. pip install meshkernel
5. python -m pip install git+https://github.com/Deltares/dfm_tools

[lucacarniato]

installing meshkernel before dfm_tools could work

[backeb]

Hi @lucacarniato,

When I follow the steps in #444 (comment) on a Mac, I get this error:

(dfm_tools_env) backeb@Bjorns-MacBook-Pro-2:bed-data-preparation>pip install meshkernel                                                                                                                                    
                                                                                                                                                                                                                           
Collecting meshkernel                                                                                                                                                                                                      
  Using cached meshkernel-2.0.2.tar.gz (364 kB)                                                                                                                                                                            
  Installing build dependencies ... done                                                                                                                                                                                   
  Getting requirements to build wheel ... error                                                                                                                                                                            
  error: subprocess-exited-with-error                                                                                                                                                                                      
                                                                                                                                                                                                                           
  × Getting requirements to build wheel did not run successfully.                                                                                                                                                          
  │ exit code: 1                                                                                                                                                                                                           
  ╰─> [18 lines of output]                                                                                                                                                                                                 
      Traceback (most recent call last):                                                                                                                                                                                   
        File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>                                                       
          main()                                                                                                                                                                                                           
        File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main                                                           
          json_out['return_val'] = hook(**hook_input['kwargs'])                                                                                                                                                            
        File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel                                   
          return hook(config_settings)                                                                                                                                                                                     
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel                     
          return self._get_build_requires(config_settings, requirements=['wheel'])                                                                                                                                         
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires                              
          self.run_setup()                                                                                                                                                                                                 
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup                                        
          super(_BuildMetaLegacyBackend,                                                                                                                                                                                   
        File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in run_setup                                        
          exec(code, locals())                                                                                                                                                                                             
        File "<string>", line 222, in <module>                                                                                                                                                                             
        File "<string>", line 70, in get_library_name                                                                                                                                                                      
      OSError: Unsupported operating system: Darwin                                                                                                                                                                        
      [end of output]                                                                                                                                                                                                      
                                                                                                                                                                                                                           
  note: This error originates from a subprocess, and is likely not a problem with pip.                                                                                                                                     
error: subprocess-exited-with-error                                                                                                                                                                                        
                                                                                                                                                                                                                           
× Getting requirements to build wheel did not run successfully.                                                                                                                                                            
│ exit code: 1                                                                                                                                                                                                             
╰─> See above for output.                                                                                                                                                                                                  
                                                                                                                                                                                                                           
note: This error originates from a subprocess, and is likely not a problem with pip.      

Will test on Linux now.

[backeb]

Following this approach on a CentOS 8 VM, I managed to install dfm_tools.

[arthurvd]

Hi @backeb, regarding your original error: "OSError: Unsupported operating system: Darwin", indeed, the latest MeshKernel(Py) does not come in a MacOSX version. I've already listed it in the hydrolib-core release notes from previous month:
https://deltares.github.io/HYDROLIB-core/0.5.2/releasenotes/#compatibility-notes
But maybe @lucacarniato can list it somewhere on the meshkernel pages as well.

[backeb]

Thanks @arthurvd, I'm working on a docker container, so I can use dfm_tools from my Mac.

[backeb]

@arthurvd I've tried some steps to getting it working in an ubuntu docker container here: https://github.com/openearth/bed-data-preparation/issues/1#issuecomment-1545822906 but the install fails on a cmake dependency.

[veenstrajelmer]

Meshkernelpy 2.1.0 was released a few weeks ago, macos/linux support was fixed here. This issue can probably be fixed by updating the dependencies.

[veenstrajelmer]

Will be fixed with:

• GRIDEDIT-798: Support macOS arm64 MeshKernelPy#112
• GRIDEDIT-841-844: Set deployment target and correct platform tag for macOS builds MeshKernelPy#130
• GRIDEDIT-796: Use macOS 11 for x86_64 build MeshKernelPy#131

In the next release of meshkernel

[priscavdsluis]

If macOS support is fixed, we should re-add the macOS in our GitHub ci.yml workflow.

[veenstrajelmer]

Update after adding macos to the github workflow:

Installation succeeds, but there is one failing testcase on macos-latest (14) even though macos-13 testbench is all green:

FAILED tests/dflowfm/test_net.py::test_create_1d_2d_1d2d

This is the plain HYDROLIB-core code, the test fails on assert network1_link1d2d.shape == (21, 2):

import numpy as np
from meshkernel import GeometryList
from hydrolib.core.dflowfm.net.models import Branch, Network

def get_circle_gl(r, detail=100):

    t = np.r_[np.linspace(0, 2 * np.pi, detail), 0]
    polygon = GeometryList(np.cos(t) * r, np.sin(t) * r)
    return polygon

# Define line (spiral)
theta = np.arange(0.1, 20, 0.01)

y = np.sin(theta) * theta
x = np.cos(theta) * theta

dists = np.r_[0.0, np.cumsum(np.hypot(np.diff(x), np.diff(y)))]
dists = dists[np.arange(0, len(dists), 20)]

# Create branch
branch = Branch(geometry=np.stack([x, y], axis=1), branch_offsets=dists)

# Create Mesh1d
network = Network()
network.mesh1d_add_branch(branch, name="branch1")

branch = Branch(geometry=np.array([[-25.0, 0.0], [x[0], y[0]]]))
branch.generate_nodes(mesh1d_edge_length=2.5)
network.mesh1d_add_branch(branch, name="branch2")

# Add Mesh2d
network.mesh2d_create_rectilinear_within_extent(
    extent=(-22, -22, 22, 22), dx=2, dy=2
)
network.mesh2d_clip_mesh(geometrylist=get_circle_gl(22))

network.mesh2d_refine_mesh(polygon=get_circle_gl(11), level=1)
network.mesh2d_refine_mesh(polygon=get_circle_gl(3), level=1)

# Add links
network.link1d2d_from_1d_to_2d(branchids=["branch1"], polygon=get_circle_gl(19))

mesh2d_output = network._mesh2d.get_mesh2d()
assert len(mesh2d_output.face_x) == 152
mesh1d_output = network._mesh1d._get_mesh1d()
assert len(mesh1d_output.node_x) == 110

network1_link1d2d = network._link1d2d.link1d2d
network1_con_m1d = network._link1d2d.meshkernel.contacts_get().mesh1d_indices
network1_con_m2d = network._link1d2d.meshkernel.contacts_get().mesh2d_indices
assert network1_link1d2d.shape == (21, 2)
assert network1_con_m1d.size == 21
assert network1_con_m2d.size == 21

Also, could you please install meshkernel in a clean environment with pip install "meshkernel>=4.1.0" and let me know if this works? Also for HYDROLIB-core, please try both pip install hydrolib-core and pip install git+https://github.com/deltares/hydrolib-core. All in a separate clean environment and let me know if it installs and what versions you get.

[backeb]

The installation fails because of an issue with installing meshkernelpy: Deltares/MeshKernelPy#162

Therefore I cannot run the test in your code yet.

[veenstrajelmer]

After some trial and error I noticed that the macos-12 and macos-13 testbanks do run (both seem to be x86_64), so the failing testcase seems to be macos-14 (arm64):

Also, the installation issues by @backeb above seem to be meshkernel related, since it does not cover macos-13-arm64 since we do not have these runners available. A meshkernelpy issues was created: Deltares/MeshKernelPy#162

Therefore, I suggest to wrap up this issue (which goal is to repair the linux and macos testbenches) by testing on macos-13 now. I have created a follow-up issue to investigate macos 14 arm64 support here: #635