Including ocp wheels in the release

[roipoussiere]

@fpq473 and I worked on building and publishing ocp wheels in the CI these last days, based on a first draft published here: CadQuery/cadquery#1048.

Publishing ocp wheels will really facilitate the project installation (CadQuery/cadquery#153).

We don't have an Azure account so we used Github Actions, but the syntax seems to be identical, so I believe that it should not be too difficult to integrate this in the current CI.

Here is the release published by Github Actions on my fork: https://github.com/roipoussiere/OCP/releases/tag/7.5.3

As you can see it includes wheels for Linux, macOS and Windows, and for Python 3.8, 3.9 and 3.10 (each is tested in the CI).
This allows a user to install cadquery with pip, for now with:

pip install <ocp-vtk wheel link> 'git+https://github.com/CadQuery/cadquery' ezdxf multimethod nlopt nptyping typish

But of course, when this will be integrated in the OCP CI, it will be trivial to update the setup.py / pyproject.toml on the cadquery repo in order to facilitate the installation process.

Relevant links:

• CI run that builds wheels (workflow file);
• CI run that publish release (workflow file).

Let me know if there is something I can do to facilitate the integration of this work on the current CI, and if you think it's relevant to start a PR here.

[whophil]

I am a conda guy, not a wheels guy, so sorry if these are stupid questions:

• Does this bundle/vendor OCCT libraries? I see a single OCP.so file. Or are OCCT shared libs not a runtime dependency of OCP? I see that they are are ocpvtk.lib/
• I see a vtkmodules/ folder, but ldd OCP.*.so, shows that the OCP library only references those found in ocpvtk.lib. This appears to be made possibly via RPATH. What is the purpose of the vtkmodules folder?
• How does this work when the venv also contains vtk installed by wheel? If I create a vtkPolyData using the version of VTK from ocpvtk.libs, can I pass that into filter classes installed by the official vtk wheel?

[fpq473]

I am new to VTK and this bundling approach so hopefully these answers are helpful:

• I initially did not bundle vtkmodules/ but observed the following bad behavior using the official vtk wheel and an ocp-only wheel (one with OCP.so with auditwheel applied):

$ python -c 'import cadquery as cq; print(cq.Workplane().box(1,2,3)._repr_javascript_())'
2022-04-13 17:59:25.929 (   0.372s) [        FCB4D740]vtkDemandDrivenPipeline:666    ERR| vtkCompositeDataPipeline (0x55986dcb7e00): Input port 0 of algorithm vtkTriangleFilter(0x55986d9a1c40) has 0 connections but is not optional.
[... snipped ...]
   <Piece NumberOfPoints=\"0\"  NumberOfVerts=\"0\"  NumberOfLines=\"0\"  NumberOfStrips=\"0\" 
 NumberOfPolys=\"0\" 
[... snipped ...]

There is a cryptic message, and the generated xml (VTKFile PolyData?) contains zero points, verts, lines, etc. This is on Linux; I have not been able to check on MacOS and Windows.

I found no solution except to bundle vtkmodules/ (and its extensions and its dependencies) as well. My wild guess is there are libraries in the vtk wheel that are duplicated in the ocp-only wheel, and those libraries reach inconsistent global state due to different code paths interacting with different copies.

If there is a better solution, please let me know. It would be great not to have to bundle vtkmodules/.

• Pip allows packages to trample over another's installed files, so ocpvtk and vtk can both be installed. I would guess the result will be wildly confusing and lead to bad behavior like the one described above. Unfortunately setuptools does not allow "anti-requirements" to be specified, so we should think about ways to prevent this ocpvtk + vtk clash, or detect when it does happen and warn the user.

[whophil]

@fpq473 thanks for the details.

When you tried this with the official vtk wheel, did you use VTK 9.0.1, or VTK 9.1.0 (the latest wheel)? The version of OCCT you are testing with (7.5.3) would have been built with VTK 9.0.1.

For what it's worth, this is exactly the type of issue that conda (and more specifically the conda-forge ecosystem) seek to alleviate. pip is clearly not built to support binary dependencies, which means that any venv containing ocp plus vtk is "use at your own peril." I am merely reiterating some of the points made in CadQuery/cadquery#153, which I skimmed over.

I do see the value in pip install cadquery - assuming that CadQuery is just being installed for a test drive. But I use ocp in applications which are not CadQuery, and I would never use an ocp wheel because of the binary dependency issues.

[fpq473]

@whophil I tried with VTK 9.1.0 wheel, and I just tried again with VTK 9.0.2 wheels. Both give the same bad behavior described above. There is no VTK 9.0.1 wheel in PyPI. Do you suspect a version mismatch as the cause -- do you know where I can find a 9.0.1 wheel? Happy to dig in further with your guidance.

Point taken about conda.

[whophil]

@fpq473 Do these work? Python 3.6 through 3.8.
https://pypi.org/project/vtk/9.0.1/#files

[fpq473]

@whophil Ah thanks, I was on 3.9. Using 3.8, I made an ocp-only wheel (which still bundles libvtk* under ocp.libs), installed vtk==9.0.1 (which also has libvtk*), and still get this vtkDemandDrivenPipeline error:

$ python -c 'import cadquery as cq; print(cq.Workplane().box(1,2,3)._repr_javascript_())'
2022-04-14 17:41:47.312 (   0.535s) [        6D606740]vtkDemandDrivenPipeline:668    ERR| vtkCompositeDataPipeline (0x55b62faac1f0): Input port 0 of algorithm vtkTriangleFilter(0x55b62f2f87a0) has 0 connections but is not optional.
[... snipped ...]
    <Piece NumberOfPoints=\"0\" NumberOfVerts=\"0\" NumberOfLines=\"0\" NumberOfStrips=\"0\" 
[... snipped ...]

Anything else worth trying?

[fpq473]

@whophil Just wanted to note a seemingly successful attempt at an ocp-only wheel on Linux.

I start with OCP.so in a wheel and apply auditwheel to bundle libraries, except I modify auditwheel not to bundle libvtk*. I then set rpath on OCP.so to $ORIGIN/ocp.libs:$ORIGIN/vtkmodules (adding the second term). Installing the resulting wheel with vtk==9.0.1 seems to work fine.

Things almost work in 3.9 + 9.0.2. OCP.so looks for libraries named libvtkXXX-9.0.so.1 but the bundled libs in vtkmodules/ are named libvtkXXX-9.0.so. If I make a bunch of symlinks in vtkmodules/, then things appear to work.

[adam-urbanczyk]

See: https://github.com/CadQuery/ocp-build-system