Is it possible to multithread H3 methods in python?

[hsbsid]

I'm running some pretty hefty .polyfill operations on python on 1000s of polygons. Any way I can split the work across cores?

[dfellis]

This would be more appropriately asked on the h3-py repo.

But some notes:

1. I can see by the use of polyfill instead of polygon_to_cells that you're on the v3 API for H3. There has been work in v4 for Python to improve the single-threaded speed of many of the methods by using NumPy or MemoryView arrays. This is still single-threaded performance only, though.
2. The GIL makes naive parallelization impossible, since only one thread can be scheduled to run Python code at a time. It is possible that a polyfill call, since 99% of the time is in the C layer, would work with that naive parallelization, though, but I'm not a CPython expert so I'd only say "it's worth a test." A cursory look through the h3-py repo shows me that this parallelization path is partially implemented for v4 but not implemented at all for v3. @ajfriend would have to chime in more on this front.
3. There is an example test for v4 using the Cython API directly in your own Cython code, so if you are familiar with Cython, you could presumably call polygon_to_cells directly from a pool of worker threads consuming a queue of polygons to convert.

[hsbsid]

I've been trying to use v4 but I use conda. And conda install h3-py only gives me v3 for some reason.

[dfellis]

So, v4 for Python is still in beta (it's been a prolonged beta because AJ wants to clean up a lot of the perf bottlenecks in the binding and they require breaking changes, so he's bundling them all up along with the breaking changes in the C library). Is there a way to tell conda to install a beta release?

[nrabinowitz]

If the main issue is splitting up the work for specific large polygons, the simplest option is to divide the polygon itself into pieces, e.g. vertical strips, and send those to different threads for processing. The default H3 polyfill mode, which only includes cells whose centers are within the polygon, will ensure that contiguous partial polygons processed in this way will yield the same set as the original polygon, without duplicates or gaps.

[hsbsid]

I have multiple polygons in a GeoDataFrame so I will try multiprocessing. I just wanted to see if there was an existing solution. Thanks.