Arbitrary benchmarks against geopandas

[deanm0000]

In taking the 85529 US census tracts and running simplify with an epsilon of 0.5:

geoarrow:

%%timeit
tracts.geometry.simplify(0.5)
2.22 s ± 305 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

geopandas:

%%timeit
tractsgpd.geometry.simplify(0.5)
53.6 s ± 1.66 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

[kylebarron]

I understand if it's too much work, but it would be great to have this in a fully reproducible script

[kylebarron]

BTW: can I do better than this for getting my new geometry into the Table

I don't think there's a better way than that right now. There's a couple main Python-specific refactors I'm planning when I find the time. If you're interested in contributing, let me know and we can coordinate

[deanm0000]

I don't think it's making copies so I'm not overly concerned about the verbosity, just asking in case I'm missing something.

There's a couple main Python-specific refactors I'm planning when I find the time. If you're interested in contributing, let me know and we can coordinate

let me know what you have in mind. I'm deanm0000 on discord if you want to take this off the public thread.

[kylebarron]

I don't think it's making copies so I'm not overly concerned about the verbosity

No, it's definitely not making copies; it's just annoying to write.

let me know what you have in mind.

It's fine for this to be public, but we can chat on discord if you want.

One of my goals is to make it easy for lots of people, not just me, to make Arrow-based Python/rust libraries. E.g. stac-utils/stac-rs#256 (comment) is an example where someone else might want to make Python bindings to an Arrow-based Rust library. But it's a massive pain for people to figure out Arrow FFI.

arro3-* is the start of a minimal collection of Python libraries for working with Arrow data in Python with a tiny footprint. But just as important in our case is arro3-internal (which I want to rename to pyo3-arrow) which manages all the FFI between Python and Rust and is targeted at Rust consumers. This doesn't use any of arrow-rs's Python support, because being tied to its version of pyo3 and its release schedule is too much IMO, and it's quite a small amount of code that I need to copy over.

arro3-compute has an example of how easy it is to export a function using arro3-internal. It's just 4 lines of code to wrap arrow_select::concat and export it to Python, where input and output is done via Arrow PyCapsule FFI.

This approach also grows the Arrow PyCapsule ecosystem (which you've also seen in polars), which I think is really beneficial to ensuring interoperability of Arrow data in Python, especially for smaller libraries like my own and hopefully others'.

So bringing this back to geoarrow-rs, I want to refactor geoarrow-rs' Python API to remove any of its own classes. geoarrow-rs should use pyo3-arrow to import any Arrow data from the pycapsule interface and should always return arro3.core Python objects. This means there's a tiny dependency on arro3-core from geoarrow.rust.core. But it also means that the returned Arrow class is always the same physical class (e.g. I think it's confusing to have arro3.core.Table and geoarrow.rust.core.Table when they're the exact same thing (or potentially different versions of the same class)). So essentially arro3.core.Table is dynamically linked via Python using the fact that Arrow is FFI stable.

[deanm0000]

I haven't had time to dig through that too much but will take a look in the next couple weeks.

[kylebarron]

I published pyo3-arrow 0.1 yesterday https://docs.rs/pyo3-arrow/latest/pyo3_arrow/

[deanm0000]

So I realize now that this is an unfair benchmark. It turns out that shapely's simplify has a preserve_topology parameter which is True by default causing it to run a much more expensive algorithm. When I set that to False then it performs about the same.

[kylebarron]

Ok yeah that makes more sense. Running with preserve_topology is probably more similar to https://docs.rs/geo/latest/geo/algorithm/simplify_vw/trait.SimplifyVwPreserve.html, which is integrated with geoarrow-rs but not sure if it's integrated with the python bindings already