Skip to content

Commit

Permalink
Remove GeoArrow glue code replacing gpu storage with cudf.Series and …
Browse files Browse the repository at this point in the history
…host storage with pyarrow (#585)

This PR: Closes #575, #584, #588, #591, and #592.

This PR removes all the old `GeoArrowBuffers` logic that was implementing GeoArrow from scratch.
Now, new geometry objects are first parsed into a `pyarrow` `DenseUnion`, then copied to the GPU as `ListSeries`. A `DenseUnion/Like` class `GeoSeries` implements the `input_types` and `union_offsets` for indexing into the four separate `ListSeries`.

When host indexing, the `ListColumns` and `types` and `offsets` are copied into a pyarrow `DenseUnion` that is used for accessing the host side coordinates. At present indexing always copies to host.

This deletes three times as many loc as it creates and trims down the GeoArrow code to nearly the minimum. It is ready for review. 

The next issue will describe the next four tasks that I or @isVoid will implement:
1. Testing and validation of the many `data` arguments types that can be made to the `GeoSeries` constructor.
2. Finalizing an arrow io loop: `x = cuspatial.from_arrow(dense_union); dense_union = x.to_arrow()`
3. Finally, the `__getitem__` method needs to be optimized for zero-copy and memory-direct buffer transfers. Presently it always copies the data into new Shapely objects for serialization.

This closes the original issue #575. I'll create another issue to validate that the new format is correct and notify/compare it with the existing error report and geopandas team.

@trxcllnt @exactlyallan @isVoid @harrism

Authors:
  - H. Thomson Comer (https://github.com/thomcom)

Approvers:
  - Paul Taylor (https://github.com/trxcllnt)
  - Michael Wang (https://github.com/isVoid)
  - Mark Sadang (https://github.com/msadang)

URL: #585
  • Loading branch information
thomcom authored Jul 28, 2022
1 parent f7d9687 commit afdd260
Show file tree
Hide file tree
Showing 18 changed files with 824 additions and 1,585 deletions.
2 changes: 1 addition & 1 deletion conda/environments/cuspatial_dev_cuda11.0.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
- cudf=22.08.*
- cudatoolkit=11.0
- gdal>=3.0.2
- geopandas>=0.9.0,<0.10.0a0
- geopandas>=0.11.0
- cmake>=3.20.1,!=3.23.0
- cython>=0.29,<0.30
- scikit-build>=0.13.1
Expand Down
2 changes: 1 addition & 1 deletion conda/environments/cuspatial_dev_cuda11.1.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
- cudf=22.08.*
- cudatoolkit=11.1
- gdal>=3.0.2
- geopandas>=0.9.0,<0.10.0a0
- geopandas>=0.11.0
- cmake>=3.20.1,!=3.23.0
- cython>=0.29,<0.30
- scikit-build>=0.13.1
Expand Down
2 changes: 1 addition & 1 deletion conda/environments/cuspatial_dev_cuda11.2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
- cudf=22.08.*
- cudatoolkit=11.2
- gdal>=3.0.2
- geopandas>=0.9.0,<0.10.0a0
- geopandas>=0.11.0
- cmake>=3.20.1,!=3.23.0
- cython>=0.29,<0.30
- scikit-build>=0.13.1
Expand Down
2 changes: 1 addition & 1 deletion conda/recipes/cuspatial/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ requirements:
- cudf {{ minor_version }}.*
- rmm {{ minor_version }}.*
- gdal >=3.3.0,<3.4.0a0
- geopandas >=0.9.0,<0.10.0a0
- geopandas >=0.11.0

test: # [linux64]
imports: # [linux64]
Expand Down
1 change: 0 additions & 1 deletion python/cuspatial/cuspatial/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
)
from .geometry.geoseries import GeoSeries
from .geometry.geodataframe import GeoDataFrame
from .geometry.geoarrowbuffers import GeoArrowBuffers
from .io.shapefile import read_polygon_shapefile
from .io.geopandas import from_geopandas

Expand Down
Loading

0 comments on commit afdd260

Please sign in to comment.