Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use rtree for large polygons #479

Merged
merged 9 commits into from
Mar 25, 2023
Merged

Use rtree for large polygons #479

merged 9 commits into from
Mar 25, 2023

Conversation

systemed
Copy link
Owner

This is a revival and reworking of #323.

The bounding boxes of large polygons (e.g. ocean) are stored in an rtree data structure rather than for each individual z14 tile. We then query this rtree when writing out each tile, and add any objects we find.

Tests with a couple of smaller areas (Devon and Maine) suggest little to no performance impact.

@systemed
Copy link
Owner Author

systemed commented Mar 23, 2023

With the acid test of Antarctica and no bbox specified, memory consumption is halved (yay!) but execution time is doubled (boo!).

I suspect much of the slowdown is due to the situation outlined in #323 (comment) where we're processing the polygon for all tiles which intersect its bounding box. Previously we were only processing those tiles which cover the polygon (as per fillCoveredTiles in coordinates.cpp).

Options to investigate:

  • use a faster clipping algorithm (boost.geometry intersection is perhaps too generalised for our rectangular clipping case)
  • store a tile bitmap with each rtree (as a vector, though this of course will increase memory consumption)
  • cut into smaller rtrees, maybe with some sort of binary divide algorithm
  • split the largest polygons into multiple tiled polygons on initial read (e.g. z8 tilesize + a buffer)
  • add multipolygon constituent polygons separately in OsmLuaProcessing::setRelation

@systemed
Copy link
Owner Author

More tests with Antarctica:

With Europe (using --store):

  • fast_clip: 4hr34, 52.2Gb
  • rtree3 + fast_clip: 4hr27, 50.1Gb

I think that's a strong case for merging this. It makes a big difference to RAM usage in the polar regions; generally it's neutral on performance; and although there is a performance impact for Antarctica, #482 reduces it and arguably it's worth it for the RAM saving.

@systemed systemed merged commit bf64cc3 into master Mar 25, 2023
@systemed systemed deleted the rtree3 branch March 25, 2023 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant