AWS 2-5: Evaluate Vector Tiles for Global Streams

[rajadain]

• Pull sample data from NGA around the Philadelphia area https://earth-info.nga.mil/
• Convert gpkg into json

  ogr2ogr -f GeoJSON output.json input.gpkg

• Convert json into mbtiles

  tippecanoe -o output.mbtiles --layer="mylayer" output.json

• Convert mbtiles into vector tiles

  mb-util output.mbtiles ./tiles

• Upload the tiles to the AWS bucket that hosts other tiles for MMW
• Add VectorGrid plugin to Leaflet
• Add the tile layer as a test

[aufdenkampe]

@rajadain, if there is a way to avoid the intermediate conversion to GeoJSON, it might be worthwhile, given the size of these files.

Also, for the Philly region, the TDXHydroRegion code (HYBAS_ID Level 2 code) for the US East Coast is 7020038340.

cc: @kieranbartels

[rajadain]

Thank you! Using the --download --continue --output flags with xh to download the source files, since the connection keeps closing (as also noted in WikiWatershed/global-hydrography#2 (comment))

xh --verbose --follow \
   --download --continue \
   --output TDX_streamnet_7020038340_01.gpkg \
   https://earth-info.nga.mil/php/download.php file==7020038340-streamnet-gpkg

xh --verbose --follow \
   --download --continue \
   --output TDX_streamreach_basins_7020038340_01.gpkg \
   https://earth-info.nga.mil/php/download.php file==7020038340-basins-gpkg

I'll investigate and report on gpkg ➡️ mbtiles without GeoJSON.

[rajadain]

Using ogr2ogr to directly convert mbtiles failed silently because there were too many features. This did create a file but it was corrupt and could not be read by QGIS.

ogr2ogr -f "MBTiles" TDX_streamnet_7020038340_01.mbtiles TDX_streamnet_7020038340_01.gpkg

I then used ogr2ogr to convert to GeoJSON first:

ogr2ogr -f "GeoJSON" TDX_streamreach_basins_7020038340_01.geojson TDX_streamreach_basins_7020038340_01.gpkg

which took 1m 21s for the streams and 4m 46s for the basins.

I then used tippecanoe with the --coalesce-densest-as-needed option to make vector tiles while coalescing dense features at lower zoom levels:

tippecanoe --coalesce-densest-as-needed -o TDX_streamreach_basins_7020038340_01.mbtiles TDX_streamreach_basins_7020038340_01.geojson
For layer 0, using name "TDX_streamreach_basins_7020038340_01"
140053 features, 494510345 bytes of geometry and attributes, 1023547 bytes of string pool, 0 bytes of vertices, 0 bytes of nodes
tile 2/1/1 size is 652706 with detail 12, >500000
Going to try keeping the sparsest 68.94% of the features to make it fit
tile 2/1/1 size is 671513 with detail 12, >500000
Going to try keeping the sparsest 46.20% of the features to make it fit
tile 2/1/1 size is 646371 with detail 12, >500000
Going to try keeping the sparsest 32.17% of the features to make it fit
tile 2/1/1 size is 590968 with detail 12, >500000
Going to try keeping the sparsest 24.49% of the features to make it fit
tile 2/1/1 size is 540966 with detail 12, >500000
Going to try keeping the sparsest 20.37% of the features to make it fit
tile 2/1/1 size is 501914 with detail 12, >500000
Going to try keeping the sparsest 18.27% of the features to make it fit
tile 3/2/3 size is 1292992 with detail 12, >500000
Going to try keeping the sparsest 34.80% of the features to make it fit
tile 3/2/3 size is 799637 with detail 12, >500000
Going to try keeping the sparsest 19.59% of the features to make it fit
tile 3/2/3 size is 588086 with detail 12, >500000
Going to try keeping the sparsest 14.99% of the features to make it fit
tile 3/2/3 size is 503840 with detail 12, >500000
Going to try keeping the sparsest 13.39% of the features to make it fit
tile 4/4/5 size is 619048 with detail 12, >500000
Going to try keeping the sparsest 72.69% of the features to make it fit
tile 4/4/5 size is 531181 with detail 12, >500000
Going to try keeping the sparsest 61.58% of the features to make it fit
tile 4/4/6 size is 1880139 with detail 12, >500000
Going to try keeping the sparsest 23.93% of the features to make it fit
tile 4/4/6 size is 936408 with detail 12, >500000
Going to try keeping the sparsest 11.50% of the features to make it fit
tile 4/4/6 size is 647020 with detail 12, >500000
Going to try keeping the sparsest 8.00% of the features to make it fit
tile 4/4/6 size is 541108 with detail 12, >500000
Going to try keeping the sparsest 6.65% of the features to make it fit
tile 5/9/11 size is 893101 with detail 12, >500000
Going to try keeping the sparsest 50.39% of the features to make it fit
tile 5/9/12 size is 835362 with detail 12, >500000
Going to try keeping the sparsest 53.87% of the features to make it fit
tile 5/8/13 size is 920395 with detail 12, >500000
Going to try keeping the sparsest 48.89% of the features to make it fit
tile 5/8/12 size is 1301132 with detail 12, >500000
Going to try keeping the sparsest 34.59% of the features to make it fit
tile 5/9/11 size is 644830 with detail 12, >500000
Going to try keeping the sparsest 35.16% of the features to make it fit
tile 5/9/12 size is 623987 with detail 12, >500000
Going to try keeping the sparsest 38.85% of the features to make it fit
tile 5/8/13 size is 662928 with detail 12, >500000
Going to try keeping the sparsest 33.19% of the features to make it fit
tile 5/9/11 size is 554818 with detail 12, >500000
Going to try keeping the sparsest 28.52% of the features to make it fit
tile 5/9/12 size is 542914 with detail 12, >500000
Going to try keeping the sparsest 32.20% of the features to make it fit
tile 5/8/12 size is 801008 with detail 12, >500000
Going to try keeping the sparsest 19.43% of the features to make it fit
tile 5/8/13 size is 562073 with detail 12, >500000
Going to try keeping the sparsest 26.57% of the features to make it fit
tile 5/9/11 size is 506529 with detail 12, >500000
Going to try keeping the sparsest 25.34% of the features to make it fit
tile 5/9/12 size is 501366 with detail 12, >500000
Going to try keeping the sparsest 28.90% of the features to make it fit
tile 5/8/12 size is 626423 with detail 12, >500000
Going to try keeping the sparsest 13.96% of the features to make it fit
tile 5/8/13 size is 515262 with detail 12, >500000
Going to try keeping the sparsest 23.21% of the features to make it fit
tile 5/8/12 size is 539148 with detail 12, >500000
Going to try keeping the sparsest 11.65% of the features to make it fit
tile 6/16/26 size is 530131 with detail 12, >500000
Going to try keeping the sparsest 84.88% of the features to make it fit
tile 6/18/23 size is 579130 with detail 12, >500000
Going to try keeping the sparsest 77.70% of the features to make it fit
tile 6/19/23 size is 642635 with detail 12, >500000
Going to try keeping the sparsest 70.02% of the features to make it fit
tile 6/16/25 size is 690931 with detail 12, >500000
Going to try keeping the sparsest 65.13% of the features to make it fit
tile 6/17/26 size is 775872 with detail 12, >500000
Going to try keeping the sparsest 58.00% of the features to make it fit
tile 6/18/24 size is 918423 with detail 12, >500000
Going to try keeping the sparsest 49.00% of the features to make it fit
tile 6/18/23 size is 517967 with detail 12, >500000
Going to try keeping the sparsest 67.51% of the features to make it fit
tile 6/17/25 size is 1166806 with detail 12, >500000
Going to try keeping the sparsest 38.57% of the features to make it fit
tile 6/19/23 size is 551690 with detail 12, >500000
Going to try keeping the sparsest 57.12% of the features to make it fit
tile 6/16/25 size is 573903 with detail 12, >500000
Going to try keeping the sparsest 51.07% of the features to make it fit
tile 6/17/26 size is 622621 with detail 12, >500000
Going to try keeping the sparsest 41.92% of the features to make it fit
tile 6/19/23 size is 508231 with detail 12, >500000
Going to try keeping the sparsest 50.57% of the features to make it fit
tile 6/18/24 size is 682958 with detail 12, >500000
Going to try keeping the sparsest 32.28% of the features to make it fit
tile 6/16/25 size is 522275 with detail 12, >500000
Going to try keeping the sparsest 44.00% of the features to make it fit
tile 6/17/25 size is 798846 with detail 12, >500000
Going to try keeping the sparsest 21.73% of the features to make it fit
tile 6/17/26 size is 551858 with detail 12, >500000
Going to try keeping the sparsest 34.18% of the features to make it fit
tile 6/18/24 size is 588964 with detail 12, >500000
Going to try keeping the sparsest 24.67% of the features to make it fit
tile 6/17/26 size is 515597 with detail 12, >500000
Going to try keeping the sparsest 29.83% of the features to make it fit
tile 6/17/25 size is 655699 with detail 12, >500000
Going to try keeping the sparsest 14.91% of the features to make it fit
tile 6/18/24 size is 536812 with detail 12, >500000
Going to try keeping the sparsest 20.68% of the features to make it fit
tile 6/18/24 size is 501275 with detail 12, >500000
Going to try keeping the sparsest 18.56% of the features to make it fit
tile 6/17/25 size is 569560 with detail 12, >500000
Going to try keeping the sparsest 11.78% of the features to make it fit
tile 6/17/25 size is 521781 with detail 12, >500000
Going to try keeping the sparsest 10.16% of the features to make it fit
tile 7/36/48 size is 506221 with detail 12, >500000
Going to try keeping the sparsest 88.89% of the features to make it fit
tile 7/37/47 size is 546900 with detail 12, >500000
Going to try keeping the sparsest 82.28% of the features to make it fit
tile 7/36/49 size is 575239 with detail 12, >500000
Going to try keeping the sparsest 78.23% of the features to make it fit
tile 7/34/52 size is 692665 with detail 12, >500000
Going to try keeping the sparsest 64.97% of the features to make it fit
tile 7/37/47 size is 506969 with detail 12, >500000
Going to try keeping the sparsest 73.04% of the features to make it fit
tile 7/33/52 size is 535875 with detail 12, >500000
Going to try keeping the sparsest 83.97% of the features to make it fit
tile 7/36/50 size is 587870 with detail 12, >500000
Going to try keeping the sparsest 76.55% of the features to make it fit
tile 7/36/49 size is 525239 with detail 12, >500000
Going to try keeping the sparsest 67.02% of the features to make it fit
tile 7/35/50 size is 617138 with detail 12, >500000
Going to try keeping the sparsest 72.92% of the features to make it fit
tile 7/33/52 size is 500558 with detail 12, >500000
Going to try keeping the sparsest 75.49% of the features to make it fit
tile 7/35/51 size is 528595 with detail 12, >500000
Going to try keeping the sparsest 85.13% of the features to make it fit
tile 7/34/52 size is 595097 with detail 12, >500000
Going to try keeping the sparsest 49.13% of the features to make it fit
tile 7/36/50 size is 532791 with detail 12, >500000
Going to try keeping the sparsest 64.65% of the features to make it fit
tile 7/33/51 size is 629585 with detail 12, >500000
Going to try keeping the sparsest 71.48% of the features to make it fit
tile 7/35/50 size is 547841 with detail 12, >500000
Going to try keeping the sparsest 59.89% of the features to make it fit
tile 7/34/51 size is 644647 with detail 12, >500000
Going to try keeping the sparsest 69.81% of the features to make it fit
tile 7/33/51 size is 551970 with detail 12, >500000
Going to try keeping the sparsest 58.27% of the features to make it fit
tile 7/36/50 size is 504142 with detail 12, >500000
Going to try keeping the sparsest 57.71% of the features to make it fit
tile 7/34/52 size is 545322 with detail 12, >500000
Going to try keeping the sparsest 40.54% of the features to make it fit
tile 7/35/50 size is 513591 with detail 12, >500000
Going to try keeping the sparsest 52.48% of the features to make it fit
tile 7/34/51 size is 562820 with detail 12, >500000
Going to try keeping the sparsest 55.81% of the features to make it fit
tile 7/33/51 size is 515896 with detail 12, >500000
Going to try keeping the sparsest 50.83% of the features to make it fit
tile 7/34/52 size is 514713 with detail 12, >500000
Going to try keeping the sparsest 35.44% of the features to make it fit
tile 7/34/51 size is 522394 with detail 12, >500000
Going to try keeping the sparsest 48.08% of the features to make it fit
  99.9%  14/4625/6297

which took 2m 40s for the streams and 21m 42s for the basins on my M1Pro MacBook Pro.

From a file size perspective, for streams:

du -sh TDX_streamnet_7020038340_01.*

670M	TDX_streamnet_7020038340_01.gpkg
1.6G	TDX_streamnet_7020038340_01.geojson
400M	TDX_streamnet_7020038340_01.mbtiles

The original dataset was 670M, the intermediate GeoJSON 1.6G, and the final mbtiles 400M.

For basins:

du -sh TDX_streamreach_basins_7020038340_01.*

2.5G	TDX_streamreach_basins_7020038340_01.gpkg
6.6G	TDX_streamreach_basins_7020038340_01.geojson
784M	TDX_streamreach_basins_7020038340_01.mbtiles

The original dataset was 2.5G, the intermediate GeoJSON 6.6G, and the final mbtiles 784M.

Next I'll extract the mbtiles into vector tiles and then upload to S3, then use in MMW.

[rajadain]

Using the script from azavea/iow-boundary-tool@a2f157a, I uploaded the generated tiles to S3 and they are available at https://a.tiles.azavea.com/TDX_streamnet_7020038340_02/metadata.json.

[rajadain]

Because we're using Leaflet 0.7 in Model My Watershed, we cannot use the VectorGrid plugin. Instead, I looked into the MapboxVectorTile plugin, which is quite old but works decently. Getting it to work was a bit challenging, since the current dist file is not importable. I made a variant that is.

With that, and now integrated into MMW, we can see that the performance is pretty good:

2024-06-12.16.58.19.mp4

There's still a couple things to figure out: styling and overzooming. But it's good to get a sense of the real world feel for this.

The lower zoom coalescing is not very helpful because it's just a solid block at that level. Will investigate if I can make a version using stream order.

[rajadain]

It's also important to see how we can combine two source TDX files to create one mbtiles folder.