Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read data from vector tile with low memory approach #422

Closed
2 tasks done
springmeyer opened this issue Mar 23, 2015 · 3 comments
Closed
2 tasks done

Read data from vector tile with low memory approach #422

springmeyer opened this issue Mar 23, 2015 · 3 comments
Milestone

Comments

@springmeyer
Copy link
Member

Flagging that this will be viable to consider once these upstream tasks are done:

@springmeyer springmeyer added this to the v3.4.0 milestone Mar 23, 2015
springmeyer pushed a commit to mapbox/tilelive-bridge that referenced this issue Apr 3, 2015
  - this will help keep memory down until mapnik/node-mapnik#419 lands
  - also related to mapnik/node-mapnik#422
springmeyer pushed a commit that referenced this issue Jul 10, 2015
 - Implements #419: `vt.parse()` no longer required
 - Keeps `vt.parse()` as a validator - refs #474
 - Prep work toward #422
 - Moves usage of google's libprotobuf and protoc generated code to cpp only
   in preparation for reducing or fully removing google dep in the future
 - Next step: port more functions away from using `vector_tile::Tile` to reduce
   the overhead of needing to create the object in most cases (except writing)
@springmeyer
Copy link
Member Author

What is done:

Next steps:

Would also be nice to hit eventually (but not performance critical):

  • port vtile.toGeoJSON to use tile_datasource_pbf
  • port vtile.toJSON to use tile_datasource_pbf

Longer term, be able to completely remove libprotobuf dep:

  • add validation function that uses protozero
  • port vtile.addGeoJSON to use protozero writing
  • port map.render(vtile) to use protozero writing
  • port vtile.toJSON to use protozero reading

@springmeyer
Copy link
Member Author

status update:

  • all tests passing
  • protozero now used in all performance critical code paths
  • therefore basically ready to:
    • test with more deps (will require usage updates to avoid vt.parse())
    • consider merging based on the outcome of testing with tilelive-bridge/tilelive-vector, etc
  • for the full perf benefit will be great to hit Implement gzip compression in the C++ layer #456 next and get it updated to protozero code.

@springmeyer
Copy link
Member Author

closing this, will now track at pull that implements the majority of what we need for performance critical paths: #505

@springmeyer springmeyer modified the milestones: v3.4.6, v3.5.0 Sep 28, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant