Create fixture suite for new 2.0 spec validations

[mapsam]

Background

This came up in a recent 2.0 spec update of mapnik-upload-validate where @springmeyer encountered errors that weren't being thrown after we had removed the deprecated vt.parse() method. Adding the check back in helped, but we are still in a spot where "our existing test coverage appears to be completely missing for this code".

The question now is do we write tests for invalid tiles within mapnik-upload-validate or do we ensure all of these potential failures have been properly tested UPSTREAM in mapnik-vector-tile?

Action

This ticket is a reminder to put together a suite of invalid vector tiles so we can test within mapnik-vector-tile to safely ensure downstream tests in other modules depending on node-mapnik are not re-testing the same exact stuff.

Tasks

We'll want to cover all of the validation errors in vector_tile_is_valid.hpp:

• Vector Tile message has two or more layers with the same name
• Vector Tile message has an unknown tag
• Vector Tile Layer message has no name
• Vector Tile Layer message has no extent
• Vector Tile Layer message has no features
• Vector Tile Layer message has an invalid version
• Vector Tile Layer message has an unknown tag
• Vector Tile Value message contains more then one values
• Vector Tile Value message contains no values
• Vector Tile Value message has an unknown tag
• Vector Tile Feature message has no geometry
• Vector Tile Feature message has multiple geometries
• Vector Tile Feature message is missing a geometry type
• Vector Tile Feature message has an invalid geometry type
• Vector Tile Feature message has an unknown tag
• Buffer is not encoded as a valid PBF
• UNKNOWN ERROR

We'll also need a solid workflow in place to actually create these tiles.

Related question: what is our current status for properly formatted tiles and tests written around them?

[mapsam]

A number of these cases are covered by evilmvt. I will work on creating the others today!

[mapsam]

@flippmoke I've created MVT buffers for every case above and more that we can start using in Mapnik Vector Tile! https://github.com/mapbox/evilmvt

[mapsam]

Update: Now at https://github.com/mapbox/mvt-fixtures which includes valid MVT fixtures as well. I'll start working on including these into the library!

[pnorman]

I recommend testing that any fields in the protobuf lie within the numeric range of the fields of the spec, i.e. that it's not writing to fields not defined in https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto

[springmeyer]

Work on a new and better fixture suite is now taking place at mapbox/mvt-fixtures#15. Once that is done, we'll integrate back here. Creating a new ticket to track that integration -> #255