Compression

[pnorman]

Currently the specification does not allow compression, but some PBFs seen in the wild are compressed, and not at the transport layer (HTTP).

The specification should be explicit on if compression is allowed.

[kkaefer]

Compression should always happen at the transportation/storage layer. Do you have an example of a PBF in the wild that is served compressed?

[pnorman]

quoting @ajashton when I sent him an uncompressed tile

Mapbox Studio expects PBF tiles to be compressed with either gzip or deflate. When served with gzip-encoding I was able to load and style the tile with the tilejson you provided.

cc @zerebubuth

[gravitystorm]

mapbox/tilelive-vector#71 (comment) and mapbox/tilelive-bridge#21 (comment) both suggest that the tiles should be compressed, even at rest on a plain disk.

[kkaefer]

Yeah, but the Spec doesn't provide a storage facility, so it is up to the storage mechanisms to transparently compress/decompress the data.

[zerebubuth]

What's confusing me is whether this should be happening at the transport (or storage) layer, as @kkaefer is saying, or is an intrinsic part of being a "mapnik vector tile" as @jfirebaugh is saying.

If it's handled at the transport layer, then an HTTP request for a mapnik vector tile which specifies no Accept-Encoding (or "Accept-Encoding: identity") should receive an uncompressed PBF. If it's intrinsic, then it should receive a gzip compressed PBF, despite still having "Encoding: identity".

[jfirebaugh]

The conventions are at the API level, not the specification level, and are driven by pragmatic constraints:

1. Vector tiles benefit from compression in transport and in storage.
2. We wish to offload decompression to the user agent in client side (GL rendering) scenarios.
3. We wish to avoid repetitive decompression/recompression when a VT travels through a tilelive pipeline.
4. gzip is the only reliable transport encoding for HTTP.

Together, these constraints led to the current solution: vector tiles are gzipped at rest, and served with Content-Encoding: gzip for HTTP transport. Browser user agents will automatically decompress HTTP bodies, satisfying #2. Non-browser user agents typically do not automatically decompress HTTP bodies, satisfying #3.

This means:

• vector-tile-js expects uncompressed vector tile data
• tilelive-* modules expect gzipped input and generate gzipped output

[pnorman]

cc @yohanboniface, as another author of vector tile consuming software

Leaving aside the storage what you're saying is vector tiles are uncompressed, but

• HTTP requests to feed tilelive-* modules will have an Accept-Encoding that only accepts gzip, e.g. Accept-Encoding: gzip;q=1, identity;q=0 and will need to not decompress the response before passing it to the tilelive-* module
• Servers based on tilelive-* modules will respond with a 406 error to Accept-Encoding: identity;q=1, gzip;q=0 as they only have a gzipped version of the file and do not decompress on the fly

Is this correct?

On-disk storage is an interesting question, as what is optimal can depend on the filesystem, the disks, and what is consuming the tiles.

[bcamper]

FWIW we ran into this issue recently when @ericfischer was trying to use the Mapzen vector tiles in Mapbox Studio. We were serving gzipped tiles when Accept-Encoding included gzip, but it looked like Studio was not including an Accept-Encoding header in its requests. We changed the service to always serve gzipped tiles for compatibility with the current Studio behavior, but ideally we would only do so when Accept-Encoding is present.

[pnorman]

We were serving gzipped tiles when Accept-Encoding included gzip, but it looked like Studio was not including an Accept-Encoding header in its requests.

Serving gzip if Accept-Encoding is not specified is okay. The HTTP spec says servers should use identity in such cases, not must. Serving gzip if identity is requested is broken.

I must admit that I haven't looked at the MBS HTTP requests - does it currently send an accept-encoding header which disallows identity encoding?

[bcamper]

I cannot recall offhand, I believe no Accept-Encoding was present, I'm sure a Mapboxer here could say :)

[flippmoke]

Currently we do not have any text in version 2.0 pull request about compression, but I feel we should likely add something at least describing it? I understand that it is often dealt with on a transport level by the browsers, but for server side processing in the wild there are a variety of compressed and uncompressed tiles. If anyone would like to create a pull request to the v2-development branch with these changes it would be GREATLY appreciated.

[e-n-f]

It matters when you are making mbtiles too, because if you don't compress (or if you use zlib instead of gzip) you can't upload it. (Should that be in the mbtiles spec, instead, though?)

[pnorman]

Currently we do not have any text in version 2.0 pull request about compression, but I feel we should likely add something at least describing it? I understand that it is often dealt with on a transport level by the browsers, but for server side processing in the wild there are a variety of compressed and uncompressed tiles. If anyone would like to create a pull request to the v2-development branch with these changes it would be GREATLY appreciated

I'm actually happy with how the spec is silent on compression.

A m-v-t file is not compressed. It is often useful to compress m-v-t when storing them, but then you have a compressed m-v-t file, not a m-v-t file. Analogously, planet-latest.osm.bz2 is bzipped XML, not just XML.

With transport-level compression and HTTP the content is uncompressed, and the gzip encoded content is compressed.

Specifications for storage (e.g. mbtiles, disk layouts, etc) should address if the tiles have compression applied to them.

---

It matters when you are making mbtiles too, because if you don't compress (or if you use zlib instead of gzip) you can't upload it

The m-v-t spec doesn't address uploading, just the format of a single vector tile. As it stands, mbtiles cannot store vector tiles, only png or jpg

#39 proposes language which is clearer on the vector tile being different than the system that encodes its bounds and projection (TMS numbering, google maps tile numbers, etc)

[kkaefer]

@ericfischer, I agree with @pnorman in that the respective storage formats/APIs should handle compression, not the vector tile spec.

[e-n-f]

Sounds good to me to address this in the mbtiles spec instead.

[flippmoke]

All of this sounds great to me! We probably should encourage the use of .mvt.gz ?

[pnorman]

yes, if storing gzipped mvts, .mvt.gz makes sense for a filename.

[flippmoke]

Have we reached a general agreement around this, if so I am going to close the ticket to mark this task as closed.

[kkaefer]

Yes.