performance test harness

[jfirebaugh]

Questions about measuring relative performance of different rendering approaches come up frequently:

• rendering layer-by-layer vs tile-by-tile
• tesselation strategy
• fill-outline vs separate layers

How can we accurately and repeatably measure rendering performance, so that we can make data-driven decisions about different possible rendering approaches?

Desiderata:

• Use real-world data
• Use real-world styles
• In browser
• Multiple browsers (Chrome + FF at least)
• Multiple OS/graphics card?
• Something automatable, that could be run in CI and flag performance regressions, would be a huge plus

[ljbade]

A few thoughts:

• Doing this in a CI will be difficult unless you use mesa but the performance will be different to real GPU.
• Important consideration is desktop vs mobile GPUs which have very different architecture
• Real world data/styles - shouldn't be too hard, random selection of Mapbox Streets locations, perhaps weighted by popularity/number of views
• Perhaps could leverage the Android cloud testing services to test performance on a large variety of devices/GPUs
• We should add IE to the browser list since a lot of people/businesses still use it as only browser (if it is to be supported)
• Will want to have offline/fast cache of any network data for tiles/styles to remove influence of network speed

[ansis]

This would be very useful. Do you have any ideas on how it should be implemented?

[lucaswoj]

Related #1336

[lucaswoj]

Most of the actionable ideas in this ticket have been done or have been ticketed out separately.