Minor pixel differences with wrench

[dati91]

We tried the following test:
https://github.com/szeged/webrender/blob/master/wrench/reftests/border/border-clamp-corner-radius.yaml
Which has a reference image: (we use it flipped for now)
https://github.com/szeged/webrender/blob/master/wrench/reftests/border/border-clamp-corner-radius-ref.png

The result of the test: https://gist.github.com/zakorgy/cb95b2b9e7158ce557f67ec71e9fc8fa
image comparison, max difference: 1, number of differing pixels: 29

Usually we use the following site to compare images quickly: https://huddle.github.io/Resemble.js/
A screenshot about the compare (the purple dots represent the differences, 29 as the result above indicate):

cc @kvark

[kvark]

Ok, I wouldn't worry about this in particular. We can just add a fuzzy value to those tests if max difference is just 1. Must be some differently configured (by the driver) samplers or the rasterizer.

[dati91]

This list contains tests with minor pixel differences (which have the same results as (windowed) opengl)

The full list for opengl test fails is here and for vulkan is here.

reftests/aa/rounded-rects.yaml | image comparison, max difference: 1, number of differing pixels: 1
reftests/border/border-clamp-corner-radius.yaml | image comparison, max difference: 1, number of differing pixels: 22
reftests/border/border-suite.yaml | image comparison, max difference: 8, number of differing pixels: 774
reftests/border/border-suite-2.yaml | image comparison, max difference: 255, number of differing pixels: 352
reftests/border/border-suite-3.yaml | image comparison, max difference: 1, number of differing pixels: 124
reftests/border/border-groove-simple.yaml | image comparison, max difference: 1, number of differing pixels: 24
reftests/border/border-ridge-simple.yaml | image comparison, max difference: 1, number of differing pixels: 24
reftests/border/degenerate-curve.yaml | image comparison, max difference: 1, number of differing pixels: 28
reftests/border/dotted-corner-small-radius.yaml | image comparison, max difference: 127, number of differing pixels: 46
reftests/border/overlapping.yaml | image comparison, max difference: 1, number of differing pixels: 4
reftests/boxshadow/boxshadow-spread-only.yaml | image comparison, max difference: 1, number of differing pixels: 8
reftests/clip/clip-45-degree-rotation.yaml | image comparison, max difference: 1, number of differing pixels: 206
reftests/mask/rounded-corners.yaml | image comparison, max difference: 8, number of differing pixels: 2
reftests/text/colors.yaml | image comparison, max difference: 1, number of differing pixels: 98
reftests/text/alpha-transform.yaml | image comparison, max difference: 66, number of differing pixels: 4300
reftests/text/subpixel-rotate.yaml | image comparison, max difference: 85, number of differing pixels: 4014
reftests/text/subpixel-skew.yaml | image comparison, max difference: 69, number of differing pixels: 4017
reftests/transforms/local-clip.yaml | image comparison, max difference: 9, number of differing pixels: 143
reftests/transforms/rotated-clip.yaml | image comparison, max difference: 1, number of differing pixels: 29
reftests/transforms/rotated-clip-large.yaml | image comparison, max difference: 1, number of differing pixels: 53
reftests/transforms/perspective.yaml | image comparison, max difference: 9, number of differing pixels: 7717
reftests/transforms/coord-system.yaml | image comparison, max difference: 8, number of differing pixels: 402
reftests/transforms/perspective-origin.yaml | image comparison, max difference: 9, number of differing pixels: 5128

[dati91]

OpenGL fails 68 tests when it is windowed.
The full list for opengl test fails is here and for vulkan is here.

Our vulkan backend passes and fails the same tests, and only 9 differs:

reftests/filters/filter-large-blur-radius.yaml == reftests/filters/filter-large-blur-radius.png | image comparison, max difference: 1, number of differing pixels: 6551 <-> our 6543
reftests/filters/blend-clipped.yaml == reftests/filters/blend-clipped.png | image comparison, max difference: 1, number of differing pixels: 1026 <-> our 47357
reftests/image/downscale.yaml == reftests/image/downscale.png | image comparison, max difference: 1, number of differing pixels: 719 <-> our 6840
reftests/image/yuv.yaml == reftests/image/yuv.png | image comparison, max difference: 1, number of differing pixels: 6240 <-> our 795671
reftests/transforms/rotated-image.yaml == reftests/transforms/rotated-image.png | image comparison, max difference: 1, number of differing pixels: 262 <-> our 275
reftests/transforms/prim-suite.yaml == reftests/transforms/prim-suite.png | image comparison, max difference: 9, number of differing pixels: 1699 <-> our 2498
reftests/transforms/content-offset.yaml == reftests/transforms/content-offset.png | image comparison, max difference: 8, number of differing pixels: 21 <-> our 37
reftests/transforms/border-zoom.yaml == reftests/transforms/border-zoom.png | image comparison, max difference: 9, number of differing pixels: 667 <-> our 59800
reftests/transforms/screen-space-blit.yaml == reftests/transforms/screen-space-blit.png | image comparison, max difference: 1, number of differing pixels: 10396 <-> our 10451

It looks to me that maybe a sampler/filter/blend is wrong in some cases.
And the the border corner is missing in the transform cases.

[dati91]

• reftests/transforms/border-zoom.yaml and
• reftests/transforms/prim-suite.yaml
  • Border corner is missing when rotated #156
• reftests/image/downscale.yaml
  • Add mipmap for trilinear filtering #157
• reftests/filters/blend-clipped.yaml
  • Incorrect dithering when running blend-clipped test #158
• reftests/image/yuv.yaml
  • Incorrect yuv feature names #159

The remaining test are only differ in some pixel, maybe these are rounding/precision errors(?)

• reftests/filters/filter-large-blur-radius.yaml
  
• reftests/transforms/rotated-image.yaml
  
• reftests/transforms/content-offset.yaml
  
• reftests/transforms/screen-space-blit.yaml
  

[dati91]

We checked and these tests have the same pixel difference as opengl on windows-vulkan:

• reftests/filters/filter-large-blur-radius.yaml
• reftests/transforms/rotated-image.yaml
• reftests/transforms/content-offset.yaml
• reftests/transforms/screen-space-blit.yaml

Also the yuv test is fixed.
On windows, the border corner is visible and the prim-suite is differs in 1 pixel from gl. (1699 vs 1700)
The border-zoom is still fails, because it is drawn, but not zoomed.

Which leaves us with 3 failing tests on linux-vulkan.

Also there are some (~7) differing tests that only present on windows-vulkan, not in linux-vulkan.

[dati91]

The previously mentioned differing tests will be resolved when #164 lands.

There are two tests that differ from linux-vulkan

REFTEST TEST-UNEXPECTED-FAIL | reftests\transforms\nested-rotate-x.yaml == reftests\transforms\nested-rotate-x.png | image comparison, max difference: 255, number of differing pixels: 36
REFTEST TEST-UNEXPECTED-FAIL | reftests\transforms\nested-preserve-3d.yaml == reftests\transforms\nested-preserve-3d.png | image comparison, max difference: 255, number of differing pixels: 36

Both draw the corner, which should be empty.
But the OpenGL and Angle version both does this on windows. So ours works as intended.

This means we have 2 fails in both windows and linux vulkan:

• reftests/image/downscale.yaml
  • Missing mipmap Add mipmap for trilinear filtering #157
• reftests/filters/blend-clipped.yaml
  • Wrong dithering Incorrect dithering when running blend-clipped test #158

Note: The #156 still exists but its working fine on windows, so must be a linux/driver thing(?)

[dati91]

The DX12 results:
3 tests differ (maybe the dithering?)

• reftests\gradient\premultiplied-angle.yaml
  

• reftests\gradient\premultiplied-angle-2.yaml
  

• reftests\gradient\repeat-border-radius.yaml
  

And we have 19 tests that uses blitting, and it is unsupported in dx12 at the moment:

• reftests\boxshadow\inset-simple.yaml
• reftests\boxshadow\inset-alpha.yaml
• reftests\boxshadow\box-shadow-clip.yaml
• reftests\boxshadow\inset-border-radius.yaml
• reftests\boxshadow\inset-offset.yaml
• reftests\boxshadow\inset-neg-offset.yaml
• reftests\boxshadow\box-shadow-large-blur-radius.yaml
• reftests\boxshadow\rounding.yaml
• reftests\boxshadow\box-shadow-border-radii.yaml
• reftests\boxshadow\inset-downscale.yaml
• reftests\boxshadow\box-shadow-cache.yaml
• reftests\boxshadow\overlap1.yaml
• reftests\boxshadow\box-shadow-stretch-mode-x.yaml
• reftests\boxshadow\box-shadow-stretch-mode-y.yaml
• reftests\filters\filter-blur.yaml
• reftests\filters\filter-large-blur-radius.yaml
• reftests\filters\filter-drop-shadow.yaml
• reftests\filters\filter-drop-shadow-on-viewport-edge.yaml
• reftests\transforms\prim-suite.yaml

Everything else is the same as the vulkan-windows results.

[dati91]

The difference is only with the OSMesa GL. Since we comparing our results with the windowed GL and those are (almost) the same, we can close this issue.