Test coverage

[dati91]

In order to check our progress, we use the wrench's reftests.

Current status: N/A

[zakorgy]

REFTEST INFO | 107 passing, 123 failing
There are passing tests, which compare two blank images, since we only support primitive shaders. Also there are tests which differs in a low number of pixels (less than 1%) and fails, due to the different driver. So the above result is not hundred percent accurate.

[dati91]

The latest result: REFTEST INFO | 111 passing, 135 failing
The full log: reftest_result.txt

[dati91]

With #62 and #63 our latest numbers: REFTEST INFO | 142 passing, 104 failing

I also run the tests with original (windowed opengl) wr: REFTEST INFO | 194 passing, 54 failing

So we are not that far!

[dati91]

With #66: REFTEST INFO | 133 passing, 113 failing

This fixes some tests and also removes some of the "fake" passing results.

[dati91]

REFTEST INFO | 175 passing, 71 failing with the current master. (This is on linux with vulkan (Intel driver))

[dati91]

So with the release of vulkan 1.1 I upgraded my driver to the latest (padoka driver) (but it is just the 1.0.57 api version), and got the following numbers for the tests:
REFTEST INFO | 162 passing, 84 failing (with this revision)
I'm not sure I have the previous test result list, but if I find it, i will check which tests went rogue.

[dati91]

https://gist.github.com/dati91/5a5184f6950d7b02bb3937b1e32338bf
A test with more than a ~1000 different pixels is most probably should fail, less than that is most probably just a driver difference and should be considered passing.
There are tests with more than 5000 different pixels, which are probably totally wrong and maybe deserve their on issues to look into.

[dati91]

REFTEST INFO | 186 passing, 73 failing with the current master. (This is on linux with vulkan (Intel driver))

[dati91]

With the current master:
REFTEST INFO | 196 passing, 62 failing (This is on linux with vulkan (Intel driver))
REFTEST INFO | 185 passing, 58 failing (This is on windows with vulkan (Intel driver))

[zakorgy]

With the current master:
REFTEST INFO | 201 passing, 58 failing (This is on linux with vulkan (Intel driver))
From the 58 only 8 differs from the original webrender OpenGL(not mesa) results.
REFTEST INFO | 187 passing, 56 failing (This is on windows with vulkan (Intel driver))
From the 56 only 6 differs from the original webrender OpenGL(not mesa) results.

[dati91]

Our first wrench result with dx12:
REFTEST INFO | 190 passing, 53 failing
Which is not totally accurate, because we skip the blitting and 3 tests still pass (from 15 that uses blitting), because its a yaml to yaml comparison.
So the real numbers are REFTEST INFO | 187 passing, 56 failing. A detailed result.
Note: the vulkan/windows numbers are the same above, but the passing/failing tests differ.

[dati91]

With the current master:
Linux

• GL REFTEST INFO | 214 passing, 66 failing
  • Our baseline, the original WR with windowed GL
• VK REFTEST INFO | 214 passing, 66 failing
  • We fail ~4 tests with less than 50 pixels differences, these are most probably precision errors
  • We fail the blend-clipped test, because of an incorrect dithering Incorrect dithering when running blend-clipped test #158
  • We fail the transformed border-corner tests, because of ... not sure Border corner is missing when rotated #156

Windows

• GL REFTEST INFO | 197 passing, 8 failing
  • Our baseline, the original WR with windowed Angle
• VK REFTEST INFO | 197 passing, 8 failing
  • We have the same results
• DX12 REFTEST INFO | 196 passing, 9 failing
  • The blitting is not supported in dx12 yet, which fails 1 test
  • Also there are 4 tests in total which uses blitting, but the other 3 tests "passes" because its not a png compare

Note: On windows the blend-clipped and the transform border-corner tests are skipped, but i tested both. The blend-clipped had the same result as on linux, but the transform border-corner tests had the expected results.

[dati91]

With the current master, we have the dx12 blitting support. So now we pass the same tests as the other backends.

[dati91]

With the current master:

Linux

• GL REFTEST INFO | 215 passing, 66 failing
  • Our baseline, the original WR with windowed GL
• VK REFTEST INFO | 215 passing, 66 failing
  • We fail the blend-clipped test, because of an incorrect dithering Incorrect dithering when running blend-clipped test #158
  • 4 tests differ in some pixels | gl vs vulkan
    • filters/filter-large-blur-radius.yaml | 6546 vs 6533
    • transforms/rotated-image.yaml | 262 vs 275
    • content-offset.yaml | 21 vs 37
    • transforms/near-plane-clip.yaml | 8181 vs 8200

Windows

• GL REFTEST INFO | 197 passing, 6 failing
  • Our baseline, the original WR with windowed GL
• VK & DX12 REFTEST INFO | 197 passing, 6 failing
  • Both Vulkan and DX12 have the exact same result as GL

Mac

N/A

[dati91]

A quick update on Mac/Metal:

REFTEST INFO | 212 passing, 58 failing

We can't really compare it to a windowed gl, because on mac it doesn't work correctly (only the headless).

So we compered to linux/vulkan (REFTEST INFO | 224 passing, 67 failing we should update the previous numbers).
Note: on mac some tests are skipped (compered to linux) and there are 2 only mac test as well.

The following tests are differ for sure:

REFTEST TEST-UNEXPECTED-FAIL | reftests/border/border-no-bogus-line.yaml == reftests/border/border-no-bogus-line-ref.png | image comparison, max difference: 255, number of differing pixels: 3138
REFTEST TEST-UNEXPECTED-FAIL | reftests/filters/filter-small-blur-radius.yaml == reftests/filters/filter-small-blur-radius.png | image comparison, max difference: 255, number of differing pixels: 252877
REFTEST TEST-UNEXPECTED-FAIL | reftests/mask/rounded-corners.yaml == reftests/mask/rounded-corners.png | image comparison, max difference: 255, number of differing pixels: 1106
REFTEST TEST-UNEXPECTED-FAIL | reftests/split/nested-preserve3d-crash.yaml == reftests/split/nested-preserve3d-crash.yaml | image comparison, max difference: 255, number of differing pixels: 63067
REFTEST TEST-UNEXPECTED-FAIL | reftests/text/shadow-grey-transparent.yaml == reftests/text/shadow-grey-ref.yaml | image comparison, max difference: 1, number of differing pixels: 662
REFTEST TEST-UNEXPECTED-FAIL | reftests/transforms/content-offset.yaml == reftests/transforms/content-offset.png | image comparison, max difference: 255, number of differing pixels: 16765