ReferenceTests Comparison scores failure on master for all three platforms

[bjarthur]

i'm trying to add a new reference image for #4293 and having trouble getting the comparison scores to all fall below the threshold when testing locally. in a --temp environment with Makie, MakieCore, GLMakie, and ReferenceTests all dev'd to a 0-day master, i get some variation of the following errors on apple silicon, xvfb-run headless ubuntu 22.04, and windows 10 (both the latter have titan rtx gpus):

error log

(jl_rwOy5p) pkg> test GLMakie
     Testing GLMakie
      Status `/private/var/folders/s5/8d629n5d7nsf37f60_91wzr40000gq/T/jl_bFliFt/Project.toml`

<SNIP>

Precompiling project...
  7 dependencies successfully precompiled in 61 seconds. 313 already precompiled.
     Testing Running tests...
┌ Warning: TESTING WITHOUT OPENGL DEBUGGING
└ @ Main ~/.julia/dev/Makie/GLMakie/test/runtests.jl:11
Test Summary: | Pass  Total  Time
mimes         |    3      3  1.9s
Test Summary: | Pass  Total  Time
shader cache  |   18     18  3.6s
Test Summary: | Pass  Total  Time
unit tests    |   30     30  5.3s
Test Summary: | Pass  Total  Time
emtpy!(fig)   |   47     47  1.3s
Test Summary: | Pass  Total  Time
empty!(ax)    |   52     52  1.0s
Test Summary: | Pass  Total  Time
closing       |    2      2  3.2s
Test Summary:               | Pass  Total  Time
destroying singleton screen |    1      1  2.1s
Test Summary:             | Pass  Total  Time
stresstest multi displays |  134    134  9.2s
Test Summary:  | Pass  Broken  Total  Time
HiDPI displays |   18       2     20  1.7s
Test Summary:      |Time
image size changes | None  0.6s
Test Summary:                       | Pass  Total  Time
Verify camera uniforms after delete |    2      2  0.5s
running   1: multi plot, error with non categorical
Used 48.393gb of 68.719gb RAM, time: 0.74892s
running   2: different types without sorting function
Used 48.54gb of 68.719gb RAM, time: 0.62007s
running   3: new random categories, interactive
Used 49.402gb of 68.719gb RAM, time: 0.11623s
running  44: Voxel - gap attribute
Used 49.401gb of 68.719gb RAM, time: 0.0846s
running  45: Plot transform overwrite
Used 49.426gb of 68.719gb RAM, time: 1.46153s
running  46: uv_transform
Used 49.479gb of 68.719gb RAM, time: 1.7829s
running  47: Scatter with FastPixel
Used 49.471gb of 68.719gb RAM, time: 0.53988s
running  48: Reverse image, heatmap and surface axes
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side y in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 4:-1:1 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side y in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 4:-1:1 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value [4, 3, 2, 1] on side y in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side y in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 4:-1:1 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 1:4 on side y in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value 4:-1:1 on side x in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
┌ Warning: Encountered an `AbstractVector` with value [4, 3, 2, 1] on side y in `convert_arguments` for the `ImageLike` trait.
│         Using an `AbstractVector` to specify one dimension of an `ImageLike` is deprecated because `ImageLike` sides always need exactly two values, start and stop.
│         Use interval notation `start .. stop` or a two-element tuple `(start, stop)` instead.
└ @ Makie ~/.julia/dev/Makie/src/conversions.jl:416
Used 49.524gb of 68.719gb RAM, time: 2.48538s
running  49: heatmap_with_labels
Used 49.503gb of 68.719gb RAM, time: 0.38535s
running  50: 2D text
Used 49.509gb of 68.719gb RAM, time: 0.71005s
running  51: Text rotation
Used 49.511gb of 68.719gb RAM, time: 0.04576s
running  52: single_strings_single_positions
Used 49.533gb of 68.719gb RAM, time: 0.22367s
running  53: multi_strings_multi_positions
Used 49.534gb of 68.719gb RAM, time: 0.35746s
running  54: single_strings_single_positions_justification
Used 49.537gb of 68.719gb RAM, time: 0.43186s
running  55: single and multi boundingboxes
Used 49.528gb of 68.719gb RAM, time: 0.21504s
running  56: 3D text
Used 49.528gb of 68.719gb RAM, time: 0.36148s
running  57: empty_lines
Used 49.528gb of 68.719gb RAM, time: 0.036s
running  58: Text offset
Used 49.575gb of 68.719gb RAM, time: 0.57942s
running  59: Log10 text
Used 49.525gb of 68.719gb RAM, time: 0.29681s
running  60: latex strings
Used 49.579gb of 68.719gb RAM, time: 2.76113s
running  61: latex (axis, scene, bbox)
Used 49.599gb of 68.719gb RAM, time: 0.408s
running  62: latex updates
Used 49.605gb of 68.719gb RAM, time: 0.33168s
running  63: update annotation style
Used 49.611gb of 68.719gb RAM, time: 0.35097s
running  64: latex ticks
Used 49.623gb of 68.719gb RAM, time: 0.24243s
running  65: dynamic latex ticks
Used 49.628gb of 68.719gb RAM, time: 0.13547s
running  66: Word Wrapping
Used 49.637gb of 68.719gb RAM, time: 1.2416s
running  67: label type change
Used 49.632gb of 68.719gb RAM, time: 0.18634s
running  68: texture atlas update
Used 49.621gb of 68.719gb RAM, time: 0.13242s
running  69: visible
Used 49.634gb of 68.719gb RAM, time: 0.07625s
running  70: RGB heatmap, heatmap + image overlap
Used 49.642gb of 68.719gb RAM, time: 1.36971s
running  71: heatmap_interpolation
Used 49.671gb of 68.719gb RAM, time: 0.16744s
running  72: poly and colormap
Used 49.642gb of 68.719gb RAM, time: 0.1437s
running  73: quiver
Used 49.676gb of 68.719gb RAM, time: 1.42856s
running  74: Arrows on hemisphere
Used 49.683gb of 68.719gb RAM, time: 0.64255s
running  75: image
Used 49.708gb of 68.719gb RAM, time: 0.60758s
running  76: FEM poly and mesh
Used 49.71gb of 68.719gb RAM, time: 1.87203s
running  77: colored triangle (mesh, poly, 3D) + poly stroke
Used 48.983gb of 68.719gb RAM, time: 1.42446s
running  78: scale_plot
Used 48.981gb of 68.719gb RAM, time: 0.14852s
running  79: Polygons
Used 48.986gb of 68.719gb RAM, time: 0.46496s
running  80: Standard deviation band
Used 48.996gb of 68.719gb RAM, time: 0.79927s
running  81: Band with NaN
Used 49.001gb of 68.719gb RAM, time: 0.3595s
running  82: Streamplot animation
Used 49.101gb of 68.719gb RAM, time: 1.68033s
running  83: Line changing colour
Used 49.106gb of 68.719gb RAM, time: 0.17248s
running  84: streamplot
Used 49.108gb of 68.719gb RAM, time: 0.45085s
running  85: Transforming lines
Used 49.112gb of 68.719gb RAM, time: 0.36731s
running  86: Axes label rotations
Used 49.112gb of 68.719gb RAM, time: 0.16504s
running  87: Colorbar label rotations
Used 49.093gb of 68.719gb RAM, time: 0.12215s
running  88: Errorbars x y low high
Used 49.151gb of 68.719gb RAM, time: 0.58927s
running  89: Errorbars log scale
Used 49.26gb of 68.719gb RAM, time: 0.31303s
running  90: Rangebars x y low high
Used 49.415gb of 68.719gb RAM, time: 0.51188s
running  91: Simple pie charts
Used 49.424gb of 68.719gb RAM, time: 0.93412s
running  92: Pie with Segment-specific Radius
Used 49.432gb of 68.719gb RAM, time: 0.65914s
running  93: Pie Position
Used 49.449gb of 68.719gb RAM, time: 0.71742s
running  94: intersecting polygon
Used 49.448gb of 68.719gb RAM, time: 0.07919s
running  95: Grouped bar
Used 49.356gb of 68.719gb RAM, time: 0.92779s
running  96: space 2D
Used 49.373gb of 68.719gb RAM, time: 0.15571s
running  97: space 2D autolimits
Used 49.374gb of 68.719gb RAM, time: 0.07008s
running  98: Scatter & Text transformations
Used 52.944gb of 68.719gb RAM, time: 0.57298s
running 184: Sphere Mesh
Used 52.943gb of 68.719gb RAM, time: 0.07409s
running 185: Unicode Marker
Used 52.945gb of 68.719gb RAM, time: 0.12904s
running 186: Merged color Mesh
Used 52.945gb of 68.719gb RAM, time: 0.03339s
running 187: Line GIF
Used 52.965gb of 68.719gb RAM, time: 0.79122s
running 188: Surface + wireframe + contour
Used 52.967gb of 68.719gb RAM, time: 0.2049s
WARNING: Method definition (::Type{Main.FitzhughNagumo{T}})(Any, Any, Any, Any) in module Main at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/tests/examples2d.jl:228 overwritten at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/tests/examples3d.jl:498.
WARNING: Method definition (::Type{Main.FitzhughNagumo{T} where T})(T, T, T, T) where {T} in module Main at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/tests/examples2d.jl:228 overwritten at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/tests/examples3d.jl:498.
running 189: Streamplot 3D
Used 52.974gb of 68.719gb RAM, time: 0.71931s
running 190: Depth Shift
Used 53.049gb of 68.719gb RAM, time: 0.21567s
running 266: Complex Lighting - DirectionalLight + specular reflection
Used 53.049gb of 68.719gb RAM, time: 0.0642s
running 267: RectLight
Used 53.049gb of 68.719gb RAM, time: 0.06551s
running 268: Signed Distance Field - FXAA interaction
Used 53.05gb of 68.719gb RAM, time: 0.3546s
[ Info: Downloading reference images
GLMakie/spurious minor tick (#3487).png: Test Failed at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117
  Expression: score <= threshold
   Evaluated: 0.08090270310640335 <= 0.05

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [4] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [5] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [6] test_comparison(scores::Dict{String, Float64}; threshold::Float64)
   @ ReferenceTests ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:115
GLMakie/Scatter with FastPixel.png: Test Failed at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117
  Expression: score <= threshold
   Evaluated: 0.7631859183311462 <= 0.05

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [4] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [5] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [6] test_comparison(scores::Dict{String, Float64}; threshold::Float64)
   @ ReferenceTests ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:115
GLMakie/Heatmap Shader.png: Test Failed at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117
  Expression: score <= threshold
   Evaluated: 1.0685279369354248 <= 0.05

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [4] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [5] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [6] test_comparison(scores::Dict{String, Float64}; threshold::Float64)
   @ ReferenceTests ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:115
GLMakie/fast pixel marker.png: Test Failed at /Users/arthurb/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117
  Expression: score <= threshold
   Evaluated: 1.1819260120391846 <= 0.05

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [4] macro expansion
   @ ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:117 [inlined]
 [5] macro expansion
   @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [6] test_comparison(scores::Dict{String, Float64}; threshold::Float64)
   @ ReferenceTests ~/.julia/dev/Makie/ReferenceTests/src/runtests.jl:115
Test Summary:                                                                                                  | Pass  Fail  Total     Time
Reference Tests                                                                                                |  610     4    614  3m51.5s
  refimages                                                                                                    |  610     4    614  3m51.0s
    Reference tests GLMakie                                                                                    |  299          299  3m25.1s
    Comparison scores                                                                                          |  311     4    315     0.2s
      GLMakie/Colorbar label rotations/step-2.png                                                              |    1            1     0.0s
      GLMakie/Colorbar.png                                                                                     |    1            1     0.0s
      GLMakie/3D Contour with 2D contour slices.png                                                            |    1            1     0.0s
      GLMakie/scatter image markers.png                                                                        |    1            1     0.0s
      GLMakie/quiver.png                                                                                       |    1            1     0.0s
      GLMakie/latex (axis, scene, bbox).png                                                                    |    1            1     0.0s
      GLMakie/FigureSpec/step-2.png                                                                            |    1            1     0.0s
      GLMakie/tricontourf with boundary nodes.png                                                              |    1            1     0.0s
      GLMakie/scale_plot.png                                                                                   |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-6.png                                                           |    1            1     0.0s
      GLMakie/Histogram.png                                                                                    |    1            1     0.0s
      GLMakie/colorscale (scatter).png                                                                         |    1            1     0.0s
      GLMakie/Sampler type/step-2.png                                                                          |    1            1     0.0s
      GLMakie/contour labels 3D.png                                                                            |    1            1     0.0s
      GLMakie/scatter with glow.png                                                                            |    1            1     0.0s
      GLMakie/FigureSpec/step-5.png                                                                            |    1            1     0.0s
      GLMakie/LaTeXStrings in Axis3 plots.png                                                                  |    1            1     0.0s
      GLMakie/Axes label rotations/step-1.png                                                                  |    1            1     0.0s
      GLMakie/Heatmap 3D.png                                                                                   |    1            1     0.0s
      GLMakie/Sampler type/step-1.png                                                                          |    1            1     0.0s
      GLMakie/Textured meshscatter.png                                                                         |    1            1     0.0s
      GLMakie/Arrows on hemisphere.png                                                                         |    1            1     0.0s
      GLMakie/Axis yticksmirrored.png                                                                          |    1            1     0.0s
      GLMakie/Plot transform overwrite.png                                                                     |    1            1     0.0s
      GLMakie/Dynamically adjusting number of particles in a meshscatter.mp4                                   |    1            1     0.0s
      GLMakie/hexbin scale.png                                                                                 |    1            1     0.0s
      GLMakie/hexbin colorrange highclip lowclip.png                                                           |    1            1     0.0s
      GLMakie/Colorbar label rotations/step-3.png                                                              |    1            1     0.0s
      GLMakie/Errorbars x y low high.png                                                                       |    1            1     0.0s
      GLMakie/Polygons.png                                                                                     |    1            1     0.0s
      GLMakie/texture atlas update/step-1.png                                                                  |    1            1     0.0s
      GLMakie/Meshscatter Function.png                                                                         |    1            1     0.0s
      GLMakie/interactive colorscale - hexbin/step-2.png                                                       |    1            1     0.0s
      GLMakie/image scatter.png                                                                                |    1            1     0.0s
      GLMakie/record test formats/step-4.png                                                                   |    1            1     0.0s
      GLMakie/Record Video.mp4                                                                                 |    1            1     0.0s
      GLMakie/interactive colorscale - mesh/step-1.png                                                         |    1            1     0.0s
      GLMakie/Animated surface and wireframe.mp4                                                               |    1            1     0.0s
      GLMakie/tricontourf.png                                                                                  |    1            1     0.0s
      GLMakie/interactive colorscale - mesh/step-2.png                                                         |    1            1     0.0s
      GLMakie/record test formats/step-3.png                                                                   |    1            1     0.0s
      GLMakie/Plotting empty polygons.png                                                                      |    1            1     0.0s
      GLMakie/Axis titles and subtitles.png                                                                    |    1            1     0.0s
      GLMakie/Simple meshscatter.png                                                                           |    1            1     0.0s
      GLMakie/menus.png                                                                                        |    1            1     0.0s
      GLMakie/updating 2d primitives/step-6.png                                                                |    1            1     0.0s
      GLMakie/space 2D.png                                                                                     |    1            1     0.0s
      GLMakie/Voxel - texture mapping.png                                                                      |    1            1     0.0s
      GLMakie/spurious minor tick (#3487).png                                                                  |          1      1     0.2s
      GLMakie/Figure with Blocks.png                                                                           |    1            1     0.0s
      GLMakie/BezierPath marker stroke.png                                                                     |    1            1     0.0s
      GLMakie/colored triangle (mesh, poly, 3D) + poly stroke.png                                              |    1            1     0.0s
      GLMakie/nonlinear colormap.png                                                                           |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-7.png                                                           |    1            1     0.0s
      GLMakie/Image on Surface Sphere.png                                                                      |    1            1     0.0s
      GLMakie/Streamplot 3D.png                                                                                |    1            1     0.0s
      GLMakie/Contour and isosurface with correct depth.png                                                    |    1            1     0.0s
      GLMakie/contour small x or y.png                                                                         |    1            1     0.0s
      GLMakie/marke glyph alignment.png                                                                        |    1            1     0.0s
      GLMakie/minor grid & scales.png                                                                          |    1            1     0.0s
      GLMakie/Voronoiplot with some custom bounding boxes may not contain all data sites.png                   |    1            1     0.0s
      GLMakie/streamplot with func.png                                                                         |    1            1     0.0s
      GLMakie/Clip planes - lines.png                                                                          |    1            1     0.0s
      GLMakie/Surface interpolate attribute.png                                                                |    1            1     0.0s
      GLMakie/contour levels and colors.png                                                                    |    1            1     0.0s
      GLMakie/Don'some_time allow mixing units incorrectly.png                                                 |    1            1     0.0s
      GLMakie/SpecApi in convert_arguments/step-3.png                                                          |    1            1     0.0s
      GLMakie/updating 2d primitives/step-1.png                                                                |    1            1     0.0s
      GLMakie/OldAxis + Surface.png                                                                            |    1            1     0.0s
      GLMakie/Stephist.png                                                                                     |    1            1     0.0s
      GLMakie/Clip planes - voxel.png                                                                          |    1            1     0.0s
      GLMakie/PolarAxis radial shift and clip.png                                                              |    1            1     0.0s
      GLMakie/Orthographic Camera.png                                                                          |    1            1     0.0s
      GLMakie/Grouped bar.png                                                                                  |    1            1     0.0s
      GLMakie/FigureSpec/step-7.png                                                                            |    1            1     0.0s
      GLMakie/Surface + wireframe + contour.png                                                                |    1            1     0.0s
      GLMakie/deletion.png                                                                                     |    1            1     0.0s
      GLMakie/updating 2d primitives/step-5.png                                                                |    1            1     0.0s
      GLMakie/Z-translation within a recipe.png                                                                |    1            1     0.0s
      GLMakie/Voronoiplots with clipped tessellation and unbounded polygons.png                                |    1            1     0.0s
      GLMakie/Word Wrapping.png                                                                                |    1            1     0.0s
      GLMakie/thick arc.png                                                                                    |    1            1     0.0s
      GLMakie/space 2D autolimits.png                                                                          |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-2.png                                                           |    1            1     0.0s
      GLMakie/Transforming lines/step-1.png                                                                    |    1            1     0.0s
      GLMakie/trimspine.png                                                                                    |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-1.png                                                           |    1            1     0.0s
      GLMakie/DateTime Observable.png                                                                          |    1            1     0.0s
      GLMakie/Merged color Mesh.png                                                                            |    1            1     0.0s
      GLMakie/Streamplot animation.mp4                                                                         |    1            1     0.0s
      GLMakie/Text offset.png                                                                                  |    1            1     0.0s
      GLMakie/tricontourf manual vs delaunay.png                                                               |    1            1     0.0s
      GLMakie/hexbin one cellsize.png                                                                          |    1            1     0.0s
      GLMakie/updating 2d primitives/step-2.png                                                                |    1            1     0.0s
      GLMakie/Rangebars x y low high.png                                                                       |    1            1     0.0s
      GLMakie/colorscale (surface).png                                                                         |    1            1     0.0s
      GLMakie/transparent mesh texture.png                                                                     |    1            1     0.0s
      GLMakie/surface + contour3d.png                                                                          |    1            1     0.0s
      GLMakie/Clip planes - CairoMakie overrides.png                                                           |    1            1     0.0s
      GLMakie/contour labels with transform_func.png                                                           |    1            1     0.0s
      GLMakie/colorscale (heatmap).png                                                                         |    1            1     0.0s
      GLMakie/Voronoiplot with a nonlinear transform.png                                                       |    1            1     0.0s
      GLMakie/char marker scenespace.png                                                                       |    1            1     0.0s
      GLMakie/dynamic latex ticks.png                                                                          |    1            1     0.0s
      GLMakie/Axis3 axis reversal.png                                                                          |    1            1     0.0s
      GLMakie/scatter inputs.png                                                                               |    1            1     0.0s
      GLMakie/single_strings_single_positions_justification.png                                                |    1            1     0.0s
      GLMakie/Text rotation.png                                                                                |    1            1     0.0s
      GLMakie/Unequal x and y sizes in surface.png                                                             |    1            1     0.0s
      GLMakie/lines per element colors, OffsetArrays, #3704.png                                                |    1            1     0.0s
      GLMakie/updating 2d primitives/step-3.png                                                                |    1            1     0.0s
      GLMakie/Voronoiplot for a centroidal tessellation with an automatic colormap.png                         |    1            1     0.0s
      GLMakie/Sampler type/step-3.png                                                                          |    1            1     0.0s
      GLMakie/contourf bug #3683.png                                                                           |    1            1     0.0s
      GLMakie/tricontourf extendhigh extendlow.png                                                             |    1            1     0.0s
      GLMakie/lines (some with NaNs) with array colors.png                                                     |    1            1     0.0s
      GLMakie/custom struct, with custom sorting function.png                                                  |    1            1     0.0s
      GLMakie/deletion and observable args.png                                                                 |    1            1     0.0s
      GLMakie/image.png                                                                                        |    1            1     0.0s
      GLMakie/Log scale histogram (barplot).png                                                                |    1            1     0.0s
      GLMakie/Order Independent Transparency.png                                                               |    1            1     0.0s
      GLMakie/Axes label rotations/step-4.png                                                                  |    1            1     0.0s
      GLMakie/interactive colorscale - heatmap/step-3.png                                                      |    1            1     0.0s
      GLMakie/heatmap and image colormap interpolation.png                                                     |    1            1     0.0s
      GLMakie/datashader.png                                                                                   |    1            1     0.0s
      GLMakie/record test.mp4                                                                                  |    1            1     0.0s
      GLMakie/contour labels 2D.png                                                                            |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-5.png                                                           |    1            1     0.0s
      GLMakie/RGB heatmap, heatmap + image overlap.png                                                         |    1            1     0.0s
      GLMakie/Lines from outside.png                                                                           |    1            1     0.0s
      GLMakie/record test formats/step-2.png                                                                   |    1            1     0.0s
      GLMakie/Line GIF.mp4                                                                                     |    1            1     0.0s
      GLMakie/updating 2d primitives/step-4.png                                                                |    1            1     0.0s
      GLMakie/Clip planes - only data space.png                                                                |    1            1     0.0s
      GLMakie/scatter rotations.png                                                                            |    1            1     0.0s
      GLMakie/changing order of categorical values.png                                                         |    1            1     0.0s
      GLMakie/Pie with Segment-specific Radius.png                                                             |    1            1     0.0s
      GLMakie/Scatter & Text transformations.png                                                               |    1            1     0.0s
      GLMakie/update annotation style/step-1.png                                                               |    1            1     0.0s
      GLMakie/lines with gaps.png                                                                              |    1            1     0.0s
      GLMakie/Fluctuation 3D.png                                                                               |    1            1     0.0s
      GLMakie/PolarAxis surface.png                                                                            |    1            1     0.0s
      GLMakie/Matrices of data in surfaces.png                                                                 |    1            1     0.0s
      GLMakie/Violin plots differently scaled.png                                                              |    1            1     0.0s
      GLMakie/single and multi boundingboxes.png                                                               |    1            1     0.0s
      GLMakie/heatmaps & surface.png                                                                           |    1            1     0.0s
      GLMakie/Auto units for observables/step-1.png                                                            |    1            1     0.0s
      GLMakie/Scene backgroundcolor.png                                                                        |    1            1     0.0s
      GLMakie/mesh textured and loaded.png                                                                     |    1            1     0.0s
      GLMakie/Complex Lighting - Ambient + SpotLights + PointLights.png                                        |    1            1     0.0s
      GLMakie/space 3D.png                                                                                     |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-3.png                                                           |    1            1     0.0s
      GLMakie/Latex labels after the fact.png                                                                  |    1            1     0.0s
      GLMakie/single_strings_single_positions.png                                                              |    1            1     0.0s
      GLMakie/Axis limits with translation, scaling and transform_func.png                                     |    1            1     0.0s
      GLMakie/colorscale (lines).png                                                                           |    1            1     0.0s
      GLMakie/SpecApi in convert_arguments/step-1.png                                                          |    1            1     0.0s
      GLMakie/contour with single alpha color.png                                                              |    1            1     0.0s
      GLMakie/bracket vector.png                                                                               |    1            1     0.0s
      GLMakie/Explicit frame rendering.png                                                                     |    1            1     0.0s
      GLMakie/Band with NaN.png                                                                                |    1            1     0.0s
      GLMakie/PolarAxis scatterlines spine.png                                                                 |    1            1     0.0s
      GLMakie/record test formats/step-1.png                                                                   |    1            1     0.0s
      GLMakie/Normals of a Cat.png                                                                             |    1            1     0.0s
      GLMakie/Clip planes - general.png                                                                        |    1            1     0.0s
      GLMakie/Errorbars log scale.png                                                                          |    1            1     0.0s
      GLMakie/new categories, inbetween old values.png                                                         |    1            1     0.0s
      GLMakie/Triplot after adding points and make sure the representative_point_list is correctly updated.png |    1            1     0.0s
      GLMakie/FigureSpec/step-4.png                                                                            |    1            1     0.0s
      GLMakie/space test in transformed axis.png                                                               |    1            1     0.0s
      GLMakie/colorscale (hexbin).png                                                                          |    1            1     0.0s
      GLMakie/2D text.png                                                                                      |    1            1     0.0s
      GLMakie/Standard deviation band.png                                                                      |    1            1     0.0s
      GLMakie/SpecApi in convert_arguments/step-4.png                                                          |    1            1     0.0s
      GLMakie/Scatter with FastPixel.png                                                                       |          1      1     0.0s
      GLMakie/Surface invert_normals.png                                                                       |    1            1     0.0s
      GLMakie/Voronoiplot after adding points.png                                                              |    1            1     0.0s
      GLMakie/Reverse image, heatmap and surface axes.png                                                      |    1            1     0.0s
      GLMakie/label type change.png                                                                            |    1            1     0.0s
      GLMakie/Legend with scalar colors.png                                                                    |    1            1     0.0s
      GLMakie/hexbin bin int.png                                                                               |    1            1     0.0s
      GLMakie/Legend overrides.png                                                                             |    1            1     0.0s
      GLMakie/poly stroke & array input.png                                                                    |    1            1     0.0s
      GLMakie/Log10 text.png                                                                                   |    1            1     0.0s
      GLMakie/BezierPath markers.png                                                                           |    1            1     0.0s
      GLMakie/Array of Images Scatter.png                                                                      |    1            1     0.0s
      GLMakie/bracket scalar.png                                                                               |    1            1     0.0s
      GLMakie/updating 2d primitives/step-8.png                                                                |    1            1     0.0s
      GLMakie/FigureSpec/step-3.png                                                                            |    1            1     0.0s
      GLMakie/Wireframe of mesh, GeoemtryPrimitive and Surface.png                                             |    1            1     0.0s
      GLMakie/Colorbar for recipes.png                                                                         |    1            1     0.0s
      GLMakie/Figure with boxes.png                                                                            |    1            1     0.0s
      GLMakie/Simple pie charts.png                                                                            |    1            1     0.0s
      GLMakie/heatmap_interpolation.png                                                                        |    1            1     0.0s
      GLMakie/Contour3d.png                                                                                    |    1            1     0.0s
      GLMakie/latex strings.png                                                                                |    1            1     0.0s
      GLMakie/Heatmap Shader.png                                                                               |          1      1     0.0s
      GLMakie/Triplot with points, ghost edges, and convex hull.png                                            |    1            1     0.0s
      GLMakie/barplot lowclip highclip nan_color.png                                                           |    1            1     0.0s
      GLMakie/Unicode Marker.png                                                                               |    1            1     0.0s
      GLMakie/different types without sorting function.png                                                     |    1            1     0.0s
      GLMakie/empty_lines.png                                                                                  |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-9.png                                                           |    1            1     0.0s
      GLMakie/updating 2d primitives/step-10.png                                                               |    1            1     0.0s
      GLMakie/Basic Shading.png                                                                                |    1            1     0.0s
      GLMakie/Pie Position.png                                                                                 |    1            1     0.0s
      GLMakie/hexbin threshold.png                                                                             |    1            1     0.0s
      GLMakie/interactive colorscale - heatmap/step-1.png                                                      |    1            1     0.0s
      GLMakie/interactive colorscale - hexbin/step-1.png                                                       |    1            1     0.0s
      GLMakie/Axes label rotations/step-2.png                                                                  |    1            1     0.0s
      GLMakie/multi plot, error with non categorical.png                                                       |    1            1     0.0s
      GLMakie/heatmap log scale, transparent colormap.png                                                      |    1            1     0.0s
      GLMakie/latex ticks.png                                                                                  |    1            1     0.0s
      GLMakie/colormap with specific values.png                                                                |    1            1     0.0s
      GLMakie/tricontourf relative mode.png                                                                    |    1            1     0.0s
      GLMakie/event ticks in record.png                                                                        |    1            1     0.0s
      GLMakie/updating multiple meshes/step-1.png                                                              |    1            1     0.0s
      GLMakie/new random categories, interactive.png                                                           |    1            1     0.0s
      GLMakie/uv_transform.png                                                                                 |    1            1     0.0s
      GLMakie/Clip planes - volume.png                                                                         |    1            1     0.0s
      GLMakie/lines inputs.png                                                                                 |    1            1     0.0s
      GLMakie/polygon markers.png                                                                              |    1            1     0.0s
      GLMakie/updating 2d primitives/step-7.png                                                                |    1            1     0.0s
      GLMakie/combining units, error for numbers.png                                                           |    1            1     0.0s
      GLMakie/hexbin bin tuple.png                                                                             |    1            1     0.0s
      GLMakie/visible.png                                                                                      |    1            1     0.0s
      GLMakie/latex updates/step-2.png                                                                         |    1            1     0.0s
      GLMakie/2D surface with explicit color.png                                                               |    1            1     0.0s
      GLMakie/LaTeXStrings linesegment offsets.png                                                             |    1            1     0.0s
      GLMakie/Axes label rotations/step-3.png                                                                  |    1            1     0.0s
      GLMakie/Time Observable.png                                                                              |    1            1     0.0s
      GLMakie/tricontourf with provided triangulation.png                                                      |    1            1     0.0s
      GLMakie/Categorical xticks yticks.png                                                                    |    1            1     0.0s
      GLMakie/Arrows on Sphere.png                                                                             |    1            1     0.0s
      GLMakie/Surface with image.png                                                                           |    1            1     0.0s
      GLMakie/Auto units for observables/step-2.png                                                            |    1            1     0.0s
      GLMakie/Triplot Showing ghost edges for a triangulation with disjoint boundaries.png                     |    1            1     0.0s
      GLMakie/Time & Date ranges.png                                                                           |    1            1     0.0s
      GLMakie/tricontourf with boundary nodes and edges.png                                                    |    1            1     0.0s
      GLMakie/Surface with NaN points.png                                                                      |    1            1     0.0s
      GLMakie/scatter with stroke.png                                                                          |    1            1     0.0s
      GLMakie/updating 2d primitives/step-9.png                                                                |    1            1     0.0s
      GLMakie/Voronoiplot for a tessellation with a custom bounding box.png                                    |    1            1     0.0s
      GLMakie/Basic units.png                                                                                  |    1            1     0.0s
      GLMakie/Line changing colour.mp4                                                                         |    1            1     0.0s
      GLMakie/Sphere Mesh.png                                                                                  |    1            1     0.0s
      GLMakie/barplot with TeX-ed labels.png                                                                   |    1            1     0.0s
      GLMakie/streamplot.png                                                                                   |    1            1     0.0s
      GLMakie/Miter Limit.png                                                                                  |    1            1     0.0s
      GLMakie/interactive colorscale - heatmap/step-2.png                                                      |    1            1     0.0s
      GLMakie/Rich text.png                                                                                    |    1            1     0.0s
      GLMakie/multipoly with multi strokes.png                                                                 |    1            1     0.0s
      GLMakie/FigureSpec/step-8.png                                                                            |    1            1     0.0s
      GLMakie/lines and linestyles.png                                                                         |    1            1     0.0s
      GLMakie/lines linesegments width test.png                                                                |    1            1     0.0s
      GLMakie/FEM poly and mesh.png                                                                            |    1            1     0.0s
      GLMakie/fast pixel marker.png                                                                            |          1      1     0.0s
      GLMakie/hexbin two cellsizes.png                                                                         |    1            1     0.0s
      GLMakie/Triplot with nonlinear transformation.png                                                        |    1            1     0.0s
      GLMakie/Signed Distance Field - FXAA interaction.png                                                     |    1            1     0.0s
      GLMakie/scene visibility.png                                                                             |    1            1     0.0s
      GLMakie/Axis xticksmirrored.png                                                                          |    1            1     0.0s
      GLMakie/Transforming lines/step-2.png                                                                    |    1            1     0.0s
      GLMakie/Arrows 3D.png                                                                                    |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-4.png                                                           |    1            1     0.0s
      GLMakie/texture atlas update/step-2.png                                                                  |    1            1     0.0s
      GLMakie/colorbuffer for axis.png                                                                         |    1            1     0.0s
      GLMakie/Voxel - lowclip and highclip.png                                                                 |    1            1     0.0s
      GLMakie/scatters.png                                                                                     |    1            1     0.0s
      GLMakie/Complex Lighting - DirectionalLight + specular reflection.png                                    |    1            1     0.0s
      GLMakie/simple volumes.png                                                                               |    1            1     0.0s
      GLMakie/3D text.png                                                                                      |    1            1     0.0s
      GLMakie/Checkbox.png                                                                                     |    1            1     0.0s
      GLMakie/hist(...; gap=0.1).png                                                                           |    1            1     0.0s
      GLMakie/Legend draw order.png                                                                            |    1            1     0.0s
      GLMakie/PolarAxis decorations.png                                                                        |    1            1     0.0s
      GLMakie/Line loops.png                                                                                   |    1            1     0.0s
      GLMakie/Tooltip.png                                                                                      |    1            1     0.0s
      GLMakie/Scalar colors from colormaps.png                                                                 |    1            1     0.0s
      GLMakie/pattern barplot.png                                                                              |    1            1     0.0s
      GLMakie/RectLight.png                                                                                    |    1            1     0.0s
      GLMakie/complex_bezier_markers.png                                                                       |    1            1     0.0s
      GLMakie/basic polygon shapes.png                                                                         |    1            1     0.0s
      GLMakie/redisplay after closing screen.png                                                               |    1            1     0.0s
      GLMakie/Triplot of a constrained triangulation with holes and a custom bounding box.png                  |    1            1     0.0s
      GLMakie/colorscale (poly).png                                                                            |    1            1     0.0s
      GLMakie/Figure and Subplots.png                                                                          |    1            1     0.0s
      GLMakie/scatter rotation.png                                                                             |    1            1     0.0s
      GLMakie/Voxel - gap attribute.png                                                                        |    1            1     0.0s
      GLMakie/FigureSpec/step-1.png                                                                            |    1            1     0.0s
      GLMakie/multi_strings_multi_positions.png                                                                |    1            1     0.0s
      GLMakie/PolarAxis limits.png                                                                             |    1            1     0.0s
      GLMakie/Connected Sphere.png                                                                             |    1            1     0.0s
      GLMakie/Depth Shift.png                                                                                  |    1            1     0.0s
      GLMakie/heatmap_with_labels.png                                                                          |    1            1     0.0s
      GLMakie/meshscatter colors, Axis3.png                                                                    |    1            1     0.0s
      GLMakie/Spy.png                                                                                          |    1            1     0.0s
      GLMakie/Label with text wrapping.png                                                                     |    1            1     0.0s
      GLMakie/marker offset in data space.png                                                                  |    1            1     0.0s
      GLMakie/Miter Joints for line rendering.png                                                              |    1            1     0.0s
      GLMakie/latex updates/step-1.png                                                                         |    1            1     0.0s
      GLMakie/px_per_unit and scalefactor/step-8.png                                                           |    1            1     0.0s
      GLMakie/Date Observable.png                                                                              |    1            1     0.0s
      GLMakie/intersecting polygon.png                                                                         |    1            1     0.0s
      GLMakie/update annotation style/step-2.png                                                               |    1            1     0.0s
      GLMakie/Voxel - colors and colormap.png                                                                  |    1            1     0.0s
      GLMakie/poly and colormap.png                                                                            |    1            1     0.0s
      GLMakie/Linecaps and joinstyles.png                                                                      |    1            1     0.0s
      GLMakie/updating multiple meshes/step-2.png                                                              |    1            1     0.0s
      GLMakie/line color interpolation with clipping.png                                                       |    1            1     0.0s
      GLMakie/SpecApi in convert_arguments/step-2.png                                                          |    1            1     0.0s
      GLMakie/FigureSpec/step-6.png                                                                            |    1            1     0.0s
      GLMakie/Colorbar label rotations/step-1.png                                                              |    1            1     0.0s
      GLMakie/Force Unitful to be rendered as Time.png                                                         |    1            1     0.0s
      GLMakie/Barplot label positions.png                                                                      |    1            1     0.0s
ERROR: LoadError: Some tests did not pass: 610 passed, 4 failed, 0 errored, 0 broken.
in expression starting at /Users/arthurb/.julia/dev/Makie/GLMakie/test/runtests.jl:29
ERROR: Package GLMakie errored during testing

(jl_rwOy5p) pkg>

as a result, there is no scores_table.tsv file saved to GLMakie/test/reference_images, and consequently ReferenceUpdater.serve_update_page_from_dir fails to show anything.

how is it that CI currently passes on github? is there a workaround here from me to add the reference images i need for the aforementioned PR? thanks!

[asinghvi17]

Could it be that your px_per_unit is wrong, and the images are being saved at a higher resolution maybe? I wonder if you get the same results for CairoMakie, which doesn't try to query screen properties.

[bjarthur]

thanks! CairoMakie tests all pass.

curiously though, after running CairoMakie's tests, ReferenceUpdater.serve_update_page_from_dir("CairoMakie/test/reference_images/") still shows no "New images without references" and no "Images with references". and scores_table.tsv is missing:

% tree -L 2 CairoMakie/test/reference_images 
CairoMakie/test/reference_images
├── new_files.txt
├── recorded
│   └── CairoMakie
├── reference
│   ├── CairoMakie
│   ├── GLMakie
│   └── WGLMakie
└── scores.tsv

if i'm reading the ReferenceUpdater/src/local_servers.jl code correctly, "scores_table.tsv" is only generated by serve_update_page not serve_update_page_from_dir, and without it, no images will ever be displayed.

i could just be using ReferenceUpdater incorrectly. its README seems quite outdated.

[ffreyer]

Reference images usually get desynced temporarily when we merge prs that change or add reference images. The workflow there is:

1. update reference images with the new/changed images from the pr
2. restart CI and wait 40min+ for it to finish
3. merge the pr if CI passes

GLMakie/Heatmap Shader.png

Was caused by that

curiously though, after running CairoMakie's tests, ReferenceUpdater.serve_update_page_from_dir("CairoMakie/test/reference_images/") still shows no "New images without references" and no "Images with references". and scores_table.tsv is missing

That's my bad, I updated things in the wrong place so it only gets generated from a CI download. #4444 should fix that.

[bjarthur]

thanks for the info and the PR @ffreyer

the README says "You should be given a choice of different backend workflow runs. Choose one (usually GLMakie when updating reference images) and confirm". i was never prompted as such. did it used to work this way?

[ffreyer]

That's probably from when we had one set of reference images for all backends.