Different screenshot result on MacOS/Ubuntu/Windows

[dalapenko]

I've made test on different platforms and saw that references made on one of platform may failed when use verify on other.

In my repository I wrote two test for three platform

Simple material button without hardware render: https://github.com/dalapenko/ShapeableRobolecticBug/blob/87388e1236f8d656456b2d07947fe62ddb29f619/app/src/test/java/tech/dalapenko/shapeablebug/SimpleViewNoHWRednerTest.kt

Shapeable custom view with hardware render: https://github.com/dalapenko/ShapeableRobolecticBug/blob/87388e1236f8d656456b2d07947fe62ddb29f619/app/src/test/java/tech/dalapenko/shapeablebug/ShapeableViewHWRednerTest.kt

and test it on three platforms. I was surprised that MacOS and Ubuntu system has same problem as Windows. On Windows was failed screenshot references of simple material button made on Mac/Ubuntu, but this references will be similar for Mac and ubuntu respective. On Mac was failed screenshot references of shapeable custom view made on Windows/Ubuntu and for Ubuntu/Windows it was across similar.

Material Button compare on MacOS of Windows ref

Shapeable custom view compare on MacOS of Windows ref

Shapeable custom view compare on MacOS of Unbuntu ref

I've understand that root of problem probably in Robolectric lib, but may be this information useful for growth of Roborazzi lib or may be you already know appropriate issue in Robolectric.

As workaround I found just increase changeThreshold

[takahirom]

Thank you for reporting this issue.
We're facing a common challenge in screenshot testing across different OS environments, as noted in a similar issue with Now in Android. Unfortunately, there are no guarantees that screenshots will render identically across platforms due to differences in underlying libraries like Skia and Minikin.

A practical solution would be to adapt your CI environment to record screenshots. This method aligns with what's been implemented in Now in Android, ensuring consistency by using the same environment for both recording and testing screenshots. This approach should help mitigate the discrepancies caused by platform-dependent rendering behaviors.

[takahirom]

I'll add this problem and the suggested solution to the FAQ section in the README to help others who might encounter the same issue.

[timothyfroehlich]

Good timing :) My team was just running into this since a couple of us started working with Macs

[takahirom]

@dalapenko @timothyfroehlich
It seems that if we specify hShift = 1, we could mitigate the difference. Do you have any thoughts on this?
#459

[timothyfroehlich]

Let me try running that on my system. Our screenshots were recorded on Ubuntu and I have a Macbook

[dalapenko]

I will test it after tomorrow on my sample project and publish result this. But if it’s can’t guarantee 100% equaled screenshot on different platform on normal usage I don’t think that needed made this option default valued

[dalapenko]

@takahirom

I tested on my sample project from this issue with two type of view - simple material button and more complicated custom view with shapeable image view. Views is not Compose - simple xml layout.

With hShift = 0 result is:

Simple material button without HW render (robolectric.pixelCopyRenderMode not set as hardware)

image ref os/run os	windows	ubuntu	macos
windows	passed	failed	failed
ubuntu	failed	passed	passed
macos	failed	passed	passed

Complicated shapeable image with HW render (robolectric.pixelCopyRenderMode set as hardware)

image ref os/run os	windows	ubuntu	macos
windows	passed	passed	failed
ubuntu	passed	passed	failed
macos	failed	failed	passed

button_mac_ref_on_windows_hShift_0

button_ubuntu_ref_on_windows_hShift_0

button_windows_ref_on_macos_hShift_0

button_windows_ref_on_ubuntu_hShift_0

complicated_shapeable_mac_ref_on_ubuntu_hShift_0

complicated_shapeable_mac_ref_on_windows_hShift_0

complicated_shapeable_ubuntu_ref_on_macos_hShift_0

complicated_shapeable_windows_ref_on_macos_hShift_0

then i change hShift to 1 and result pleasantly surprised - all tests is passed:

Simple material button without HW render (robolectric.pixelCopyRenderMode not set as hardware)

image ref os/run os	windows	ubuntu	macos
windows	passed	passed	passed
ubuntu	passed	passed	passed
macos	passed	passed	passed

Complicated shapeable image with HW render (robolectric.pixelCopyRenderMode set as hardware)

image ref os/run os	windows	ubuntu	macos
windows	passed	passed	passed
ubuntu	passed	passed	passed
macos	passed	passed	passed

after this tests i'm change opinion and changing default value of hShift property looks like good update for tool

[takahirom]

Thank you for your excellent investigation, dalapenko! Your thorough testing has provided invaluable insights. Based on these impressive results, we're considering making hShift = 1 the new default setting.

[timothyfroehlich]

I finally tried it, after figuring out how to change the setting without needing to modify the roborazzi repo. (Note that the PR numbers are out of order because I had to untangle some git history that I'd botched)

In google/automotive-design-compose#1499 I made two changes: I added the comparator option to set hShift to 1, and I re-recorded my screenshots on my MacBook (M3) and checked those in.

In google/automotive-design-compose#1498 I removed the hShift option, re-recorded (which did not change the screenshots as expected) and uploaded that.

Our GitHub actions are all Ubuntu, so the test results on the PRs are comparing committed Mac screenshots with code built/run on Ubuntu. You can see from our snapshot diff action that there isn't any apparent image diff, but the verify task failed. The same is true for #1498, with the Mac Screenshots and no hShift.

[timothyfroehlich]

Also ran verify on m MacBook ywith hShift=1 on our main branch (so with Linux-generated screenshots) and got numerous failures:

com.android.designcompose.testapp.validation.StateCustomizationsUnitTest > testTextCustomization FAILED
    java.lang.AssertionError: Roborazzi: /Users/froeht/Code/DesignCompose/integration-tests/validation/src/testDebug/roborazzi/StateCustomizationsUnitTest/State-Customizations.png is changed.
    See the compare image at /Users/froeht/Code/DesignCompose/integration-tests/validation/build/outputs/roborazzi/State-Customizations_compare.png
        at com.github.takahirom.roborazzi.RoborazziOptionsKt.getAssertErrorOrNull(RoborazziOptions.kt:226)
        at com.github.takahirom.roborazzi.RoborazziOptionsKt.access$getAssertErrorOrNull(RoborazziOptions.kt:1)
        at com.github.takahirom.roborazzi.RoborazziOptions$CaptureResultReporter$VerifyCaptureResultReporter.report(RoborazziOptions.kt:186)
        at com.github.takahirom.roborazzi.RoborazziOptions$CaptureResultReporter$DefaultCaptureResultReporter.report(RoborazziOptions.kt:150)
        at com.github.takahirom.roborazzi.ProcessOutputImageAndReportKt.processOutputImageAndReport(processOutputImageAndReport.kt:174)
        at com.github.takahirom.roborazzi.RoborazziKt.processOutputImageAndReportWithDefaults(Roborazzi.kt:634)
        at com.github.takahirom.roborazzi.RoborazziKt$captureRoboImage$2.invoke(Roborazzi.kt:283)
        at com.github.takahirom.roborazzi.RoborazziKt$captureRoboImage$2.invoke(Roborazzi.kt:276)
        at com.github.takahirom.roborazzi.RoborazziKt.capture(Roborazzi.kt:614)
        at com.github.takahirom.roborazzi.RoborazziKt.captureRoboImage(Roborazzi.kt:276)
        at com.github.takahirom.roborazzi.RoborazziKt.captureRoboImage(Roborazzi.kt:268)
        at com.github.takahirom.roborazzi.RoborazziKt.captureRoboImage$default(Roborazzi.kt:263)
        at com.android.designcompose.test.internal.DefaultRoborazziRuleKt.captureRootRoboImage(DefaultRoborazziRule.kt:29)
        at com.android.designcompose.testapp.validation.StateCustomizationsUnitTest.testTextCustomization(StateCustomizationsUnitTest.kt:56)

[takahirom]

I'm still not sure what is happening. There may be differences other than anti-aliasing.

228, 160

reference(linux)

new(mac)

[timothyfroehlich]

Ah damn it, that's a new test and the creator didn't add the roborazzi rule. False alarm, redoing the test.

[timothyfroehlich]

@takahirom
Getting back to this: Running verifyRoborazzi on my Macbook, comparing with the committed screenshots from Linux still results in multiple failures with hshift set to 1 for all tests.

Here's a new PR with the screenshots recorded on my Macbook and hShift set to 1. The diffs in the comments will be the GitHub Ubuntu runners comparing their results with my Macbook's recorded screenshots. google/automotive-design-compose#1523

I could also put up a PR with a compare job running on a Mac Arm runner if that would help.