Native visual regression testing (take two)

[drewvolz]

• Targets More home screen personalization #6916
• Alternative implementation of WIP native visual regression testing #6920

To see the feature in action, checkout the target PR to view the functionality.

This PR performs an e2e visual regression check by verifying the app is able to successfully set a background image for the app. When the default background image is changed from the initial value to a photo from the simulator's camera roll, swift-snapshot-testing quickly performs a pixel check to verify against an image of the desired outcome.

The e2e test introduced in this PR performs an end to end check for:

• Asserting that the context menu works.
• Ensuring the settings screen opens.
• Guaranteeing an image can be chosen.
• Asserting the new background of the app matches the expected outcome pixel for pixel.

The output file we are comparing against lives at the following path (customizable, currently stored with the tests).

ios/AllAboutOlafUITests/__Snapshots__/CustomBackgroundImageTests/*.heic

Things to note

Swift Package Manager additions

Why SwiftPM? Neither of these packages support Cocoapods.

I've made sure to add these two SwiftPM additions to the AllAboutOlafUITests target and not AllAboutOlaf.

1. Instead of rolling our own regression checking (WIP native visual regression testing #6920), we can use swift-snapshot-testing from pointfree. It's very customizable, well supported, and gives us a lot of control over formats we can diff views in.

2. We are storing the output file as an HEIC instead of a PNG (with SnapshotTestingHEIC) because the HEIC storage format reduces file sizes in comparison to PNG.

Ultimately we will want to store and perform regression checking in a separate repository, but this at least lets us begin performing these types of checks in the codebase today.

Preventing false positives

To avoid issues of

• homescreen icons changing,
• homescreen footer messages randomizing,
• statusbar datetime text changing (solvable via Take screenshots of the app with Detox #6850 (comment)),
• and other values that could throw off this test,

this test runs on the component library version of the background image picker. It changes both the library sample and real background of the app, but if something were to change the e2e flow of the real component, perhaps we wouldn't know immediately.

I feel the compromise of knowing that (i) the data has changed, (ii) the hooks are functioning, and (iii) the image is visually checked may be a good set of tradeoffs in lieu of testing the actual homescreen but that is open for discussion!

Jest vs XCUITest

As described in #6916 (comment), challenges were presented that required us to go native for this test; Apple renders the native image picker view out of the view hierarchy process, which means we do not have access to inspect PHPickerViewController with detox. A native test here-and-there won't hurt, XCUITest doesn't bite!

Reusability

To get this in state where it could be tested and reusable across the component library and settings, a few changes were made across the app.

• testID is now an available prop for <PushButtonCell /> components
• The navigation bar context menu is separated into its own <RightHomeContextMenu /> component

[codecov]

Codecov Report

Merging #6952 (2951411) into drew/background (656dff9) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

Additional details and impacted files

@@                Coverage Diff                 @@
##           drew/background   #6952      +/-   ##
==================================================
- Coverage             8.43%   8.40%   -0.04%     
==================================================
  Files                  304     306       +2     
  Lines                 5158    5178      +20     
  Branches              1389    1392       +3     
==================================================
  Hits                   435     435              
- Misses                4701    4721      +20     
  Partials                22      22              

[drewvolz]

@hawkrives ideas on how to resolve package.resolved for Swift Package Manager in our build and caching steps?

Resolve Package Graph
xcodebuild: error: Could not resolve package dependencies:
  Package.resolved file is corrupted or malformed; fix or delete the file to continue: unsupported schema version 2

[hawkrives]

Does

xcodebuild -workspace ios/AllAboutOlaf.xcworkspace -scheme AllAboutOlaf -configuration Release -destination 'platform=iOS Simulator,name=iPhone 11 Pro,OS=15.0' -derivedDataPath ios/build build

pass if you run it locally?

Edit: because we may need to update the detox command string.

[drewvolz]

I know the pipeline is setup differently but xcodebuild worked fine locally when I built with iOS 16.2 and an iPhone 14 Pro.

That sounds like the problem if v2 package.resolved is invalid; this needs to be created with Xcode 13.

Update: that looked like it did the job to fix the previous error.

[drewvolz]

I think you're still correct. We'll need to add a step for xcodebuild ... test to see the native e2e test on the pipeline.

[drewvolz]

Evaluating this again.

Choosing the detox approach may be better in the end. We can assume the photo picker is out of our control, so if we assume it behaves correctly, we can go the mocked approach of detox and avoid adding native dependencies on SwiftPM and its packages.

Will try again with the approach laid out in #6916 (comment)