Run tests on either macOS 12 or macOS 13 #5089

vashworth · 2023-10-09T03:09:14Z

In preparation for migrating our fleet to macOS 13, we're updating tests to run on either macOS 12 or macOS 13.

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

vashworth · 2023-10-09T15:10:14Z

.ci/scripts/create_simulator.sh

@@ -11,5 +11,19 @@ readonly DEVICE_NAME=Flutter-iPhone
 readonly DEVICE=com.apple.CoreSimulator.SimDeviceType.iPhone-14
 readonly OS=com.apple.CoreSimulator.SimRuntime.iOS-16-4

+# Delete any existing devices named Flutter-iPhone. Having more than one may
+# cause issues when builds target the device.


For reference, here's an example of this problem in action: https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20custom_package_tests%20master/7040/overview

Unclear if it's a problem new to macOS 13

vashworth · 2023-10-09T15:11:40Z

.ci/targets/ios_platform_tests.yaml

-    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=iPhone 14,OS=latest"]
+    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=Flutter-iPhone,OS=16.4"]
+  - name: boot simulator
+    # Ensure simulator is still booted


While I was testing, sometimes the simulator would shutdown after "native test" and before "drive examples"

See https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac_arm64%20ios_platform_tests_shard_4%20master/7162/overview for example (I added logs to drive example, where you'll see the Flutter-iPhone simulator is shutdown)

That's really weird. Is there any indication of why it's shutting down?

No, the "native test" step shows the simulator is booted and works as expected. There's no logs that indicate it would be triggered to be shutdown. It also doesn't seem to happen every time.

.ci.yaml

vashworth · 2023-10-09T15:17:29Z

.ci/targets/ios_platform_tests.yaml

@@ -21,7 +21,11 @@ tasks:
    args: ["xcode-analyze", "--ios", "--ios-min-version=13.0"]
  - name: native test
    script: script/tool_runner.sh
-    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=iPhone 14,OS=latest"]
+    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=Flutter-iPhone,OS=16.4"]


As we upgrade, there's no guarantee that there will be an iPhone 14 simulator. Better to use the simulator we create in the "create simulator" step (same for the pigeon test_suites.dart).

Pigeon's test_suites is a weird case; we need it to run locally as well as in CI. So we don't want to hard-code it to a simulator that only exists in CI.

Okay, so I updated Pigeon's test_suites to create and delete it's own simulator so that it'll work in CI and locally

FYI couple reasons I think that's the best approach:

Due to new way Simulator runtimes are handled, when you use OS=latest or you omit the OS (which defaults to latest), it may use a newer simulator runtime that's not the default for the Xcode version. For example, if iOS 17 runtime is mounted, it'll use that instead of iOS 16.4 even if you're using Xcode 14.3. So even if there's an "iPhone 14" device with OS 16.4, it won't match against it since it's looking for iOS 17.

There's no guarantee to be a Simulator named "iPhone 14" unless we create it beforehand (both in CI and locally)

Also btw, I ran dart run tool/run_tests.dart in packages/packages/pigeon locally to make sure it passed locally

.ci/scripts/boot_simulator.sh

stuartmorgan · 2023-10-09T15:19:13Z

.ci/scripts/boot_simulator.sh

+
+xcrun simctl boot "$DEVICE_NAME" || :
+echo -e ""
+xcrun simctl list


Do we want to keep this echo?

It's not needed, but I liked it - it puts a newline between the boot output and list output. Just gives it a little separation

stuartmorgan · 2023-10-09T15:20:52Z

.ci/targets/ios_platform_tests.yaml

@@ -21,7 +21,11 @@ tasks:
    args: ["xcode-analyze", "--ios", "--ios-min-version=13.0"]
  - name: native test
    script: script/tool_runner.sh
-    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=iPhone 14,OS=latest"]
+    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=Flutter-iPhone,OS=16.4"]


Pigeon's test_suites is a weird case; we need it to run locally as well as in CI. So we don't want to hard-code it to a simulator that only exists in CI.

stuartmorgan · 2023-10-09T15:21:22Z

.ci/targets/ios_platform_tests.yaml

-    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=iPhone 14,OS=latest"]
+    args: ["native-test", "--ios", "--ios-destination", "platform=iOS Simulator,name=Flutter-iPhone,OS=16.4"]
+  - name: boot simulator
+    # Ensure simulator is still booted


That's really weird. Is there any indication of why it's shutting down?

.ci/scripts/boot_simulator.sh

…ewline

stuartmorgan

LGTM with nit (and with the .ci.yaml os changed to 12-or-13 per your comment).

.ci/scripts/create_simulator.sh

.ci/targets/ios_platform_tests.yaml

flutter/packages@7de08ec...4b483f2 2023-10-10 15619084+vashworth@users.noreply.github.com Run tests on either macOS 12 or macOS 13 (flutter/packages#5089) 2023-10-10 ychris@google.com [google_map_flutter_ios] fix `didBeginDraggingMarker` typo (flutter/packages#5085) 2023-10-09 stuartmorgan@google.com [process] Import the `process` package (flutter/packages#5095) 2023-10-09 hrkadam.92@gmail.com [go_router] Fixes missing state.extra in onException() (flutter/packages#5077) 2023-10-09 49699333+dependabot[bot]@users.noreply.github.com Bump github/codeql-action from 2.22.0 to 2.22.1 (flutter/packages#5094) 2023-10-09 engine-flutter-autoroll@skia.org Roll Flutter from 5207a30 to f52fe4f (18 revisions) (flutter/packages#5093) 2023-10-09 chrislangham@rightnow.org [platform] Example app for platform (flutter/packages#4834) 2023-10-09 49699333+dependabot[bot]@users.noreply.github.com Bump ossf/scorecard-action from 2.2.0 to 2.3.0 (flutter/packages#5091) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

In preparation for migrating our fleet to macOS 13, we're updating tests to run on either macOS 12 or macOS 13.

update test scripts and ci.yaml for macOS 13

e76652d

github-actions bot added the p: pigeon label Oct 9, 2023

vashworth commented Oct 9, 2023

View reviewed changes

.ci.yaml Outdated Show resolved Hide resolved

vashworth marked this pull request as ready for review October 9, 2023 15:12

vashworth requested a review from tarrinneal as a code owner October 9, 2023 15:13

vashworth requested a review from stuartmorgan October 9, 2023 15:13

vashworth commented Oct 9, 2023

View reviewed changes

stuartmorgan reviewed Oct 9, 2023

View reviewed changes

vashworth added 2 commits October 9, 2023 12:38

create and delete simulator for pigeon tests, add some comments and n…

6fb70b7

…ewline

Merge remote-tracking branch 'upstream/main' into macos_12_or_13

eef0513

vashworth requested a review from stuartmorgan October 9, 2023 19:40

stuartmorgan approved these changes Oct 9, 2023

View reviewed changes

.ci/scripts/create_simulator.sh Outdated Show resolved Hide resolved

.ci/targets/ios_platform_tests.yaml Show resolved Hide resolved

fix nits, run on macOS 12 or 13

9d8d5c1

vashworth mentioned this pull request Oct 9, 2023

Update .ci.yaml to use either macOS 12 or 13 flutter/flutter#134736

Closed

17 tasks

tarrinneal approved these changes Oct 9, 2023

View reviewed changes

vashworth added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 10, 2023

auto-submit bot merged commit 4b483f2 into flutter:main Oct 10, 2023

engine-flutter-autoroll mentioned this pull request Oct 10, 2023

Roll Packages from 7de08ecd1cba to 4b483f242936 (8 revisions) flutter/flutter#136258

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Oct 10, 2023

4b483f2 Run tests on either macOS 12 or macOS 13 (flutter/packages#5089)

0d1d0cb

HugoOlthof pushed a commit to moneybird/packages that referenced this pull request Dec 13, 2023

Run tests on either macOS 12 or macOS 13 (flutter#5089)

a40b683

In preparation for migrating our fleet to macOS 13, we're updating tests to run on either macOS 12 or macOS 13.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run tests on either macOS 12 or macOS 13 #5089

Run tests on either macOS 12 or macOS 13 #5089

vashworth commented Oct 9, 2023

vashworth Oct 9, 2023

vashworth Oct 9, 2023

stuartmorgan Oct 9, 2023

vashworth Oct 9, 2023

vashworth Oct 9, 2023

stuartmorgan Oct 9, 2023

vashworth Oct 9, 2023

vashworth Oct 9, 2023

stuartmorgan Oct 9, 2023

vashworth Oct 9, 2023

stuartmorgan Oct 9, 2023

stuartmorgan Oct 9, 2023

stuartmorgan left a comment

Run tests on either macOS 12 or macOS 13 #5089

Run tests on either macOS 12 or macOS 13 #5089

Conversation

vashworth commented Oct 9, 2023

Pre-launch Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuartmorgan left a comment

Choose a reason for hiding this comment