macOS 13 runner also needs Xcode UI testing capabilities

[erikng]

Description

Someone had reported this for the macOS 12 runner #5410

#5417 solved for it, but unfortunately this logic wasn't updated for the 13 runner https://github.com/actions/runner-images/blob/main/images/macos/provision/configuration/configure-machine.sh#L60

This needs to be updated to be performed on macOS 12 and higher

My CI job is failing due to this

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

https://github.com/macadmins/nudge/actions/runs/5026342911/jobs/9014503504

Is it regression?

yes

Expected behavior

xcode ui unit tests should work

Actual behavior

they don't work

Repro steps

Run xcode ui tests on a project, see that they fail.

[erik-bershel]

Hey @erikng!
We'll take a look.

[mikhailkoliada]

Should be fixed

[erikng]

Thanks. I'll revert some of my Ui test skips and test it out.

[drewvolz]

I think I am still seeing macOS 13 time out around automationmodetool with a combination of:

• macOS 13.0 (cannot test on 13.1, odd payment issues are being returned)
• Xcode 14.2.0

Here is a link to the job.

Underlying Error: Timed out while enabling automation mode.

[drewvolz]

Could the underlying issue here be #7531 where macOS 13 runners are having issues running applescript/osascript?

[idigra]

Could the underlying issue here be #7531 where macOS 13 runners are having issues running applescript/osascript?

As far as I can see the root cause is the same as the original bug here and not related to #7531. I added automationmodetool powershell command in my pipeline that is failing too, and got following output:

Automation Mode is disabled.
This device requires user authentication to enable Automation Mode.

While in macos-12 I got:

Automation Mode is disabled.
This device DOES NOT REQUIRE user authentication to enable Automation Mode.

[erik-bershel]

@erikng new image on board. May I ask you to check if the issue still persist?

[idigra]

@erikng new image on board. May I ask you to check if the issue still persist?

It's still failing for me - but it's weird: We have a pipeline with two parallel jobs (for debug and release), the debug job passed while the release job failed.

[erik-bershel]

Hey @idigra!
Can you please check the Runner version? I suspect that the failed job was run with an old image (20230524.1), while the successful one was assigned to the new image (20230611.2).
For example:

[idigra]

Hey @idigra! Can you please check the Runner version? I suspect that the failed job was run with an old image (20230524.1), while the successful one was assigned to the new image (20230611.2). For example:

Yes, that's right. Versions precisely matching the ones you mentioned.
Do we have any deadline that after it we're guaranteed that all available agents are updated?

[erik-bershel]

Hey @idigra! Can you please check the Runner version? I suspect that the failed job was run with an old image (20230524.1), while the successful one was assigned to the new image (20230611.2). For example:

Yes, that's right. Versions precisely matching the ones you mentioned. Do we have any deadline that after it we're guaranteed that all available agents are updated?

Due to macOS 13 still being in beta, it's difficult for me to give an exact timeline. It shouldn't take more than a couple of working days, but I'll come back separately and let you know when everything is ready. I'm afraid I may have bothered you with checks too early this time.
cc @erikng @drewvolz

[idigra]

Hey @idigra! Can you please check the Runner version? I suspect that the failed job was run with an old image (20230524.1), while the successful one was assigned to the new image (20230611.2). For example:

Yes, that's right. Versions precisely matching the ones you mentioned. Do we have any deadline that after it we're guaranteed that all available agents are updated?

Due to macOS 13 still being in beta, it's difficult for me to give an exact timeline. It shouldn't take more than a couple of working days, but I'll come back separately and let you know when everything is ready. I'm afraid I may have bothered you with checks too early this time. cc @erikng @drewvolz

Thanks! That's fine, for us it's not urgent. I can wait until we're sure that it's done. Also we're fine with small percentages of agents that are not upgraded, the only implication is that developers will be required to rerun which is a cost we can be tolerated for, as long as it's rare and is going to be solved.

[erikng]

Yes, I'm also waiting on the new image version as I have gone ahead and started doing development on Xcode 15 so the runner would fail the unit tests anyway now as some functionality is only in the new Swift 5.9 frameworks. It appears to only be 29% deployed out, but as soon as it's fully deployed, I can try it.

I will try it now and see if I get the luck of the draw with Image Version: 20230611.2

Edit: Got Version: 20230524.1 so will just keep waiting.

[erikng]

It's deployed to about 90% now so my luck of the draw was much better this morning. My unit tests are working now and I was able to cut a new pre-release of my application, including the use of Xcode 15.

Thanks!

[idigra]

Same here, tests are passing. Thanks!

[drewvolz]

Passing on my end. Thank you!

[erik-bershel]

@drewvolz @erikng @idigra all the agents are updated at the moment. Please feel free to open new issues in case of else concerns.

[sssuourabh]

@idigra i am still facing the issue.

[idigra]

@idigra i am still facing the issue.

I guess you'd like to tag one of the issues owner rather than me, who only encountered this issue.
But as a preliminary check, I suggest to verify that you're using the same type of agent (MS hosted macOS-13), and to try the automationmodetool check as I mentioned above.

[sssuourabh]

@idigra where in the script i should add automationmodetool in pipeline?

[idigra]

@idigra where in the script i should add automationmodetool in pipeline?

IIRC I added it within a bash script step.

[sssuourabh]

i added this - automationmodetool enable-automationmode-without-authentication but getting an error Enter the password for user 'runner': Authentication failed, incorrect password or not an administrator. Error: Process completed with exit code 1.
@idigra

[idigra]

i added this - automationmodetool enable-automationmode-without-authentication but getting an error Enter the password for user 'runner': Authentication failed, incorrect password or not an administrator. Error: Process completed with exit code 1. @idigra

Sorry, too long ago, I don't remember the details.

[sssuourabh]

@idigra i am still facing the issue.

@erik-bershel

[erik-bershel]

Hey @sssuourabh!
What exactly you are doing or trying to do?

I checked automation-mode:

jobs:
  test:
    strategy:
      matrix:
        os: [ macos-12, macos-13, macos-14 ]
    runs-on: ${{ matrix.os }}

    steps:
    - name: Run command
      run: automationmodetool
    - name: Run command
      run: sudo /usr/sbin/DevToolsSecurity --enable

Got next (the same for all macOS images):

It seems fine. There is a possibility that you trying to use not authorised operation.

[sssuourabh]

I have added run command @erik-bershel

but i get an error -
Testing failed:

Simulator device returned an error for the requested operation.
Test session results, code coverage, and logs:
ABCApp encountered an error (Failed to install or launch the test runner. (Underlying Error: Simulator device returned an error for the requested operation. The request was denied by service delegate (SBMainWorkspace) for reason: NotFound ("Unknown application display identifier com.exampleidentifier.app"). (Underlying Error: The request to open "com.exampleidentifier.app" failed. The request was denied by service delegate (SBMainWorkspace) for reason: NotFound ("Unknown application display identifier com.exampleidentifier.app"). (Underlying Error: The operation couldn’t be completed. Unknown application display identifier com.exampleidentifier.app.))))

[erik-bershel]

Hey @sssuourabh!
This output is not related to the root issue. It looks more like project config issue than automation mode issue. "com.exampleidentifier.app" was not found. It's fair output for misconfigured run.

[sssuourabh]

@erik-bershel Now when i solved that identifier problem then it says this problem.

All other unit tests passes but UI tests failed so it exited with error - Error: Process completed with exit code 65.

[erik-bershel]

@sssuourabh please provide archive with logs and link to the failed run, no matter if private.

[sssuourabh]

I am attaching logs, please have a look.
and thanks for such quick response from .

logs_22566947746.zip

[erik-bershel]

Hey @sssuourabh!
Basing on the logs I may say next.

2024-04-09T09:09:04.4786430Z 2024-04-09 09:09:02.669 xcodebuild[20053:114125]  iOSSimulator: 97C573C1-C829-42F5-8F4A-E13807B80A3B: Failed to launch app with identifier: com.doxel.app and options: {

This indicates that the application failed to run on the simulator with ID "com.doxel.app".

The reasons for this error may be different:

• Build problem: The application may not be built correctly or completely, resulting in a launch error.
• Simulator Problem: Sometimes the simulator may be not configured correctly.
• Test configuration problem: Your tests may be expecting a different configuration that doesn't match the actual application configuration.

To resolve this issue, you may need to follow these steps:

• Try running the app manually on the simulator from Xcode to make sure it launches correctly.
• Check your testing configuration in GitHub Actions to see if your application launch options need to be adjusted or a different configuration may need to be selected.
• Check that the simulator you are trying to run tests on is available and configured correctly.

If you continue to experience problems after checking the above steps, you may need to analyse the simulator logs or build logs in more detail to identify the specific cause of the error.

Separately, I would like to note that firstly I would try to run the same tests in the same way configured local environment. There is always a possibility that your test did not run simply because the base runner did not have enough resources. I can say right away that we have no influence on performance issues. Example of this kind: fastlane/fastlane#21025 (comment)

Unfortunately, the problem is not in the automation itself - this is a simple problem to solve, but is located somewhere between the simulator and your application.

[sssuourabh]

logs_23597673684.zip

hi @erik-bershel can you please check my logs my build is getting succeeded but TEST BUILD FAILED.

[erik-bershel]

Hey @sssuourabh! Please, update your message. There is no logs at all.

[sssuourabh]

Hey @sssuourabh! Please, update your message. There is no logs at all.

can you please check -
logs_23597673684.zip

[erik-bershel]

Hey @sssuourabh!
I see two issues.
One is XCode configuration related: LLVM Profile Error: Failed to write file "default.profraw": Operation not permitted. You should enable Code Coverage for your Test Scheme to avoid it.
Second one is the test suite output: Some model objects have changed since the build was started. Build again to continue. It's a marker of Swift Compiler issue (or transitive compilation error), unfortunately I'm not so good with it to help you. Could be possibly related to your configuration or project/compiler compatibility. I would firstly try to build/test the same project using XCode 15.1 and 15.0 to check if compiler version related issue.

[sssuourabh]

<img width="812" alt="Screenshot 2024-05-18 at 10 24 22 A@erik-bershel rc="https://github.com/actions/runner-images/assets/2918573/d71be61f-7951-4708-84f2-403339b74aa0">
@erik-bershel
it's already ON but still but libraries it fails