Mac CI hangs periodically due to VPN timeout

[williamjallen]

The Mac CI hangs every few weeks due to the VPN timing out after a period of inactivity. This issue first started occurring in February 2024. @BetsyMcPhail and I have met with MacStadium several times to discuss this issue, but their proposed solution does not appear to have resolved the issue.

This issue serves as a place to log instances of these VPN failures in an attempt to track down what specifically is causing them.

[williamjallen]

9/30

[williamjallen]

10/20 (Sunday morning)

[williamjallen]

10/20 (Sunday afternoon)

Note: the series of commands MacStadium gave us to bring the tunnel back up did resolve the issue this time. The tunnel appeared to be functional both before and after the reset but no jobs are making their way through the queue. Waiting until MacStadium performs the upgrade discussed in #21910 (comment) to investigate further. For documentation purposes, I just opened this ticket with MacStadium.

[BetsyMcPhail]

11/4 (Monday), nightly jobs did not launch

The keepalive job ran successfully at 11:47pm
The 12:17 am job never launched

As far as I can tell, no other mac jobs were running at the time

[BetsyMcPhail]

11/20 (Tuesday morning)

keepalive job ran at 8:17am

8:47am job did not launch
experimental job started at 9:18am also did not launch