infrastructure/ tests don't work with Safari Technology Preview 68

[foolip]

In #13769, STP is being pinned to 67, because STP 68 doesn't seem to work. The configuration was developed in foolip#7, and https://dev.azure.com/foolip/wpt/_build/results?buildId=112&view=logs is a build from when STP 68 was used. The logs contain this over and over:

 0:45.55 DEBUG new state: initializing_browser
 0:45.55 DEBUG Dispatch init
 0:45.56 DEBUG Init called, starting browser and runner
 0:45.56 DEBUG Starting browser with settings {}
 0:45.56 DEBUG Starting WebDriver: /Applications/Safari Technology Preview.app/Contents/MacOS/safaridriver --port=4444
 0:45.56 DEBUG Waiting for WebDriver to become accessible: http://127.0.0.1:4444/
 1:00.95 ERROR WebDriver was not accessible within the timeout:
Traceback (most recent call last):
  File "/Users/vsts/agent/2.141.1/work/1/s/tools/wptrunner/wptrunner/webdriver_server.py", line 73, in _run
    wait_for_service((self.host, self.port))
  File "/Users/vsts/agent/2.141.1/work/1/s/tools/wptrunner/wptrunner/webdriver_server.py", line 265, in wait_for_service
    raise socket.error("Service is unavailable: %s:%i" % addr)
error: Service is unavailable: 127.0.0.1:4444

@burg, are you aware of any trouble with SafariDriver with STP 68?

[burg]

I am not aware of any issues. Is there anything interesting in the Console.app log related to WebDriver? Are you able to launch safaridriver manually from this release?

[foolip]

@burg is there a way to enable logging for SafariDriver? I can run /Applications/Safari\ Technology\ Preview.app/Contents/MacOS/safaridriver -p 4444 locally and curl http://localhost:4444/status suggests that it works, but it's not behaving the same in CI. One thing that could explain it is if the --enable bit has changed, since I've already run that locally long ago, but it's done every time in CI.

Although, in https://dev.azure.com/foolip/safari-ci-starter-kit/_build/results?buildId=103&view=logs I got STP 68 working with a much simpler setup, so it's not that it's entirely broken. I'm not sure it's deterministic, because I feel like I've seen it fail sometimes and sometimes not. Any debugging tips appreciated.

[burg]

You can set defaults write com.apple.WebDriver ProtocolTracingEnabled -bool YES to get protocol spew sent to a log file and to Console.app. However, anything interesting is probably already being logged to Console (i.e., sandbox problem, sudden termination, crashing).

[foolip]

@burg what log file will it get written to?

[burg]

"""
Opt-in to generating diagnostic logs by running the following in Terminal.app:

defaults write com.apple.WebDriver ProtocolTracingEnabled 1

After setting this, safaridriver will create diagnostic logs in ~/Library/Logs/com.apple.WebDriver/. These logs contain all the HTTP traffic between safaridriver and your test so that we can see the exact WebDriver commands that failed. If you have existing logs in that directory, remove them first so that extraneous logging is not included.
"""

[foolip]

Looks like the problem is resolved with STP 70: #14198

Thanks @burg for the debugging tips, although I didn't end up debugging. Sometimes waiting works too :/