[Question] what does "emulating" a mobile device actually mean? How feasible would it be to launch a PlayWright test on Safari in the Xcode device simulator?

[GrayedFox]

Ahoy hoy,

I have read through the emulation docs and also related resources (BrowserStack's article about simulators vs emulators, SauceLabs also has a good one). I am also subbed to the thread about real iOS device support.

What I want to know is, aside from changing the user agent and viewport (and making other mobile related browser/OS property tweaks), what does "emulation" actually mean in the context of a PlayWright test right now? (6/03/2023). For example:

1. Is PlayWright really doing what SL and BS define as emulation, that is "... [PlayWright emulation is a] complete re-implementation of the mobile software written in a machine level assembly language. The Android (SDK) emulator is one such example."? I would be surprised if this is the case - my guess is that it's just duck typing the browser and OS to look and behave like a mobile device as much as possible.

2. From 1, does PlayWright, when specifying an emulated iOS device, run the test inside, say, some sort of mini/bundled XCode Device Simulator?

3. Would installing playwright-webkit on a real or simulated iOS device theoretically allow playwright tests to run on iOS? Basing this on PlayWright using (I assume) the Safari W3C driver which is the same for Safari desktop and mobile.

4. How "close" is the emulation to a real device? Like, do Playwright have any internal stats about how often a test on an emulated device exhibits the same bugs as the same test on a real device (even if this is based on Android)?

That's about it. We are trying to figure out if we, as a team, can afford to just emulate iOS testing for now or if we need to invest some time and resources into writing some tests in Appium (or some other mobile framework) to allow us to test our web app.

[mxschmitt]

1. No we don't. We just emulate viewport, deviceScaleFactor, hasTouch, isMobile. This is usually enough to be pretty close to a real mobile device. You can run your tests on a real Android device (also with an emulator), see here. iOS not yet.

2. No it does not. WebKit runs on the host without any OS/kernel emulation.

3. Apple does not allow having a custom WebKit build on iOS devices, since we don't rely on WebDriver, this means we are not working on iOS as of today. (we need a custom WebKit in order to automate it).

4. We don't have stats about it, but if you run your WebKit tests on macOS, which means the networking stack is the same like on iOS, then you are getting pretty close. For Chromium almost any platform behaves the same.

General rule what we recommend: 99% Playwright tests and 1% real integration tests with a real device for the low likelihood that something breaks or that you want to test a special feature.

[GrayedFox]

Perfect, that answers all my queries and then some, thank you 🤓