-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Run Playwright tests on real mobile device browsers? #1122
Comments
There is no way to attach to Chromium on Android or MobileSafari on iOS. Playwright aims at running on your desktop and in your CI/CD, targeting device farms is outside its scope. We will emulate device metrics, geolocation, orientation, etc on Chromium on Android and MobileSafari though. Could you share more about your use case? |
We took into account the possibility of moving away from Selenium WebDriver based tools and switching to DevTools API based tools. A really big portion of our tests are running on mobile web browsers, where we tests our own player that makes use of browser related technologies (e.g. what types of medias is able to render, what's the appropriate format, etc) and sends specific tracking events. Our concern is that there will be differences between a real device and an emulated one, as we're able to emulate any device resolution, but only the viewport size is guaranteed to be accurate. |
Yes, there will be
More than the viewport size, but yes, if you need mobile-specific media to be accurate, you need to attach to the device. This is technically feasible, but is outside of the scope for v1. |
This could be useful for us too. We are a fully fledged PWA email app. Testing on actual devices would catch bugs. We have also faced this issue with non-existent media queries on some phones which @Cosmin-Gramada mentioned. Especially older phones. |
Since Playwright is trying to be more testing-friendly, it would be really great to have real mobile device support. More and more users are coming to our websites from mobile devices, it is nearly 50:50 for us. Most of our bugs on mobile devices are not discoverable via simple mobile emulation. It is really important for us to be able to test on real mobile devices, or at least on emulated Android/iOS systems. |
@Filipoliko - thank you for sharing. Can you elaborate on which bugs are not discoverable via emulation? Examples will really help! |
@arjun27 - They are usually some very specific (and kinda weird) use cases, typically on iOS devices. One current bug on our website, that is discoverable only on real iOS device (safari, or chrome) can be currently found on this page. (The fix will be probably released during next week, hopefully you can check it before that time). If you click on Probably most typical use cases (as far as I can think of) are related to some invisible elements overlapping other elements and making it impossible to click on the overlapped element. This behaviour is usually iOS specific, other systems interpret these css rules in a bit different way. |
Thanks @Filipoliko. I just tried running this case with iPhone emulation on Playwright WebKit and I'm able to repro this issue. Chrome mobile emulation is not reproducing it, as you've rightly pointed it out. Also, you're right about an invisible overlapping element (which is why I had to use const { webkit, devices } = require('playwright');
(async() => {
const browser = await webkit.launch({headless: false});
const context = await browser.newContext({...devices['iPhone 11 Pro']});
const page = await context.newPage();
await page.goto('https://www.seznamzpravy.cz/clanek/v-korupcni-kauze-stoka-obvinila-policie-dalsi-27-lidi-111820');
await page.click('[data-dot="ogm-breadcrumb-navigation"] >> text=Domácí', {force: true});
})(); |
Hi, I would like to also give support for this idea. Our use cases requires our Angular site to get input from other apps (e.g. BankID for authentication). So the real flow can only be properly tested on a real device. |
Hi, |
We have already seen enough interest in it, so we will invest into this area. There are several questions about the use cases and the API, so we would appreciate your feedback, the more details the better. I'll post some initial questions below as comments, please upvote them in case they apply! |
Please 👍 if you'd like to Use Playwright with the Chrome Browser (WebView integration is not essential for you). |
Please 👍 if you'd like to Use Playwright against your WebView and can bring WebView into a testable state over adb. |
Please 👍 if you'd like to Use Playwright against AVD (Emulator) as opposed to the real device. |
Please 👍 if you'd like to Use Playwright against the physical device as opposed to AVD (Emulator). |
Please 👍 if Your testing setup requires Android w/ Play Store |
Please 👍 if Your testing setup does not require Android w/ Play Store |
Please 👍 if You intend to run tests in the cloud. |
See https://playwright.dev/docs/api/class-android for experimental support. |
Edit: Please reference to the guide here: https://playwright.dev/docs/api/class-android |
@pavelfeldman that's great! Thanks for the initial implementation efforts. Are there any plans to do the same for iOS Safari? That would be awesome to get the main mobile contenders covered. Also, do you know how/if this could work with @web/test-runner-browserstack (a test runner that supports Playwright)? |
@alejandroiglesias so far as I know BrowserStack now supports Selenium protocol only. |
@vania-pooh BrowserStack also supports Cypress and has an HTTP-based API, and if @web/test-runner-browserstack does what it promises and can be combined with @web/test-runner-playwright, I would imagine that Playwright it should work in BrowserStack... in theory and unless I'm missing something. Alternatively, SauceLabs supports Playwright as it's stated in their docs. |
support on real mobile devices |
Hello everyone, looking at this issue and some other issues (here on github and on playwright slack channel), we can see that people are requesting support for running Playwright tests on cloud mobile devices (similar to the desktop Playwright tests). We have raised a PR and a Feature Request for this. We are happy to collaborate on this. |
I talked to Browserstack and they are currenlty working on the Android part. Its in a private alpha and I hope it will soon be in beta. |
@pavelfeldman ahoy, I understand good things take time and am not asking for a specific timeline or anything, but I wanted to clarify a few things if that's alright:
I tried accessing the beta mobile docs link but has moved (or been removed):
If experimental support has been added, or there is a nightly or dev build with such support, could you provide an up to date link? Thanks all your hard work! |
https://www.browserstack.com/blog/introducing-android-playwright/ in beta |
This link is missed. |
This is possibly a dumb question, but would it be technically possible for Playwright to support the Webdriver protocol? Or is this explicitly a non-goal for the Playwright project? Naively, it seems like supporting Webdriver would unlock lots of possibilities (for example, running on real Safari on OSX) |
I'd like to add to this question. Even if the full capabilities of Playwright are not possible, would it be possible to have partial support for it. One of the things I like about TestCafe is how it supports basically any browser in a tiered structure. It has native support for a number of browsers. But can also run in many others like native mobile browsers, just with some features unavailable. The ability to take a Playwright test and run it on any Webdriver compatible browser would be game changing. Even if it didn't have the full Playwright functionality. e.g. It would be lovely if I could develop a robust set of feature tests in Playwright for the major browsers. Then run a partial set of those as smoke tests in a bunch of other browsers just to check for any browser specific bugs. Without needing to rewrite tests in a completely different framework. |
A Playwright fan's answer: From my understanding, a lot of Playwright's strengths like the efficient auto-waiting are relying on the protocol to be bi-directional. The current Webdriver protocol is too restricting and fundamentally different. |
is there any update on this? |
I would like to know this too. Where I work, we are looking into using Playwright for our test automation, but being able to test on real devices with iOS (e.g. using Browserstack) is a must have for some test cases. |
Please support android and ios native. Playwright is a wonderful tool to test automation and if support this mobile features is a one-tool-all-automation testing! the automation testing world need a tool with all automation phases. |
500+ 👍 |
Not only that, but it's the top most requested feature. I should probably add that for me and other non-TS peeps the problem is even worse, because let's say this gets added to the TS world, it still needs to be ported to other languages. e.g. in the .NET world where I use Playwright we don't even have Android support yet. |
Just want to share a article Playwright vs Cypress: A Comparison published by browserstack. |
It's not really what you want, but the Onslip Automation Library I've mentioned here before recently gained compatibility with |
Is there any way to connect to a device, start a browser (e.g. chromium instance on Android) and run the tests?
If not, what is missing at this point to be able to use it on both Android and iOS?
Note from maintainers:
Please refer to the https://playwright.dev/docs/next/api/class-android for
instructions on how to use Playwright with Chrome for Android, Android
WebViews and Android itself
The text was updated successfully, but these errors were encountered: