Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect Selenium results for beta browsers #951

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Elchi3
Copy link
Member

@Elchi3 Elchi3 commented Dec 13, 2023

I've been manually collecting results on beta browsers but ideally we want to collect results automatically. I think this PR should do that.
This will help to reduce manual BCD PRs that people usually submit during the beta phase of browsers. See also https://github.com/mdn/community-meetings/blob/main/23-12-11/minutes.md#bcd where Ruth asked about this.

@Elchi3
Copy link
Member Author

Elchi3 commented May 31, 2024

In Beta browsers there are sometimes A/B tests which can skew the test results.

For Chrome, I found https://chromium.googlesource.com/chromium-variations/ and so open -a Google\ Chrome\ Beta --args --fake-variations-channel=stable should hopefully disable the variation and make Beta act like a stable browser release. Is it somehow possible to tell Selenium to launch Chrome beta with --fake-variations-channel=stable?
(mdn/browser-compat-data#23207 (comment))

For Firefox, something similar exists, but I haven't done the research yet how to prevent it from happening there.

Copy link
Member

@queengooborg queengooborg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filterVersions is also used in the find-missing-reports script, which notifies us if there is any critical report files missing. We should add a new argument that allows us to specify if we want to include beta releases.

One of my other concerns is that we have no way of tracking if a report came from a beta or stable version of the browser, which could potentially skew our results when running the update-bcd script. It would also make it harder to know if we need to re-run the collector on the stable version of the browser and obtain support results from a stable release. If we could update the filename saved by the Selenium script to clarify it's for a beta release, that would be helpful.

@caugner
Copy link
Contributor

caugner commented Nov 14, 2024

One of my other concerns is that we have no way of tracking if a report came from a beta or stable version of the browser,

See: #1957

PS: Another way would be to amend the result export with the browser release data, i.e. adding a key release after the userAgent key:

diff --git a/scripts/selenium.ts b/scripts/selenium.ts
index 202d1b4c..0ad5f105 100644
--- a/scripts/selenium.ts
+++ b/scripts/selenium.ts
@@ -658,8 +658,9 @@ const run = async (
         filename.replace(browser, `${browser}-beta`);
       }
       log(task, `Downloading ${filename} ...`);
-      const report = await (await fetch(downloadUrl)).buffer();
-      await fs.writeFile(path.join(RESULTS_DIR, filename), report);
+      const report = await (await fetch(downloadUrl)).json();
+      report['release'] = bcdBrowsers[browser].releases[version];
+      await fs.writeFile(path.join(RESULTS_DIR, filename), JSON.stringify(report));
     }
   } finally {
     driver.quit().catch(() => {});

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants