Exclude unit tests from upstream that are known to fail in Brave #7901

mariospr · 2021-02-09T19:16:32Z

This patch disables entire *_unittests.cc files by making them empty
via chromium_src overrides, so that we avoid running any test that
we know are going to fail because of the many ways in which Brave is
different than Chromium upstream.

Note that it's possible that this way of excluding tests is a bit too
agressive (i.e. it's likely that some unit tests inside the excluded
files would pass), but for now it's a good enough initial approach as
it enables us to considerably increase test coverage without having
to maintain patches in a too intrusive way.

As a reference, a this time of this patch's writing (on top of Brave
1.23.30 / Chromium 89.0.4389.86), this are the stats when running
upstream's unit tests on a Linux/Debug build, without this patch:

Total run: 13504 tests
Passed: 11585 tests
Not passed: 1919 tests
- Failed: 182 tests
- Crashed: 1737 tests
- Timed out: 0 tests

With this patch applied, the numbers look like this:

Total run: 10045 tests
Passed: 10045 tests
Not passed: 0 tests

That is, what we have with this patch applied looks as follows:

10045/13504 -> 74.39% of ALL the original tests being run
10045/11585 -> 86.71% of the PASSING TESTS still being run

In other words, we're increasing test coverage in 10045 unit tests
in a relatively clean way (i.e. no complex patching) while "only"
losing 13.29% of the unit tests that would run and pass if we were
not excluding them in such an agressive way.

Fix brave/brave-browser#8376

Submitter Checklist:

There is a ticket for my issue
Used Github auto-closing keywords in the PR description above
Wrote a good PR/commit description
Added appropriate labels (QA/Yes or QA/No; release-notes/include or release-notes/exclude; OS/...) to the associated issue
Checked the PR locally: npm run test -- brave_browser_tests, npm run test -- brave_unit_tests, npm run lint, npm run gn_check, npm run tslint
Ran git rebase master (if needed)
Requested a security/privacy review as needed

Reviewer Checklist:

New files have MPL-2.0 license header
Adequate test coverage exists to prevent regressions
Major classes, functions and non-trivial code blocks are well-commented
Changes in component dependencies are properly reflected in gn
Code follows the style guide
Test plan is specified in PR before merging

After-merge Checklist:

Test Plan:

Run all the enabled unit tests from upstream with npm run test -- unit_tests and check that they all pass

mariospr · 2021-02-09T19:17:22Z

Note: Creating a draft PR for now, since this PR depends on #7899 being merged first, whose only commit is temporarily being included here for convenience.

bridiver · 2021-03-03T23:54:38Z

patches/chrome-browser-media-router-BUILD.gn.patch

+       "providers/openscreen/network_service_quic_packet_writer_unittest.cc",
+     ]
+   }
+  sources -= brave_chromium_unit_test_exclusions_media_router


I don't see where these are defined, but can we just remove them by using chromium_src overrides to make the file empty?

I don't see where these are defined

GitHub is not very nice collapsing the content of the newly added unit_tests_exclusions.gni file, which I think is why you might have overlooked its definition, but it's indeed there. See https://github.com/brave/brave-core/pull/7901/files#diff-663d5215178a6aab5144409eee430f87810538a46612375d5e702fb0af75a8d6R276

but can we just remove them by using chromium_src overrides to make the file empty?

We could do that, but before I do so please be aware that there are many *_unittest.cc files that are being disabled via definitions like this one from unit_tests_exclusions.gni, so there would be quite a bunch of overrides to be made.

In the other hand, emptying those *_unittests.cc files would prevent us from having to maintain the tedious collection of if's I had to use in unit_tests_exclusions.gni to make sure that we only remove a test that has been previously added according to whatever build options we're using, so it's certainly worth a go.

Will try this out now and update the PR accordingly once it's ready for another review.

Thanks!

@bridiver I haven't yet finished doing the change for browser tests (I'm getting some strange failures there I still need to figure out) but this one ready for another look already when you have a moment. Thanks!

PS: Please ignore the first commit, won't be part of this PR once #7899 lands

This patch disables entire *_unittests.cc files by making them empty via chromium_src overrides, so that we avoid running any test that we know are going to fail because of the many ways in which Brave is different than Chromium upstream. Note that it's possible that this way of excluding tests is a bit too agressive (i.e. it's likely that some unit tests inside the excluded files would pass), but for now it's a good enough initial approach as it enables us to considerably increase test coverage without having to maintain patches in a too intrusive way. As a reference, a this time of this patch's writing (on top of Brave 1.23.30 / Chromium 89.0.4389.86), this are the stats when running upstream's unit tests on a Linux/Debug build, without this patch: * Total run: 13504 tests * Passed: 11585 tests * Not passed: 1919 tests - Failed: 182 tests - Crashed: 1737 tests - Timed out: 0 tests With this patch applied, the numbers look like this: * Total run: 10045 tests * Passed: 10045 tests * Not passed: 0 tests That is, what we have with this patch applied looks as follows: * 10045/13504 -> 74.39% of ALL the original tests being run * 10045/11585 -> 86.71% of the PASSING TESTS still being run In other words, we're increasing test coverage in 10045 unit tests in a relatively clean way (i.e. no complex patching) while "only" losing 13.29% of the unit tests that would run and pass if we were not excluding them in such an agressive way. Fix brave/brave-browser#8376

mariospr · 2021-03-12T15:10:30Z

@bridiver Sorry for the extremely long delay, but I had some difficulties addressing similar review comments for #7953 (for browser_tests) which delayed this, but now that I've just pushed changes to that other PR, I feel better about pinging you here again.

Now, as for how things changed now that we're emptying files via overrides compared to the previous approach of removing them via GN files, these are the numbers:

BEFORE addressing your review comments (i.e. removing failing test suites via GN):

Brave/Chromium Version: Brave 1.23.15 / Chromium 89.0.4389.72
Results withOUT any exclusion:
- Total run: 13486 tests
  - Passed: 11625 tests
  - Not passed: 1861 tests
    - Failed: 168 tests
    - Crashed: 1693 tests
Results after excluding failing test suites:
- Total run: 10057 tests (all passing)

In other words:

10057/13486 -> 74.57% of ALL the original tests being run
10057/11625 -> 86.51% of the PASSING TESTS still being run

AFTER addressing your review comments (i.e. removing failing test suites via chromium_src overrides):

Brave/Chromium Version: Brave 1.23.30 / Chromium 89.0.4389.86
Results withOUT any exclusion:
- Total run: 13504 tests
  - Passed: 11585 tests
  - Not passed: 1919 tests
    - Failed: 182 tests
    - Crashed: 1737 tests
      Timed out: 0 tests
Results after excluding failing test suites:
- Total run: 10045 tests (all passing)

In other words:

10045/13504 -> 74.39% of ALL the original tests being run
10045/11585 -> 86.71% of the PASSING TESTS still being run

Thus, results are very similar between the two approaches, although it's true this one of skipping test suites via overrides is simpler and likely easier to maintain than messing with GN files and the different conditions that need to be taken into account not to get build failures (e.g. we can't remove a files via sources -= <file> if not previously added).

@bridiver PTAL and let me know what you think, thanks!

mariospr · 2021-03-22T13:12:05Z

@bridiver Thanks for the review, landing now then.

Similarly to what we've done earlier in the year for other tests [1], this patch disables entire *_unittests.cc files by making them empty via chromium_src overrides, so that we avoid running any test that we know are going to fail because of the many ways in which Brave is different than Chromium upstream. [1] #7901

mariospr added the tests label Feb 9, 2021

mariospr added this to the 1.22.x - Nightly milestone Feb 9, 2021

mariospr requested review from bridiver and mkarolin February 9, 2021 19:16

mariospr self-assigned this Feb 9, 2021

mariospr force-pushed the issues/8376 branch 3 times, most recently from 553197b to 6933502 Compare February 11, 2021 12:24

mariospr mentioned this pull request Feb 11, 2021

Get Chromium browser_tests working brave/brave-browser#8297

Closed

mariospr force-pushed the issues/8376 branch 3 times, most recently from 34e81a7 to 1be944a Compare February 12, 2021 17:44

bridiver reviewed Mar 3, 2021

View reviewed changes

mariospr mentioned this pull request Mar 4, 2021

Exclude browser tests from upstream that are known to fail in Brave #7953

Merged

23 tasks

mariospr force-pushed the issues/8376 branch 3 times, most recently from 7ee89bc to 8b1cf8b Compare March 10, 2021 15:47

mariospr marked this pull request as ready for review March 10, 2021 15:48

mariospr requested a review from a team as a code owner March 10, 2021 15:48

mariospr force-pushed the issues/8376 branch from 8b1cf8b to 858f854 Compare March 10, 2021 15:50

bridiver approved these changes Mar 21, 2021

View reviewed changes

mariospr merged commit 306f2bc into master Mar 22, 2021

mariospr deleted the issues/8376 branch March 22, 2021 13:12

mariospr mentioned this pull request Dec 21, 2021

Get upstream's browser and unit tests running and passing brave/brave-browser#20189

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exclude unit tests from upstream that are known to fail in Brave #7901

Exclude unit tests from upstream that are known to fail in Brave #7901

mariospr commented Feb 9, 2021 •

edited

Loading

mariospr commented Feb 9, 2021

bridiver Mar 3, 2021

mariospr Mar 4, 2021

mariospr Mar 4, 2021 •

edited

Loading

mariospr commented Mar 12, 2021

mariospr commented Mar 22, 2021

Exclude unit tests from upstream that are known to fail in Brave #7901

Exclude unit tests from upstream that are known to fail in Brave #7901

Conversation

mariospr commented Feb 9, 2021 • edited Loading

Submitter Checklist:

Reviewer Checklist:

After-merge Checklist:

Test Plan:

mariospr commented Feb 9, 2021

bridiver Mar 3, 2021

Choose a reason for hiding this comment

mariospr Mar 4, 2021

Choose a reason for hiding this comment

mariospr Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

mariospr commented Mar 12, 2021

mariospr commented Mar 22, 2021

mariospr commented Feb 9, 2021 •

edited

Loading

mariospr Mar 4, 2021 •

edited

Loading