Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

Intermittent UI test failure - < SmokeTest.editCustomSearchEngineTest > #22256

Closed
sv-ohorvath opened this issue Nov 2, 2021 · 9 comments · Fixed by #23854
Closed

Intermittent UI test failure - < SmokeTest.editCustomSearchEngineTest > #22256

sv-ohorvath opened this issue Nov 2, 2021 · 9 comments · Fixed by #23854
Labels
eng:disabled-test For any test that has been disabled eng:intermittent-test Evil test failures that only happen now-and-then and are a productivity killer eng:ui-test UI Tests 🙅 waiting Issues that are blocked or has dependencies that are not ready

Comments

@sv-ohorvath
Copy link
Contributor

sv-ohorvath commented Nov 2, 2021 •
edited by data-sync-user
Loading

Firebase Test Run:

https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/9223008020099075063/executions/bs.38fc48eea92dfdb7/testcases/1/test-cases

Stacktrace:

java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertTrue(Assert.java:53) at org.mozilla.fenix.ui.robots.SettingsSubMenuSearchRobot.saveEditSearchEngine(SettingsSubMenuSearchRobot.kt:140) at org.mozilla.fenix.ui.SmokeTest$editCustomSearchEngineTest$4.invoke(SmokeTest.kt:571) at org.mozilla.fenix.ui.SmokeTest$editCustomSearchEngineTest$4.invoke(SmokeTest.kt:563) at org.mozilla.fenix.ui.robots.SettingsRobot$Transition.openSearchSubMenu(SettingsRobot.kt:136) at org.mozilla.fenix.ui.SmokeTest.editCustomSearchEngineTest(SmokeTest.kt:563)

Build:

Debug 11/2

Started with #22244

Blocked by #23536

┆Issue is synchronized with this Jira Task
@sv-ohorvath sv-ohorvath added the eng:ui-test UI Tests label Nov 2, 2021
@github-actions github-actions bot added the needs:triage Issue needs triage label Nov 2, 2021
@sv-ohorvath sv-ohorvath added eng:intermittent-test Evil test failures that only happen now-and-then and are a productivity killer and removed needs:triage Issue needs triage labels Nov 2, 2021
@csadilek
Copy link
Contributor

csadilek commented Nov 3, 2021

Happened again in: #22296

@csadilek csadilek mentioned this issue Nov 4, 2021
Merged
@sv-ohorvath
Copy link
Contributor Author

The problem is with the snackbar - even if it's visible the test doesn't recognize it. Same thing happens in #17484 (comment) or in the context menu tests.

@sv-ohorvath
Copy link
Contributor Author

Started failing 2/2 tries on 11/4:
logs
logs

sv-ohorvath pushed a commit that referenced this issue Nov 4, 2021
For #22304 & #22256: Disabled failing visitedUrlHistoryTest & editCus…
248e5bf 
…tomSearchEngineTest tests
@sv-ohorvath sv-ohorvath added the eng:disabled-test For any test that has been disabled label Nov 4, 2021
mergify bot pushed a commit that referenced this issue Nov 4, 2021
@mergify
For #22304 & #22256: Disabled failing visitedUrlHistoryTest & editCus…
4d88c52 
…tomSearchEngineTest tests
@csadilek csadilek mentioned this issue Nov 5, 2021
Merged
3 tasks
AndiAJ added a commit to AndiAJ/fenix that referenced this issue Nov 8, 2021
@AndiAJ
For mozilla-mobile#22256 re-enable editCustomSearchEngineTest UI test
a67aae2
@AndiAJ AndiAJ mentioned this issue Nov 8, 2021
Merged
3 tasks
mergify bot pushed a commit that referenced this issue Nov 9, 2021
@AndiAJ @mergify
For #22256 re-enable editCustomSearchEngineTest UI test
1ade096
@AndiAJ
Copy link
Collaborator

AndiAJ commented Nov 11, 2021 •
edited by sv-ohorvath
Loading

Started failing 2/2 tries on 11/4: logs logs

Re-enabled in #22339

Based on the logs the failures were caused due to a URL request related issue which happened at a certain time.
I//system/bin/netd(1912): gethostby*.getanswer: asked for "www.elefant.ro IN A", got type "RRSIG"

We decided to re-enable the test as it is and if the problem is recurring try a different solution.

@AndiAJ AndiAJ closed this as completed Nov 11, 2021
@AndiAJ AndiAJ removed the eng:disabled-test For any test that has been disabled label Nov 11, 2021
@sv-ohorvath
Copy link
Contributor Author

Let's leave it open for the initial issue here: the snackbar verification.

@sv-ohorvath sv-ohorvath reopened this Nov 11, 2021
@csadilek
Copy link
Contributor

This test failed twice for #23233 which doesn't seem to contain any relevant changes (also passes locally on that branch). Will try again, but we might need to set it to ignore again.

csadilek added a commit to MickeyMoz/fenix that referenced this issue Jan 17, 2022
@csadilek
Issue mozilla-mobile#22256: Ignore intermittent editCustomSearchEngin…
a14ef09 
…eTest
csadilek added a commit to MickeyMoz/fenix that referenced this issue Jan 17, 2022
@csadilek
Issue mozilla-mobile#22256: Ignore intermittent editCustomSearchEngin…
d721b9e 
…eTest
csadilek added a commit to MickeyMoz/fenix that referenced this issue Jan 17, 2022
@csadilek
Issue mozilla-mobile#22256: Ignore intermittent editCustomSearchEngin…
230c05b 
…eTest
csadilek added a commit to MickeyMoz/fenix that referenced this issue Jan 17, 2022
@csadilek
Issue mozilla-mobile#22256: Ignore intermittent editCustomSearchEngin…
52fb889 
…eTest
mergify bot pushed a commit that referenced this issue Jan 18, 2022
@csadilek @mergify
Issue #22256: Ignore intermittent editCustomSearchEngineTest
12a9071
@csadilek
Copy link
Contributor

Re-enabled in #22339

Hi @AndiAJ, I wasn't able to reproduce the failure locally (on different devices) and since it was blocking the A-C Nightly upgrade I set it to ignored again for now. Could you take another look at this test to double-check? Thanks!

@AndiAJ
Copy link
Collaborator

AndiAJ commented Jan 18, 2022

Hi @csadilek ,thanks for disabling it, sure, we'll take a look.

@AndiAJ AndiAJ added the eng:disabled-test For any test that has been disabled label Jan 18, 2022
@jonalmeida jonalmeida mentioned this issue Feb 1, 2022
Merged
jonalmeida pushed a commit that referenced this issue Feb 1, 2022
@csadilek @jonalmeida
Issue #22256: Ignore intermittent editCustomSearchEngineTest
dedcdc8
jonalmeida pushed a commit that referenced this issue Feb 1, 2022
@csadilek @jonalmeida
Issue #22256: Ignore intermittent editCustomSearchEngineTest
3971670
@sv-ohorvath
Copy link
Contributor Author

I found this issue: #23536 which explains the intermittent failures of this test. Setting this as waiting for a bug fix, won't re-enable until then.

@sv-ohorvath sv-ohorvath added the 🙅 waiting Issues that are blocked or has dependencies that are not ready label Feb 2, 2022
@sv-ohorvath sv-ohorvath linked a pull request Feb 22, 2022 that will close this issue
[UI smoke test] For #21002: Ads new search group UI tests #23854
Merged
3 tasks
pkirakosyan pushed a commit to gexsi/user-agent-android that referenced this issue Mar 7, 2022
@pkirakosyan
For mozilla-mobile#22304 & mozilla-mobile#22256: Disabled failing vis…
1b42b5b 
…itedUrlHistoryTest & editCustomSearchEngineTest tests
pkirakosyan pushed a commit to gexsi/user-agent-android that referenced this issue Mar 7, 2022
@AndiAJ @pkirakosyan
For mozilla-mobile#22256 re-enable editCustomSearchEngineTest UI test
669dec0
pkirakosyan pushed a commit to gexsi/user-agent-android that referenced this issue Mar 9, 2022
@csadilek @pkirakosyan
Issue mozilla-mobile#22256: Ignore intermittent editCustomSearchEngin…
05353ac 
…eTest
@mergify mergify bot closed this as completed in #23854 Mar 15, 2022
lillabldv added a commit to karmaSearch/fenix that referenced this issue Mar 26, 2022
@lillabldv
Merge branch 'releases_v97.0.0' into karma_main
c3431fa 
* releases_v97.0.0: (455 commits)
  Automatic version bump CLOSED TREE NO BUG a=release
  Bump version to 97.3.0.
  Update to Android-Components 97.0.13.
  Automatic version bump CLOSED TREE NO BUG a=release
  Update to Android-Components 97.0.12.
  Bump version to 97.2.0.
  Automatic version bump CLOSED TREE NO BUG a=release
  Fix signing format for mozillaonline builds (mozilla-mobile#23603)
  Update to Android-Components 97.0.11.
  Automatic version bump CLOSED TREE NO BUG a=release
  Update to Android-Components 97.0.10. (mozilla-mobile#23519)
  Issue mozilla-mobile#22256: Ignore intermittent editCustomSearchEngineTest
  No issue: Update PWA package name in mainMenuInstallPWATest (mozilla-mobile#23335)
  For mozilla-mobile#23426:Changed expiring perf probes to never expire (mozilla-mobile#23515)
  Bump version to 97.1.0.
  [Uplift] For mozilla-mobile#22870: Renew metrics set to expire in February (mozilla-mobile#23496)
  Strings - app/src/main/res/values-zh-rCN/strings.xml
  Strings - app/src/main/res/values-tr/strings.xml
  Strings - app/src/main/res/values-tg/strings.xml
  Strings - app/src/main/res/values-sk/strings.xml
  ...

# Conflicts:
#	app/metrics.yaml
#	app/src/main/java/org/mozilla/fenix/FeatureFlags.kt
#	app/src/main/java/org/mozilla/fenix/FenixApplication.kt
#	app/src/main/java/org/mozilla/fenix/components/Core.kt
#	app/src/main/java/org/mozilla/fenix/components/toolbar/BrowserToolbarView.kt
#	app/src/main/java/org/mozilla/fenix/home/HomeFragment.kt
#	app/src/main/java/org/mozilla/fenix/home/HomeFragmentStore.kt
#	app/src/main/java/org/mozilla/fenix/home/HomeMenu.kt
#	app/src/main/java/org/mozilla/fenix/home/recentbookmarks/controller/RecentBookmarksController.kt
#	app/src/main/java/org/mozilla/fenix/home/sessioncontrol/SessionControlAdapter.kt
#	app/src/main/java/org/mozilla/fenix/home/sessioncontrol/SessionControlInteractor.kt
#	app/src/main/java/org/mozilla/fenix/home/sessioncontrol/SessionControlView.kt
#	app/src/main/java/org/mozilla/fenix/home/topsites/TopSiteItemViewHolder.kt
#	app/src/main/java/org/mozilla/fenix/theme/FirefoxTheme.kt
#	app/src/main/res/drawable-xhdpi/ic_logo_wordmark_normal.png
#	app/src/main/res/drawable-xhdpi/ic_logo_wordmark_private.png
#	app/src/main/res/drawable-xxhdpi/ic_logo_wordmark_private.png
#	app/src/main/res/drawable-xxxhdpi/ic_logo_wordmark_private.png
#	app/src/main/res/drawable/ic_baidu.png
#	app/src/main/res/drawable/ic_jd.png
#	app/src/main/res/drawable/ic_pdd.png
#	app/src/main/res/drawable/ic_tc.png
#	app/src/main/res/layout/top_site_item.xml
#	app/src/main/res/values-ar/strings.xml
#	app/src/main/res/values-be/strings.xml
#	app/src/main/res/values-bg/strings.xml
#	app/src/main/res/values-bn/strings.xml
#	app/src/main/res/values-ca/strings.xml
#	app/src/main/res/values-co/strings.xml
#	app/src/main/res/values-cs/strings.xml
#	app/src/main/res/values-cy/strings.xml
#	app/src/main/res/values-da/strings.xml
#	app/src/main/res/values-de/strings.xml
#	app/src/main/res/values-dsb/strings.xml
#	app/src/main/res/values-el/strings.xml
#	app/src/main/res/values-en-rCA/strings.xml
#	app/src/main/res/values-en-rGB/strings.xml
#	app/src/main/res/values-es-rAR/strings.xml
#	app/src/main/res/values-es-rCL/strings.xml
#	app/src/main/res/values-es-rES/strings.xml
#	app/src/main/res/values-es-rMX/strings.xml
#	app/src/main/res/values-es/strings.xml
#	app/src/main/res/values-eu/strings.xml
#	app/src/main/res/values-fa/strings.xml
#	app/src/main/res/values-fi/strings.xml
#	app/src/main/res/values-fr/strings.xml
#	app/src/main/res/values-fy-rNL/strings.xml
#	app/src/main/res/values-gd/strings.xml
#	app/src/main/res/values-gn/strings.xml
#	app/src/main/res/values-hi-rIN/strings.xml
#	app/src/main/res/values-hr/strings.xml
#	app/src/main/res/values-hsb/strings.xml
#	app/src/main/res/values-hu/strings.xml
#	app/src/main/res/values-hy-rAM/strings.xml
#	app/src/main/res/values-ia/strings.xml
#	app/src/main/res/values-in/strings.xml
#	app/src/main/res/values-is/strings.xml
#	app/src/main/res/values-it/strings.xml
#	app/src/main/res/values-iw/strings.xml
#	app/src/main/res/values-ja/strings.xml
#	app/src/main/res/values-ka/strings.xml
#	app/src/main/res/values-kab/strings.xml
#	app/src/main/res/values-kk/strings.xml
#	app/src/main/res/values-kmr/strings.xml
#	app/src/main/res/values-ko/strings.xml
#	app/src/main/res/values-lt/strings.xml
#	app/src/main/res/values-nb-rNO/strings.xml
#	app/src/main/res/values-night/colors.xml
#	app/src/main/res/values-nl/strings.xml
#	app/src/main/res/values-nn-rNO/strings.xml
#	app/src/main/res/values-oc/strings.xml
#	app/src/main/res/values-pa-rIN/strings.xml
#	app/src/main/res/values-pl/strings.xml
#	app/src/main/res/values-pt-rBR/strings.xml
#	app/src/main/res/values-pt-rPT/strings.xml
#	app/src/main/res/values-rm/strings.xml
#	app/src/main/res/values-ru/strings.xml
#	app/src/main/res/values-sat/strings.xml
#	app/src/main/res/values-sk/strings.xml
#	app/src/main/res/values-sl/strings.xml
#	app/src/main/res/values-sq/strings.xml
#	app/src/main/res/values-sr/strings.xml
#	app/src/main/res/values-su/strings.xml
#	app/src/main/res/values-sv-rSE/strings.xml
#	app/src/main/res/values-te/strings.xml
#	app/src/main/res/values-tg/strings.xml
#	app/src/main/res/values-th/strings.xml
#	app/src/main/res/values-tr/strings.xml
#	app/src/main/res/values-tt/strings.xml
#	app/src/main/res/values-uk/strings.xml
#	app/src/main/res/values-ur/strings.xml
#	app/src/main/res/values-uz/strings.xml
#	app/src/main/res/values-vi/strings.xml
#	app/src/main/res/values-zh-rCN/strings.xml
#	app/src/main/res/values-zh-rTW/strings.xml
#	app/src/main/res/values/colors.xml
#	app/src/main/res/values/strings.xml
#	app/src/main/res/values/styles.xml
#	app/src/release/res/drawable-hdpi/fenix_search_widget.png
#	app/src/release/res/mipmap-hdpi/ic_launcher.png
#	app/src/release/res/mipmap-xhdpi/ic_launcher.png
#	app/src/release/res/mipmap-xxhdpi/ic_launcher.png
#	app/src/release/res/mipmap-xxxhdpi/ic_launcher.png
#	app/src/test/java/org/mozilla/fenix/components/history/PagedHistoryProviderTest.kt
#	app/src/test/java/org/mozilla/fenix/home/DefaultSessionControlControllerTest.kt
#	app/src/test/java/org/mozilla/fenix/home/SessionControlInteractorTest.kt
#	app/src/test/java/org/mozilla/fenix/home/recentbookmarks/DefaultRecentBookmarksControllerTest.kt
#	app/src/test/java/org/mozilla/fenix/home/recentvisits/interactor/RecentVisitsInteractorTest.kt
#	app/src/test/java/org/mozilla/fenix/home/sessioncontrol/SessionControlViewTest.kt
#	buildSrc/src/main/java/AndroidComponents.kt
#	version.txt
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
eng:disabled-test For any test that has been disabled eng:intermittent-test Evil test failures that only happen now-and-then and are a productivity killer eng:ui-test UI Tests 🙅 waiting Issues that are blocked or has dependencies that are not ready
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[UI smoke test] For #21002: Ads new search group UI tests mozilla-mobile/fenix
3 participants
@csadilek @sv-ohorvath @AndiAJ