Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #890. Remove intermittent failing search tests. #1029

Merged
merged 3 commits into from
Apr 29, 2016
Merged

Conversation

miketaylr
Copy link
Member

It dawned on me that the reason these sometimes fail is because of Search API rate limits (non-authed tests are sending requests from the Travis IP, not the authed webcompat proxy).

If that theory is true, then moving the "search works" to the auth tests should never fail (which is good, because we don't lose test coverage -- sort of).

r? @karlcow

Mike Taylor added 2 commits April 28, 2016 14:14
So if we break search for logged in users, at least we'll find out!
@miketaylr
Copy link
Member Author

Erm.

FAIL: main - Search (auth) - Search works (3577ms)

AssertionError: The search results show up on the page.: expected 'Issue 280: w3.org - desktop site instead of mobile site' to include 'vladvlad'

@karlcow
Copy link
Member

karlcow commented Apr 29, 2016

ok I have 3 failed tests.

capture d ecran 2016-04-29 a 12 09 13

Let see:

× firefox 45.0.2 on MAC - Issues (auth) - Closing and reopening an issue
× firefox 45.0.2 on MAC - Reporting (non-auth) - Problem type validation
× firefox 45.0.2 on MAC - Reporting (non-auth) - URL validation

hmm not related to your modifications at least.
Let me try again.

grmph

capture d ecran 2016-04-29 a 12 15 58

× firefox 45.0.2 on MAC - Issues (auth) - Closing and reopening an issue
AssertionError: Closed state text is displayed: expected 'Ready for Outreach' to equal 'Closed'
× firefox 45.0.2 on MAC - Search (auth) - Search works
AssertionError: The search results show up on the page.: expected 'Issue 369: www.testing-things.com - mobile site is not usable' to include 'vladvlad'

@karlcow
Copy link
Member

karlcow commented Apr 29, 2016

so unstable these tests… Sniff. 💣

@miketaylr
Copy link
Member Author

so unstable these tests… Sniff. 💣

Yeah. It really sucks. :(

@miketaylr
Copy link
Member Author

ok I have 3 failed tests.

And yet Travis likes all the tests: https://travis-ci.org/webcompat/webcompat.com/builds/126474683

Feels like playing Russian roulette. 🔫

@karlcow
Copy link
Member

karlcow commented Apr 29, 2016

ok let's merge. Something to discuss in London or somewhere. In the same bucket of fixtures and Mocking.

@karlcow karlcow merged commit c7c65ff into master Apr 29, 2016
@karlcow
Copy link
Member

karlcow commented Apr 29, 2016

Thanks @miketaylr 💡

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants