Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

squelched the Akka.Streams benchmarks with the highest flip rate #2506

Merged

Conversation

Aaronontheweb
Copy link
Member

@Aaronontheweb Aaronontheweb commented Feb 8, 2017

Reviewed the past couple of months worth of data from TeamCity's Flaky Test Report, and opted to disable assertions for the Akka.Streams benchmarks with the highest flip rate. Some of these tests have a fail rate that's approaching ~50%, and I think it's because all of these specs work with the highest number of flows (import 1000 flows,etc...)

I'm proposing that we squelch them so CI fails less often. The cases these benchmarks are asserting are just the largest in the group (except for maybe FusedGraphsBenchmark+RepeatTakeSelectAndAggregate) and there are smaller versions of them all pass with no problem. Alternatively, we can just raise the ceiling on these assertions, since they assume that "if 10 X can be done in N milliseconds, then 100 X can be done in 10 * N milliseconds," which probably doesn't hold true in these cases.

@Danthar
Copy link
Member

Danthar commented Feb 9, 2017

Squelsh them now, and create an seperate issue to raise the ceiling ? That way we can solve the issue now. And have someone with to much time contribute by having him find out what the right timings should be.

(or if we fix it in a different way, e.g. with improvements to nbench or something). Either way, by creating a seperate issue for them, that way it would be visible for more then just the people who frequent the TC status logs :P

@marcpiechura marcpiechura merged commit a56383b into akkadotnet:dev Feb 9, 2017
@marcpiechura
Copy link
Contributor

Maybe we need some more predictable hardware for those benchmarks, would also give us more trust in failing tests. But I'm fine if we disable the biggest ones for now, performance drops in feature changes are hopefully covered with the smaller ones.

@Aaronontheweb Aaronontheweb modified the milestone: 1.1.4 Feb 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants