Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timepicker helper with added expectation to reduce flakiness #13029

Merged
merged 5 commits into from
Jan 2, 2025

Conversation

mkllnk
Copy link
Member

@mkllnk mkllnk commented Dec 11, 2024

What? Why?

What should we test?

  • Specs only.
  • 100/100 passes locally.
  • 140/140 passed on CI.

Release notes

Changelog Category (reviewers may add a label for the release notes):

  • User facing changes
  • API changes (V0, V1, DFC or Webhook)
  • Technical changes only
  • Feature toggled

The title of the pull request will be included in the release notes.

Dependencies

Documentation updates

@mkllnk mkllnk added the technical changes only These pull requests do not contain user facing changes and are grouped in release notes label Dec 11, 2024
@mkllnk mkllnk self-assigned this Dec 11, 2024
@mkllnk mkllnk force-pushed the flaky-order-cycle-spec branch from 0a6892e to 203ddf7 Compare December 12, 2024 03:24
@mkllnk mkllnk force-pushed the flaky-order-cycle-spec branch 7 times, most recently from 0005079 to 4163b33 Compare December 20, 2024 00:12
@mkllnk mkllnk force-pushed the flaky-order-cycle-spec branch from 4163b33 to c1f8d30 Compare December 20, 2024 00:15
Copy link
Member Author

@mkllnk mkllnk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may still fail if CI is having a bad day and is very slow but then we can just increase the sleep time. It's only a fraction of a second and not noticeable by a human at the moment.

@mkllnk mkllnk marked this pull request as ready for review December 20, 2024 00:33
@dacook dacook self-requested a review January 2, 2025 02:39
# 50ms -> 87% success
# 100ms -> 100% success in 112 runs
# Let's double that to reduce flakiness even further.
sleep 0.2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be great if we could find an expectation for when Flatpickr has updated. But in the meantime this is a very reasonable fix!

@dacook
Copy link
Member

dacook commented Jan 2, 2025

Thank for fixing this, I hope this is the last we see of it!
I'm curious, how did you run the test 140 times on CI? Did you write a temporary custom script on your branch?

@dacook dacook merged commit 7f3b1c0 into openfoodfoundation:master Jan 2, 2025
51 checks passed
@mkllnk mkllnk deleted the flaky-order-cycle-spec branch January 2, 2025 03:40
@mkllnk
Copy link
Member Author

mkllnk commented Jan 2, 2025

c37bdeb#diff-5c3fa597431eda03ac3339ae6bf7f05e1a50d6fc7333679ec38e21b337cb6721R87

I changed the CI script to run only the one test case. The existing matrix for Knapsack ran that 14 times. Example:

I then created an empty commit locally and pushed to Github. That triggered 14 runs in my repo on the branch and 14 in the OFN repo on the pull request. That's 28 in total. I repeated that a few times and I easily got to 100 hundred.

In the end, I removed the commit above, I could have reverted it for transparency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
technical changes only These pull requests do not contain user facing changes and are grouped in release notes
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

[Flaky] ./spec/system/admin/order_cycles/edit_spec.rb:65
2 participants