Flaky test: CMS.Views.Metadata.VideoList

[timmc-edx]

A/C:

• Follow the Flaky Test Process (2U private link) to remove it and notify the owning team

Details:

We've seen the following transient failure in the js-tests CI check several times today:

 TOTAL: 1 FAILED, 788 SUCCESS


1) Initialize
     CMS.Views.Metadata.VideoList
     Expected 'click.delegateEventsview1622' to equal 'input'.
@http://localhost:9876/base/js/spec/video/transcripts/videolist_spec.js?246bc1162924d294473de293d23bc3161c083dec:9:7951
then/</</<@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3366:30
fire@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3187:11
fireWith@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3317:7     Expected '.action-secondary' to equal '.videolist-settings-item input'.
@http://localhost:9876/base/js/spec/video/transcripts/videolist_spec.js?246bc1162924d294473de293d23bc3161c083dec:9:8027
then/</</<@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3366:30
fire@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3187:11
fireWith@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3317:7

Failing line: https://github.com/openedx/edx-platform/blob/9dac65a35a3b20291edd6f3a83b2ec1ef1c0c37f/cms/static/js/spec/video/transcripts/videolist_spec.js#L228

[timmc-edx]

Another recent run failed on the Render test instead of Initialize:

 1) is rendered without opened extra videos bar
     CMS.Views.Metadata.VideoList Render
     Expected spy closeExtraVideosBar to have been called.
@http://localhost:9876/base/js/spec/video/transcripts/videolist_spec.js?246bc1162924d294473de293d23bc3161c083dec:9:12828
then/</</<@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3366:30
fire@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3187:11
fireWith@http://localhost:9876/base/common/js/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3317:7

[robrap]

Another failure in Initialize: https://github.com/openedx/edx-platform/actions/runs/5508776235/jobs/10040523895

[robrap]

The removal is in this PR: #32697

[robrap]

Notes:

• @bszabo found some possible PRs that could be culprits:
  • fix: videoblockpreview to videoblockdisplay #32673
  • XModule Style Decoupling: Follow-up fixes & refactoring #32592
  • FYI: @kdmccormick
• If there is a loading issue, it is possible that the test is working perfectly, and it has discovered a new flaky experience in production. Would we know what type of JS failures to look for?
• @Syed-Ali-Abbas-Zaidi confirmed that recent linting changes should not have caused this problem. Changes reviewed:
  • https://github.com/Syed-Ali-Abbas-Zaidi/edx-platform/blob/ce35b5bdc158848e6852c084c5d72f727c0815e6/cms/static/js/spec/video/transcripts/videolist_spec.js
  • https://github.com/openedx/edx-platform/blob/484c7eae5eeeda83cb8e8aac58d687eb4fe6f250/cms/static/js/spec/video/transcripts/videolist_spec.js

[kdmccormick]

@robrap @timmc-edx Jansen had two follow-up fixes to #32592:

• fix: videoblockpreview to videoblockdisplay #32673 (which Robert lists above), and
• fix: public video page missing sass #32682

Both fixes involved adding back in CSS that'd accidentally been removed. The flaky error message, "is rendered without opened extra videos bar", sounds to me like something that could happen if CSS is missing.

My optimistic hypothesis is that the test failures were happening on branches which didn't contain both of Jansen's fixes. To test that, I've opened #32719 to see if we can get a green build.

[robrap]

@kdmccormick: I'm pretty sure the test was flaky on branches that did not change commits between a failure and passing. I re-ran the tests myself on one of the failed PRs.

UPDATE:

See:

• https://github.com/openedx/edx-platform/actions/runs/5508776235/attempts/2 (passed)
• https://github.com/openedx/edx-platform/actions/runs/5508776235/attempts/1 (failed)

Both seem to be for: master b528a27

[kdmccormick]

I'm pretty sure the test was flaky on branches that did not change commits between a failure and passing

Of course, unless I am misunderstanding you, that's what makes it "flaky", right?

I was thinking that my PR, before Jansen's fixes, could have caused the test to become flaky (ie, a true-negative when it fails, and a false-postive when it passes).

That said, the build you linked has both of Jansen's commits, so the theory is bunk.

[kdmccormick]

@robrap It looks like first failing build you linked is on a branch that does not include my PR at all : https://github.com/openedx/edx-platform/commits/1db8c9793fa9ff4b58a8710b25a98832acc1880f

Does anyone know whether edx-platform JS tests are run on HEAD or merge(HEAD, master)? In other words, are they run directly against the PR's branch, or are they run against a merge commit between the PR's branch and master?

[robrap]

Sorry @kdmccormick. I misunderstood and thought you were questioning whether it was actually flaky.

I don’t know the answer to your question. However, we do use these failures to stop our deployments. Also, I feel like I’ve gotten workflow failures after merging, even if for some flaky reason. But these are just anecdotes.

[jmbowman]

@robrap With the flaky tests removed, can this issue be closed now since we have a separate (although 2U-private) issue for fixing them?