Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated failing functional tests. #286

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

mateuszzagorski
Copy link
Contributor

@mateuszzagorski mateuszzagorski commented Nov 5, 2021

Tests: Functional tests have been updated and are now working correctly. Fixes: #281.

This issue most likely will result in fixing the: #285 as well.

Copy link
Contributor

@mlewand mlewand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seems to be a race condition in the "Issue should edit the created comment" test case.

I ran the full suite three times out of which it failed once and passed twice.

When failed the error was a following:

  1) Issue
       should edit the created comment:
     Error: No editor found for the root `JSHandle@node`.
      at IssuePage.getEditorByRoot (tests\_pom\githubpage.js:90:10)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async IssuePage.editComment (tests\_pom\pages\commentstimelinepage.js:48:10)
      at async Context.<anonymous> (tests\functional\issue.js:62:18)

Wiki unit test is also randomly failing it passed once per three runs.

Interestingly in both fails error message was different:

  6) Wiki
       should create a new wiki:
     Error: No node found for selector: [name="wiki[name]"]
      at Object.assert (node_modules\puppeteer\lib\cjs\puppeteer\common\assert.js:26:15)
      at DOMWorld.type (node_modules\puppeteer\lib\cjs\puppeteer\common\DOMWorld.js:307:21)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async NewWikiPage.setTitle (tests\_pom\pages\newwikipage.js:28:3)
      at async Context.<anonymous> (tests\functional\wiki.js:27:3)

and the other was:

7) Wiki
       should create a new wiki:

      AssertionError: expected '<p>Typing inside <strong>GitHub Writer</strong>.</p>\n<p>Time stamp: 2021-11-08T10:23:05.463Z.Welcome to the test-project wiki!</p>' to equal '<p>Typing inside <strong>GitHub Writer</strong>.</p>\n<p>Time stamp: 2021-11-08T10:23:05.463Z.</p>'
      + expected - actual

       <p>Typing inside <strong>GitHub Writer</strong>.</p>
      -<p>Time stamp: 2021-11-08T10:23:05.463Z.Welcome to the test-project wiki!</p>
      +<p>Time stamp: 2021-11-08T10:23:05.463Z.</p>

      at Context.<anonymous> (tests\functional\wiki.js:40:44)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)

All five unit tests for pull request are (consistently) failing for me:

Errors are:

1) Pull Request
       should create a new pull request:
     Error: No root element found for the selector `form.js-blob-form`.
      at FileEditPage.getEditorByRoot (tests\_pom\githubpage.js:84:10)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async FileEditPage.getMainEditor (tests\_pom\pages\fileeditpage.js:29:10)
      at async FileEditPage.appendText (tests\_pom\pages\fileeditpage.js:39:18)
      at async Context.<anonymous> (tests\functional\pullrequest.js:27:3)

  2) Pull Request
       should create a new comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:50:27)
      at processImmediate (internal/timers.js:461:21)

  3) Pull Request
       should edit the created comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:69:27)
      at processImmediate (internal/timers.js:461:21)

  4) Pull Request
       should add a code line comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:88:27)
      at processImmediate (internal/timers.js:461:21)

  5) Pull Request
       should add a review comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:107:27)
      at processImmediate (internal/timers.js:461:21)
2) Pull Request
       should create a new pull request:
     Error: No root element found for the selector `form.js-blob-form`.
      at FileEditPage.getEditorByRoot (tests\_pom\githubpage.js:84:10)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async FileEditPage.getMainEditor (tests\_pom\pages\fileeditpage.js:29:10)
      at async FileEditPage.appendText (tests\_pom\pages\fileeditpage.js:39:18)
      at async Context.<anonymous> (tests\functional\pullrequest.js:27:3)

  3) Pull Request
       should create a new comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:50:27)
      at processImmediate (internal/timers.js:461:21)

  4) Pull Request
       should edit the created comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:69:27)
      at processImmediate (internal/timers.js:461:21)

  5) Pull Request
       should add a code line comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:88:27)
      at processImmediate (internal/timers.js:461:21)

  6) Pull Request
       should add a review comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:107:27)
      at processImmediate (internal/timers.js:461:21)
 1) Pull Request
       should create a new pull request:
     Error: No root element found for the selector `form.js-blob-form`.
      at FileEditPage.getEditorByRoot (tests\_pom\githubpage.js:84:10)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async FileEditPage.getMainEditor (tests\_pom\pages\fileeditpage.js:29:10)
      at async FileEditPage.appendText (tests\_pom\pages\fileeditpage.js:39:18)
      at async Context.<anonymous> (tests\functional\pullrequest.js:27:3)

  2) Pull Request
       should create a new comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:50:27)
      at processImmediate (internal/timers.js:461:21)

  3) Pull Request
       should edit the created comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:69:27)
      at processImmediate (internal/timers.js:461:21)

  4) Pull Request
       should add a code line comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:88:27)
      at processImmediate (internal/timers.js:461:21)

  5) Pull Request
       should add a review comment:
     AssertionError: expected { Object (url, needsLogin, ...) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests\functional\pullrequest.js:107:27)
      at processImmediate (internal/timers.js:461:21)

@mateuszzagorski
Copy link
Contributor Author

Observations:

1. When the button ( [role="button"].timeline-comment-action ) is clicked without prior hover the editor can't be created fast enough (that's not a 100% replicable behavior for all added comments) - even on a normal internet connection.

Error in the browser console:

The most common error in the console is:

1) Issue
 	  should edit the created comment without prior hover:
    Error: No editor found for the root `JSHandle@node`.
     at IssuePage.getEditorByRoot (tests/_pom/githubpage.js:97:10)
     at processTicksAndRejections (internal/process/task_queues.js:93:5)
     at async IssuePage.editComment (tests/_pom/pages/commentstimelinepage.js:83:10)
     at async Context.<anonymous> (tests/functional/issue.js:101:18)```

2. Additional checks (added while testing) inside the clickListener in the commenteditor.js file, and editComment function in the  commentstimelinepage.js will confirm the issue described in the point above.

3. Tried to add multiple specific waitForSelector, or waitVisible elements, but it did not resolve the issue.

4. Checking the task-lists:not([disabled]) selector (as it's the one that's initially disabled, and updated automatically when fetching finishes) did not resolve the issue.

Possible fix:

1. Enforce the creation of the commentEditor to happen only after the background fetch has been completed and all the selectors are available in the DOM

2. Optional tests temporary "patch":

actionButton.hover() should not be skipped.
Wait for the fetch that's mentioned in the #285 issue to finish (using browserPage.waitForResponse() and the response.url() should contain either actions_menu or editor_form ) after the hover, add at least 1sec (given my test results 2sec should be optimal including slower connections/throttling) timeout as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Functional tests are not working
2 participants