Experiment/selenium logs #1412

cliffmeyers · 2017-09-21T20:06:07Z

Description

Run the ATH in Chrome so we can view the browser's console logs.

Submitter checklist

Link to JIRA ticket in description, if appropriate.
Change is code complete and matches issue description
Appropriate unit or acceptance tests or explanation to why this change has no tests
Reviewer's manual test instructions provided in PR description. See Reviewer's first task below.

Reviewer checklist

Run the changes and verified the change matches the issue description
Reviewed the code
Verified that the appropriate tests have been written or valid explanation given

cliffmeyers · 2017-09-21T20:19:39Z

cc @michaelneale an experiment to use Chrome instead of Firefox so we can view the browser logs from ATH. Doesn't appear to be coming to Firefox any time soon per SeleniumHQ/selenium#1161 and mozilla/geckodriver#284

cliffmeyers · 2017-09-21T21:45:37Z

@michaelneale this ran once, successfully, but fails intermittently with "no such container" errors. Any idea why? Works fine in my local environment but I don't fully grasp how we have Docker configured in CI.

michaelneale · 2017-09-21T22:36:07Z

@cliffmeyers running it again. A glitch in the matrix.

michaelneale · 2017-09-21T22:56:21Z

I see this in logs when this fails:

docker: failed to register layer: ApplyLayer exit status 1 stdout:  stderr: operation not permitted.

when pulling the image, so it is something about that new image that won't work in some environments, but it seems to work on the bare-docker instance.. just a bad image (possibly, but probably something environmental). It seems a bit change to switch to chrome, is tha tthe only way?

given it works on the "bare docker" instance I think I might ask yoann to rework the agents to work that way...

michaelneale · 2017-09-21T23:50:26Z

@cliffmeyers so it will run when it is on a certain agent, but not the bulk. I have asked @ydubreuil to see if we can migrate to the "bare docker" approach, but if there is a way to can dance around this for now... would be good (not sure when he will get to it).

…output

…ing overall event if object structure is unexpected

…rg name - this was broken after GitHub OrgFolder - MBP change

…aved due to a setValue timing issue that shows up in Chrome

cliffmeyers · 2017-10-04T19:59:23Z

Still working a few things on this one...

Fix an issue in GitHubEditorTest where Selenium would click the "Save" button too quickly, before the pipeline had actually loaded, resulting in a hang. Now we hide Cancel + Save until pipeline is loaded.
Fix an issue in Nightwatch tests where saving of classic pipelines in Classic UI was not actually saving the script, due to a focus/timing issue in Chrome.
Fix in an issue in Nightwatch failingStages test where clicking the Rerun button doesn't actually refresh the pipeline graph. This might actually be a bug in the UI or this could be a flaky test that just happens to fail more often in Chrome.
Fix other Nightwatch failures once they are found...

…to happen first; using "waitForLocationContains" seems more robust

michaelneale · 2017-10-10T02:54:53Z

running locally to check sniffs ok, then should be ok - can we get a blessing from @vivek or @kzantow ?

kzantow · 2017-10-10T02:59:00Z

acceptance-tests/src/main/java/io/blueocean/ath/AthModule.java


 public class AthModule extends AbstractModule {
    @Override
    protected void configure() {

-        DesiredCapabilities capability = DesiredCapabilities.firefox();
+        DesiredCapabilities capability = DesiredCapabilities.chrome();


We should really pass this in from tests so we can run on multiple browsers at some point

We also need to take a look at selenium-grid as it's the "selenium way" of firing off multiple test runs in different browsers.

kzantow

LGTM 🔥

michaelneale · 2017-10-10T04:15:28Z

@cliffmeyers I get errors when I run tests locally individually, eg testEditor from intelliJ (see below).

Are you able to run tests locally? one at a time or all together? I am worried about the supportability of this locally if I am seeing errors like this (on master I can run it fine) 🐛

org.openqa.selenium.StaleElementReferenceException: stale element reference: element is not attached to the page document
  (Session info: chrome=61.0.3163.100)
  (Driver info: chromedriver=2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4),platform=Linux 4.9.12-moby x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 165 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/stale_element_reference.html
Build info: version: '2.53.0', revision: '35ae25b1534ae328c771e0856c93e187490ca824', time: '2016-03-15 10:43:46'
System info: host: 'celebrian-4126.local', ip: '192.168.1.14', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '1.8.0_05'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4), userDataDir=/tmp/.org.chromium.Chromium.ETw9NF}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=61.0.3163.100, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, setWindowRect=true, unexpectedAlertBehaviour=}]
Session ID: 4b204ea515f659dc85f233993e5fd163

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:206)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:158)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:678)
	at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:327)
	at org.openqa.selenium.remote.RemoteWebElement.click(RemoteWebElement.java:85)
	at io.blueocean.ath.pages.blue.GithubCreationPage.selectPipelineToCreate(GithubCreationPage.java:106)
	at io.blueocean.ath.pages.blue.GithubCreationPage.completeCreationFlow(GithubCreationPage.java:145)
	at io.blueocean.ath.pages.blue.GithubCreationPage.createPipeline(GithubCreationPage.java:121)
	at io.blueocean.ath.live.GithubEditorTest.testEditor(GithubEditorTest.java:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at io.blueocean.ath.ATHJUnitRunner$1.evaluate(ATHJUnitRunner.java:79)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at io.blueocean.ath.ATHJUnitRunner.runTest(ATHJUnitRunner.java:154)
	at io.blueocean.ath.ATHJUnitRunner.runChild(ATHJUnitRunner.java:139)
	at io.blueocean.ath.ATHJUnitRunner.runChild(ATHJUnitRunner.java:35)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

so not sure if this is a 🐛 - but we have to be able to run it locally, so not sure what this is about or just my environment.

Can we trust chrome to not update itself inside the image as it launches? It seems that I see different things to what CI does

@kzantow's suggestion is to have a flag to choose browsers (but not sure how much of the code change here could apply to ffx)

michaelneale · 2017-10-10T04:24:20Z

perhaps @vivek could try this locally - if testEditor works for him from IntelliJ - it might just be my environment.

need to confirm this runs locally for people

michaelneale

need to confirm it can run discrete tests locally - which doesn't seem to work for me

NicuPascu · 2017-10-10T06:13:38Z

Tested it on my local machine and it works for me, I was able to run individual tests from IntelliJ

cliffmeyers · 2017-10-10T11:02:55Z

@michaelneale yes, I've been running these repeatedly locally from IntelliJ and from command line. Are you able to run Java and nightwatch tests from command line using mvn clean test and the appropriate profile?

cliffmeyers · 2017-10-10T19:23:15Z

These tests are still a bit flaky apparently. I will put a little more time in to see if I can tighten then up before we consider a merge to master.

michaelneale · 2017-10-10T21:12:17Z

@cliffmeyers I tried nightwatch using the nightwatch CLI - it seemed ok in parts. Running from IntelliJ would fail at some point but started ok... can show you if you like.

michaelneale · 2017-10-12T00:10:48Z

ok looks like an interesting single failure (just testEditor).

note the funky screenshot: https://ci.blueocean.io/job/blueocean/job/experiment%252Fselenium-logs/24/artifact/acceptance-tests/target/screenshots/io.blueocean.ath.live.GithubEditorTest_testEditor.png

getting close @cliffmeyers this is awesome!

…le, then click. if we can repro the "can't click on element" error, then this is a good place for retry logic

…t will be a more reliable click target for Selenium

cliffmeyers · 2017-10-12T15:59:56Z

@michaelneale that last screenshot is interesting, I haven't seen that one before. Haven't been able to repro it but I think I have made some progress on this overall.

I was finally able to repro an error we see pretty frequently: "selector is not clickable at point (x, y)". This happens when you try to click an item that's being animated or lives in an animating container. Unfortunately even adding ExceptedConditions.elementToBeClickable doesn't help this because the element has moved out of the way between the point that Selenium finds the element and the point it attempts to click on it.

I found a fix that appears 100% reliable but I think we should actually use a two-phase approach to handle this:

My fix, a utility method that uses a built-in retry when receiving this exception: https://github.com/jenkinsci/blueocean-plugin/pull/1412/files#diff-9d7946f8ccd8a4191146940351b51999 and
Some good ideas that @kzantow has around either 1. adding an implicit wait around REST API fetches or animations, and/or 2. finding a way to disable transitions in ATH mode.

I think the correct answer is "both" because I am not sure we can write cross-cutting logic that handles every case, and I don't see anything bad about retrying a click since this impl logs very explicitly what it's doing:

11:58:48 INFO  - Editing simple pipeline
11:58:50 WARN  - By.xpath: (//a[@class='back-from-sheet'])[2] not clickable: will retry click
11:58:50 INFO  - retry click successful for By.xpath: (//a[@class='back-from-sheet'])[2]
11:58:51 INFO  - Using branch master

If we are in agreement, I'd like to land this in master, let it soak for a few days, then do a separate PR where I go through all Java ATH and change everything to use WaitUtil.click

cliffmeyers · 2017-10-12T16:04:21Z

Just to add a little more background: after writing WaitUtil.click and running it around 10 times locally, I only saw it succeed without the retry once. And in every case, it succeeded on the first retry. My hope is that this will finally stabilize GithubEditorTest.

michaelneale · 2017-10-13T06:11:15Z

@cliffmeyers still not sure - I can't run testEditor locally at all - fails immediately when trying to click.

Also might want to update this to master as a few things have gone in that may help... I may have to show you next week. It is most odd.
I am trying to think what would be different: slower machine, also it is using my personal github org which has a LOT of repos, maybe that makes it more fragile? (just trying to think what is different)

Can @vivek run testEditor locally?

it is ok if it is just me, but if I can't help people debug ATH locally, that is a problem.

All I did was a clean build - ./run.sh --dev and then run testEditor from IntelliJ

	at io.blueocean.ath.pages.blue.GithubCreationPage.selectPipelineToCreate(GithubCreationPage.java:106)

//code: 

        By xpath = By.xpath("//div[contains(@class, 'repo-list')]//div[contains(@class,'List-Item')]//span[text()='"+pipeline+"']");
        wait.until(ExpectedConditions.visibilityOfElementLocated(xpath)).click();

is there it fails. There are no errors in the chrome, and it is an instant failure, no timeout or anything.

org.openqa.selenium.StaleElementReferenceException: stale element reference: element is not attached to the page document
  (Session info: chrome=61.0.3163.100)
  (Driver info: chromedriver=2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4),platform=Linux 4.9.12-moby x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 42 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/stale_element_reference.html

michaelneale · 2017-10-16T01:23:13Z

hrm... I just saw this on another branch, so perhaps it literally is just my machine being annoying @cliffmeyers ;)

not needed

michaelneale · 2017-10-16T03:32:25Z

@cliffmeyers ok I was able to see it pass. It seems something related to timing. I know that chrome on my machine really slams the CPU a lot, maybe that is part of it?

So I guess if it is just me, this is looking pretty good ... we are expecting this to be more reliable?

# Conflicts: # acceptance-tests/src/test/js/failingStages.js

cliffmeyers · 2017-10-18T17:56:46Z

@michaelneale although we discussed your questions offline, will summarize here for posterity: overall I am not sure there is a big increase in stability but it should not be less. I fixed two issues in GithubEditorTest, one that was definitely flaky overall, another that might have been flaky only in Chrome. Overall we do however gain three important benefits:

Anything logged to browser console is now visible in log view, especially useful for "CI only" failures that are hard to repro locally
We have a new click utility method that will retry when clicks fail due to elements moving, especially during animations.
We run in Chrome which is the browser that most of our users have.

cliffmeyers · 2017-10-18T17:57:37Z

Merged latest master and this branch passed again in CI. Going to merge now.

Cliff Meyers added 2 commits September 21, 2017 15:58

use chrome instead of firefox and enable console logs integration

7ace9de

direct Selenium Chrome's console logs to log4j

4002984

cliffmeyers added the work-in-progress label Sep 21, 2017

suppress superfluous log entries

4f0b122

cliffmeyers requested a review from imeredith September 21, 2017 20:17

cliffmeyers mentioned this pull request Sep 21, 2017

Improvement/jenkins 43211 redux markdown cleanup #1406

Closed

7 tasks

Cliff Meyers added 13 commits October 3, 2017 10:57

try to improve error handling / logging code especially for Selenium …

60cca2c

…output

another attempt to make Chrome + Selenium error stack readable

2744998

another attempt to make Chrome + Selenium error stack readable

4013f1b

make the error logging code a bit more explicit and fall back to logg…

f8e73b1

…ing overall event if object structure is unexpected

Merge branch 'master' into experiment/selenium-logs

9d595b2

switch nightwatch config to use chrome

8865161

update the "deletePipeline" logic to delete by repo name instead of o…

af4c8de

…rg name - this was broken after GitHub OrgFolder - MBP change

hide editor's Cancel and Save buttons until the pipeline is loaded

d0f62f0

just remove the browser name altogether from messaging

05d72b9

try to fix an issue in ATH where pipeline scripts were not properly s…

cede23b

…aved due to a setValue timing issue that shows up in Chrome

use the latest Chrome, since the rest of the world will too

ebcf1b5

add some TODOs around areas of test fragility

c9f0bf7

Merge branch 'master' into experiment/selenium-logs

27b37ba

Cliff Meyers added 5 commits October 5, 2017 10:55

wait for Branches tab to be visible before clicking it

48928b3

asserting the location immediately could fail since a redirect needs …

f03aabf

…to happen first; using "waitForLocationContains" seems more robust

wait for Branches tab to be visible before clicking it

00934a3

be more specific about which "replay" button to click

9e30db6

add another note about a flaky assertion / test

b113970

kzantow reviewed Oct 10, 2017

View reviewed changes

kzantow previously approved these changes Oct 10, 2017

View reviewed changes

michaelneale previously requested changes Oct 10, 2017

View reviewed changes

Merge branch 'master' into experiment/selenium-logs

2bfb300

lock version of docker chrome image

f296d07

Cliff Meyers added 3 commits October 12, 2017 10:29

introduce "click" utility methods that wait for element to be clickab…

5226edc

…le, then click. if we can repro the "can't click on element" error, then this is a good place for retry logic

use an anchor instead of span for sheet's back button with the hope i…

d3b5372

…t will be a more reliable click target for Selenium

implement retry logic for click

fe13d06

cliffmeyers mentioned this pull request Oct 12, 2017

[ATH IMPROVEMENT] Allow local Jenkins instance, simplified API #1484

Merged

Merge branch 'master' into experiment/selenium-logs

f4e6174

# Conflicts: # acceptance-tests/src/test/js/failingStages.js

cliffmeyers merged commit 1c3c11e into master Oct 18, 2017

cliffmeyers deleted the experiment/selenium-logs branch October 18, 2017 17:57

cliffmeyers mentioned this pull request Oct 18, 2017

JENKINS-45393: Bitbucket server ATH #1492

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experiment/selenium logs #1412

Experiment/selenium logs #1412

cliffmeyers commented Sep 21, 2017 •

edited

Loading

cliffmeyers commented Sep 21, 2017

cliffmeyers commented Sep 21, 2017 •

edited

Loading

michaelneale commented Sep 21, 2017

michaelneale commented Sep 21, 2017 •

edited

Loading

michaelneale commented Sep 21, 2017

cliffmeyers commented Oct 4, 2017 •

edited

Loading

michaelneale commented Oct 10, 2017

kzantow Oct 10, 2017

cliffmeyers Oct 11, 2017

kzantow left a comment

michaelneale commented Oct 10, 2017 •

edited

Loading

michaelneale commented Oct 10, 2017

michaelneale left a comment

NicuPascu commented Oct 10, 2017

cliffmeyers commented Oct 10, 2017

cliffmeyers commented Oct 10, 2017

michaelneale commented Oct 10, 2017

michaelneale commented Oct 12, 2017

cliffmeyers commented Oct 12, 2017 •

edited

Loading

cliffmeyers commented Oct 12, 2017

michaelneale commented Oct 13, 2017 •

edited

Loading

michaelneale commented Oct 16, 2017

michaelneale commented Oct 16, 2017

cliffmeyers commented Oct 18, 2017

cliffmeyers commented Oct 18, 2017

Experiment/selenium logs #1412

Experiment/selenium logs #1412

Conversation

cliffmeyers commented Sep 21, 2017 • edited Loading

Description

Submitter checklist

Reviewer checklist

cliffmeyers commented Sep 21, 2017

cliffmeyers commented Sep 21, 2017 • edited Loading

michaelneale commented Sep 21, 2017

michaelneale commented Sep 21, 2017 • edited Loading

michaelneale commented Sep 21, 2017

cliffmeyers commented Oct 4, 2017 • edited Loading

michaelneale commented Oct 10, 2017

kzantow Oct 10, 2017

Choose a reason for hiding this comment

cliffmeyers Oct 11, 2017

Choose a reason for hiding this comment

kzantow left a comment

Choose a reason for hiding this comment

michaelneale commented Oct 10, 2017 • edited Loading

michaelneale commented Oct 10, 2017

michaelneale left a comment

Choose a reason for hiding this comment

NicuPascu commented Oct 10, 2017

cliffmeyers commented Oct 10, 2017

cliffmeyers commented Oct 10, 2017

michaelneale commented Oct 10, 2017

michaelneale commented Oct 12, 2017

cliffmeyers commented Oct 12, 2017 • edited Loading

cliffmeyers commented Oct 12, 2017

michaelneale commented Oct 13, 2017 • edited Loading

michaelneale commented Oct 16, 2017

michaelneale commented Oct 16, 2017

cliffmeyers commented Oct 18, 2017

cliffmeyers commented Oct 18, 2017

cliffmeyers commented Sep 21, 2017 •

edited

Loading

cliffmeyers commented Sep 21, 2017 •

edited

Loading

michaelneale commented Sep 21, 2017 •

edited

Loading

cliffmeyers commented Oct 4, 2017 •

edited

Loading

michaelneale commented Oct 10, 2017 •

edited

Loading

cliffmeyers commented Oct 12, 2017 •

edited

Loading

michaelneale commented Oct 13, 2017 •

edited

Loading