Support for Selenium’s logging interface

[graingert]

The logging interface supported by Selenium FirefoxDriver isn’t available in Marionette.

I'd like support for the client.log('browser') call.

[jgraham]

This isn't in the W3C spec at this time, so we are delaying support until the behaviour is well specified. But your request is noted.

[graingert]

Ah I see, is there any way to follow the development of the spec (mailing
list github etc)?

On 18 Oct 2016 10:57, "jgraham" notifications@github.com wrote:

This isn't in the W3C spec at this time, so we are delaying support until
the behaviour is well specified. But your request is noted.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#284 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZQTCR96Pk73MW0q3LzDrFlKXKIz25kks5q1JgEgaJpZM4KZliw
.

[jgraham]

On 18/10/16 11:24, Thomas Grainger wrote:

Ah I see, is there any way to follow the development of the spec (mailing
list github etc)?

github.com/w3c/webdriver or the public-browser-tools-testing@w3.org
mailing list.

[graingert]

Linkifying for other users: https://github.com/w3c/webdriver public-browser-tools-testing@w3.org

[graingert]

Here's the specific issue: w3c/webdriver#406

[ghost]

I'm looking forward to this feature too.

[shepmaster]

I don't suppose that there is some temporary alternative we can use? Verifying that there are no JS errors in our automated tests is a very nice feature.

[lucaspiller]

@shepmaster I've been looking at this today, as we also want to verify that no JS errors are thrown.

My solution probably won't go down too favourably in this repo, but it does the trick: Switch to using Chrome for integration tests, as chromedriver supports that standard Selenium browser logging interface.

[andreastt]

@lucaspiller Chrome uses a standards incompatible version of the logging interface and we have no intention of implementing that.

A brief update on this is that we discussed logging at the last WebDriver working group meeting and I wrote down some quick words here: https://sny.no/2017/12/wdupdate

[whimboo]

Given that someone filed a bug for that, we now have https://bugzilla.mozilla.org/show_bug.cgi?id=1453962.

[yonjah]

@andreastt any news on this one ?
If I understand it correctly the recommendation currently by w3c team is to use the chromedriver w3c/webdriver#710 (comment) . It doesn't look like this has any priority to be entered into the standard w3c/webdriver#406

I was really hopping to have firefox as my main target for automated testing but not having logs means its not going to be practical

[Herst]

@yonjah If I understand it correctly the WebDriver clients themselves use more powerful protocols to communicate with the applications. In the case of ChromeDriver it's CDP and in the case of Firefox it's Marionette, so while a certain feature might not be part of WebDriver it might still be part of the browser-specific control protocol and you can make use it of it in this way.

[andreastt]

@yonjah In fear of derailing this discussion, the WebDriver WG did talk about logging at the previous F2F in Lyon.

The fear the WG has is that the logging API as implemented in Chrome is limited by the pull-push model restriction of HTTP: it builds up a log cache in a ring buffer internally, and then returns the cache when requested to by the client. This runs the risk of missing out on log entries (because of the ring buffer) and is generally speaking a feature not suited to a blocking pull-push transport mechanism such as HTTP/1.1.

Since this issue was filed there has been some interest amongst vendors to allow CDP as an upgrade mechanism in WebDriver, so a large part of the discussion in Lyon centred around what a bidirectional asynchronous protocol might enable in terms of features. An obvious publish-subscribe candidate would be logging.

At the moment I see little hope for geckodriver implementing the Selenium logging API if (a) it is not part of the WebDriver standard, and (b) a log service on top of CDP exists and there is interest in the WG for an upgrade mechanism to that.

[yonjah]

@Herst Could you please explain how can I obtain the network log and console error log when I'm running my tests with selenium and firefox ?

I have a few flaky tests that fail with blank screen (from a screenshot) though page html seem to load. I can only guess there some network or JS error is preventing the app from loading but I can't debug it with no access to the logs. I can switch our initial tests to run with chrome but I was hoping firefox will be the main target for our testing.

@andreastt as long as there is some way to get this logs I guess from a user perspective it doesn't matter that much. But I think it would be useful to make it clear how to get this logs ( what is CDP? how do I use the log service to get this logs ?) on since it seem like the really easy solution is just to switch to chrome, which doesn't help Mozilla or the standard.

[Herst]

@yonjah I am sorry don't know how to do it in Firefox, I just outlined how it might be theoretically possible. I know that it is possible in Chrome through CDP (Chrome Devtools Protocol), a more powerful but also more low-level protocol for programmable interaction with Chrom(e|ium). Clients like Selenium for Python then allow you to execute these commands right away. I have no idea whether the same is possible with Mozilla's equivalent for Firefox, I would also be interested to know so Firefox will be less of a second-grade browser in our CI setup.

BTW, I personally would always test in both browsers. (And maybe even Safari if you want an extra challenge.)

@andreastt I understand the reasoning, and yet "perfect is the enemy of good"… From a perspective of a testing engineer writing automated tests for large-scale web projects is simply better in Chrome because thanks to this feature I can catch potential problems which show up in the error console but not in the tests (e.g. Javascript errors which lead to graphical/interaction glitches but don't affect any of the tested functionality, or missing resources…, and yes I know that former I could catch using pixel testing and latter by looking at traffic e.g. with the help of a proxy, but that would mean a lot of effort for unknown gain).

[yonjah]

Thanks @Herst
It seem like in webdriverjs I can use sendDevToolsCommand to send CDP commands to chrome but I can't see any similar API commands available when using the firefox driver so I assume it is not supported.

I plan to test on a few more browsers once I get all the kinks out of our UI testing in general but it doesn't make much sense to do it now. Also I only need this feature for debugging the tests at the moment, but it makes sense to fail the tests if a the logs contain unexpected errors. If I'll decide to add such a feature to our tests it means they will not be able to run with firefox which is a shame.

I guess I'll go back and try to reproduce the flakiness with chrome for now

[snowymike]

we have to rotate browsers in our testing anyway (ie, firefox, chrome are the 3 we use), and for my team we've just accepted that chrome is the only one we can get meaningful console logs from. fortunately we haven't had any JS errors in our product (yet) that were firefox specific and that couldn't also be figured out from the screenshot and HTML we also save off (in all 3 browsers) when a test fails. but it'd be great if we could get firefox console logs like we can for chrome, someday.

[whimboo]

Please note that with the upcoming Firefox 65 release there will be the new preference devtools.console.stdout.content users can set which will route every Console API call (like info(), warn(), and error()) to stdout. That means the output will end-up in the geckodriver log. It's not perfect but at least a start to get some of the logs.

[hathiphant]

Just for simplicity, isn't it possible to do something simple on HTTP with Server-Sent Events endpoints, without adding the complexity of a BiDi protocol on WebSocket.

When you open a window with WebDriver, you configure optionally log type/level and you obtain in response the URL on geckodriver of an SSE endpoint with events of navigation. This is the only SSE endpoint living as long as window and keeping only history of navigation (like a browser). In the events of navigation, the message contains the URL of each new location in the window and also the URL of an SSE endpoint sending the log events occurring for this page but without warranty to be available more than the time where the page is currently loaded in the window (this does not need more cache of events than devtools currently use when launched on a page).

---

POST http://127.0.0.1:4444/session
{"capabilities", {"alwaysMatch": {"browserName", "firefox", "logs": {"javascript": "error", "css": "warning", "request": "trace"}}}}
responding with:
{"value": {"sessionId": "$sessionId", "navigation": "$navigation_endpoint"}}

---

GET http://127.0.0.1:4444/navigation/$navigation_endpoint
This SSE endpoint is only available when window is open and sends SSE events like:

event: location
data: {"url": "...", "logs": "$pagelog_endpoint"}

---

GET http://127.0.0.1:4444/logs/$pagelog_endpoint
This SSE endpoint is only available when the driven window is on the corresponding page; after that, geckodriver is free to forget logs and to respond HTTP 410 Gone or 404. It sends SSE events of logs, like:

event: request
data: {"level": "trace", "url": "..."}

event: css
data: {"level": "error", "url": "...", "line": ..., "column": ..., "message": "..."}

---

Just my opinion,
Thanks for reading.

[whimboo]

Note that with Selenium 4 there will be support for CDP (Chrome DevTools Protocol). I'm currently working on making geckodriver to offer an opt-in for a websocket URL that can be used with our own implementation of a subset of the CDP APIs (called Remote Protocol. We already have a good amount of APIs (commands and events) implemented, and we want to see that especially the logging events will work. That means it can be used until the WebDriver Bidi support is available by sometime next year.

[PhilippSelenium]

@whimboo Should it already work with selenium 4.0 and geckodriver 30.0?

[whimboo]

Selenium 4.0.0 makes use of the CDP implementation for both Firefox and Chrome. Note that this a temporary solution to allow users to subscribe to log events. Currently we are working on WebDriver BiDi support for log.entryAdded which should be fully available by at latest end of this year.

The Selenium implementation has been built to cause a minimal impact on consumers when the move from CDP to BiDi will happen. So yes, you should be able to get started but note that our CDP implementation for log events is not complete, and won't ever be compared to the WebDriver BiDi standard implementation.

Given that geckodriver will not be used for WebDriver BiDi but Firefox itself will contain the full implementation I do not see a need to further track this issue. As such I'm going to close it now.

I hope you all will enjoy the soon upcoming feature!

[PhilippSelenium]

@whimboo Thanks, but this answer caused more confusion for me than it claryfied:

So CDP will not be (completely) supported but the BiDi will not be supported by geckodriver since it will be an inherent feature of firefox??

What has to happen on the selenium side (or any other side) to get the console log via client.log("browser")?

[whimboo]

Right, geckodriver is only used for WebDriver HTTP, and as helper to opt-in to WebDriver BiDi via the webSocketUrl capability. Everything else will be directly implemented in Firefox itself, and communication will happen via its own WebSocket connection.

I cannot speak about the Selenium side given that this is its own project. But what you want is:

1. Start a WebDriver session via geckodriver with webSocketURL set to true.
2. Retrieve the server address for WebDriver BiDi via the webSocketURL capability.
3. Use a WebSocket client to connect to that address.
4. Subscribe for log.entryAdded events.
5. Handle log.entryAdded events in your code

Note that there is still the hen and egg problem with not having a proper BiDi client available yet. But AFAIK Selenium folks want to integrate the BiDi feature probably in the 4.1 release.

To give you an impression here is some code from the web-platform-tests repository with a basic test for log.entryAdded that checks console messages.