Headless Chrome integration with TagUI

[kensoh]

ISSUE - PhantomJS maintainer is stepping down. Unless a white knight appears to take over development and maintenance, PhantomJS project will over time become obsolete and not an ideal web automation endpoint due to differences in behavior compared to modern web browsers.

FIX - Main maintainer of CasperJS (the tool that TagUI is base on) would like to try to maintain CasperJS with headless Chrome (https://github.com/casperjs/casperjs/issues/1825). That should be able to transition TagUI directly to headless Chrome, besides the current Firefox endpoint through SlimerJS.

In the meantime however, it is worthwhile to dedicate at least a couple of weeks to try to integrate headless Chrome directly with TagUI. For JavaScript projects, this is typically done through chrome-remote-interface. An example is the Chromy project as pointed out by CasperJS user @tobiastom.

Headless Chrome integration is chosen for now instead of Selenium WebdriverIO integration because it is a more direct path (TagUI -> chrome-remote-interface -> headless Chrome, versus TagUI -> WebdriverIO -> Selenium -> headless Chrome and other browsers). Assumption is the shorter path leads to lesser chance of something breaking and result in a better integration for TagUI.

UPDATE - see updates below on the new approach that integrates TagUI directly with Chrome / headless Chrome. A separate PHP thread is used to manage the websocket communications so that there is no new dependency introduced. The original idea to use chrome-remote-interface or Chromy will make the feature unusable by users without Node.js environment.

---

TagUI v1.8 should be able to be released in a couple of days and work on headless Chrome integration can start. v1.8 is the visual automation release where there is ability to handle interaction by visually looking for webpage objects or desktop objects (through Sikuli integration).

Other new features include improved outgoing API for advance API calls through different methods/headers/body, auto-upload option of run result and automation flow to hastebin.com, and improved step/code block handling using {} which allows powerful and convenient automation when combined with if/for/while/popup/frame. Chrome extension element identification is also improved, and [enter] keyword can be used to type enter keys. More details at readme and release notes.

[kensoh]

Writing some POC (proof-of-concept) tests works great! In fact, from the POC, my sensing is there should be no big technical roadblock to integrate TagUI directly with Chrome / headless Chrome through the chrome debugging protocol. Without going through external dependencies such as the chrome remote interface node.js project and Chromy which is built on top of chrome remote interface.

There is no doubt chrome remote interface and Chromy makes it much easier to do the integration as a lot of work needed to be done to communicate with Chrome is already there. However, it introduces Node.js dependency into TagUI. And they have much more code built-in than what is needed for TagUI steps.

Thus I favor coding the integration directly into TagUI such as below example, without adding additional dependency through Node.js projects such as Chrome Remote Interface or Chromy. If done successfully, this should give the best performance for TagUI execution and let TagUI remain usable by users without Node.js environment, while having access to Chrome + headless Chrome. This approach should take more development effort but should be the preferred way to try unless some other technical roadblock appears.

below starts Chrome on macOS with debugging port on. for headless mode use additional switches --headless --disable-gpu (this disable gpu switch is temporary according to Chrome documentation)

open -a Google\ Chrome --args --remote-debugging-port=9222 about:blank

below retrieves websocket url for the blank page entry point, can convert to json object instead and read programmatically

curl -s localhost:9222/json | grep -A 1 '"url": "about:blank"' | grep webSocketDebuggerUrl

test code embedded in TagUI automation flow to click on web element

var ws = new WebSocket("ws://localhost:9222/devtools/page/5965f284-1c86-47bb-b8c2-9576f97dec90");

ws.onopen = function() {
var message = {'id': 1, 'method': 'Runtime.evaluate', 'params': {expression: 'document.querySelector(\"#giticon\").click()'}};
ws.send(JSON.stringify(message));
casper.echo("Message is sent...");};

ws.onmessage = function (evt) {
var received_msg = evt.data; casper.echo("Message is received... " + received_msg);};

ws.onclose = function() {
casper.echo("Connection is closed...");};

The websocket connection will return error accordingly if the specified locator cannot be found. Running above in visible mode I am able to see the web element with id giticon being clicked. There will be more work to code it and maintain the integration code, but this really should be the path to try first.

[kensoh]

Adding commit tebelorg@0bf4d12 with below comment -

adding project - https://github.com/Textalk/websocket-php

to manage chrome web socket in a separate thread so that there can be concurrency versus single-threaded js and lack of await or promises in current casperjs/phantomjs framework.

using php also does not add new dependencies as current packaged releases already come with php for windows. and linux/macos already has php installed.

---

Now trying another approach to manage websocket communications concurrently from a separate php thread instead of the same single javascript thread as casperjs (lots of challenges which cannot be easily fixed even if willing to make large code changes to TagUI).

The idea is similar to how the TagUI Sikuli integration works. The separate thread should give the best performance and flexibility in the automation implementation. The PHP thread takes only 0.5% CPU time on a 3 year-old macbook air. While polling at 100 ms (configurable in tagui_chrome.php).

[kensoh]

Above new commit with working framework (run with chrome or headless option)

• working direct websocket integration with chrome (or headless chrome)
• concurrent communications using separate php thread
• supporting dynamic css and xpath selectors as per usual use
• done following CasperJS API - exists, getTitle, getCurrentURL, click
• handling initial opening of url and subsequent urls in automation flow

Other CasperJS API needed for TagUI steps / functionality

• sendKeys - used by type / enter step
• mouse.xxx - used by hover / move step
• fetchText - used by read / fetch / show / print steps
• capture - used by snap step
• captureSelector - used by snap step
• download - used by download / receive steps
• evaluate - used by dom step to run in dom context
• getHTML - used by save step to get webpage html

Other stuffs can think of before releasing v2.0

• other CasperJS API debugHTML, reload, back, forward
• selectOptionByValue custom option selection function
• live mode of TagUI steps and supported CasperJS API
• support frame and popup step for websocket chrome
• autoload/kill of chrome or headless chrome from option
• autoload/kill of php helper thread to manage websocket
• port-over main runner tagui to windows tagui.cmd
• update flowchart and readme

Until autoloading of chrome and php helper is implemented, following prepares for TagUI execution through chrome or headless option. Chrome version >= 59 is needed for headless (now released).

# eg 1 - launching macos chrome with remote debugging port on, at certain window size
open -a Google\ Chrome --args --remote-debugging-port=9222 about:blank --window-size=1366,768

# eg 2 - launching macos chrome in headless mode (invisible)
open -a Google\ Chrome --args --remote-debugging-port=9222 about:blank --headless --disable-gpu

# get websocket url for the blank page to be used by php helper
curl -s localhost:9222/json | grep -A 1 '"url": "about:blank"' | grep webSocketDebuggerUrl | cut -d'"' -f 4

# starts php helper for concurrent TagUI communications with chrome
php tagui_chrome.php chrome_websocket_url_from_above

For other custom chrome calls, use chrome_step(method, params). It returns a JSON response from chrome, which we can use JSON.parse to deal with the data returned. (examples in tagui_header.js)

Chrome debugging protocol reference (v1.2 stable) - https://chromedevtools.github.io/devtools-protocol/1-2 and (version tip-of-tree) - https://chromedevtools.github.io/devtools-protocol/tot/ which contain much more methods but many of which are still in experimental status.

[kensoh]

Various components for TagUI direct integration to Chrome + headless Chrome are done. Closing issue.

[TMM21042]

@kensoh, I am using TagUI 2.3 and Chrome Version 60. When I run an automation flow file (.txt) from the command line (Windows 10), TagUI generates a JS flowfile (.txt.js). This file contains a lot of code blocks before it gets to the casper.start() command and the casperJS script that is specific to the automation flow. From what I can understand in the readme, all methods of running TagUI (command line, schedule, URL) use a TagUI command. My question pertains to repeated use of an automation flow and whether or not the casperJS script generated by TagUI (.txt.js) can be called/run independently as a CasperJS command or if the TagUI command must be used each time?

The situation is that I want to make small changes to the casperJS script generated by TagUI for the repetitive automation tests, and running the TagUI automation flow from the start appears to over-right these changes.

[kensoh]

Hi @TMM21042 thanks for asking this question. In most cases, CasperJS code can be written directly in the TagUI flow file, as it is directly being ported over to the output CasperJS .js file. Most of the generated code the header part are supporting functions for TagUI (eg smart XPath/CSS locator), api etc, and the Chrome implementation which is executed only when chrome or headless option is used.

You can check out tagui_header.js to see if what you want to change is there, so that you can make a change there for your changes to be used each time it is run, as a generated file is created on the fly each time a flow is run, base on the options chosen. Otherwise, editing a generated CasperJS script works as well, but you'll have to run it with the appropriate switches for CasperJS.

I don't think all cases can be done that way. For example you may run with TagUI chrome or test option. The generated script is already in a format that have the appropriate CasperJS code structure to do that. It can't be easily changed to do something else such as run in firefox or non-test mode. Separate files can be generated though for editing, using different options.

---

Adding notes for this thread, in a sense, you can say TagUI is a CasperJS script generator, adding on with various functions / integrations to do things which is not in the CasperJS feature-set. Theoretically, it is possible to make changes to TagUI so that it can directly run a normal CasperJS script, but output to headless Chrome. But I think some users who explicitly wants to use headless Chrome have started exploring the Chrome-based tools directly. Also, TagUI only made Chrome implementation on a subset of CasperJS API for areas used by TagUI steps. To make all the CasperJS API available for Chrome is a nightmare. CasperJS/PhantomJs are really mature and expressive libraries for their job.

The past few months I'm actively supporting CasperJS issues and mailing list, but TagUI features are not something that can be implemented as pull requests into CasperJS. Technical incompatibility (basically the way PhantomJS works is not compatible with the way newer Chrome-based tools work) and features being too specific for very specific use cases which really shouldn't be part of CasperJS in the first place (for eg visual automation, uploading run results online etc). TagUI is more like a framework for the sole purpose of automating web processes easily. It tries to make the creation, deployment and maintenance easy. But in doing so, many specific design choices have to be made, which means they shouldn't be implemented to CasperJS to retain its neutral / flexible API.