[test-runner-commands] Mouse/Click interface to Playwright/Puppeteer?

[Westbrook]

Building from https://modern-web.dev/docs/test-runner/commands/#send-keys, I'd like to open a conversation about supporting some sort of interface on the Mouse or ElementHandle.click() APIs to empower browser originating mouse events. While sendKeys or the new proposed a11ySnapshot are both without direct corollary in the JS context, mouse events can be pretty successfully synthesized in JS, so this might not be needed. However, let's discuss what it might or might not open up as testing possibilities.

In JS Clicks

First, let's think about what control we have today in JS, both the el.click() method (which is actually a good context to leverage while testing as it allow for you element to perform in test similarly to how it would perform via imperative interactions in to a consumers element or application), and the various events that would otherwise be dispatched in this workflow. That is to say, we already have full control over the entire process of a click with:

el.dispatchEvent(new Event('mousedown'));
el.dispatchEvent(new Event('mousemove'));
el.dispatchEvent(new Event('mouseup'));
el.dispatchEvent(new Event('click'));

Negatives
The main difficulty that would arise from this would have to do with remembering to include all of the browser native options of dispatched events. Note that none of the above events use bubbles, composed, or cancellable all of which may in some way be needed to ensure the functionality in response to this interaction. We can certainly add these in the JS space, but it points to a complexity that we must keep in mind while doing so.

Positives
The main benefit of this is that you have fine control over the event cycle. In this context, having access to each of the click "phases" means that you can test for specific response in each phase, which can help to ensure you code is performing as needed.

Mouse

Here the Playwright and Puppeteer APIs are roughly the same and offer the following:

mouse.click(x, y[, options])

Which is a wrapper for:

mouse.move(x, y[, options]);
mouse.down([options]);
mouse.up([options]);

An important difference here is that this workflow is not bound to an element. This means that we can test, without the knowledge that something exists, what happens when we click in a specific part of a page. Testing an interaction in this way leans as close to an Integration/e2e test as possible in the "unit" space in that it doesn't not make any assumptions about the unit under test and could be a valuable addition to a @web/test-runner tool belt.

It would seem that the above is both the negative and the positive of leveraging this API. The fact that it's not directly the same as el.dispatchEvent(new Event('click')) could point to it being the best addition to the tools of the two options the runner afford us here.

For a usage API point of view, a sendKeys relative API might look like:

sendMouse({
    move?: [x, y, options],
    down?: [options],
    up?: [options],
    click?: [x, y, options],
});

I'm still working on my TS, but having the options locations like this separate should allow for more strictly typing the various shapes of that argument:

interface MoveOptions {
    steps: 1;
}

interface DownOrUpOptions {
    button: 'left' | 'middle' | 'right;
    clickCount: 1;
}

interface ClickOptions {
    button: 'left' | 'middle' | 'right;
    clickCount: 1;
    delay: 0;
}

However, I could see benefits around validation simplicity of something like:

sendMouse({
    type: 'move' | 'down' | 'up' | 'click';
    position?: [x, y];
    options: MoveOptions | DownOrUpOptions | ClickOptions;
});

ElementHandle.click()

This API is fairly similar to the el.click() options that we have native to the JS context. However, it does comes with some nice ergonomics around things like:

• Wait for actionability checks on the element, unless force option is set.
• Scroll the element into view if needed.
• Use page.mouse to click in the center of the element, or the specified position.
• Wait for initiated navigations to either succeed or fail, unless noWaitAfter option is set.

The first three are things we have some access to in the JS context, but include direct mitigation to ensure, while the fourth has some interested possibilities within it. While we'd not want to actually be changing the URL in a unit context, we may want to mock data request, or file download, and having the test runner manage the wait for those things to resolve could be a neat side effect of leveraging this API.

Not being able to pass a ElementHandle over the commands interface (to the best of my understanding), we would need to take element resolution into account in the API for a command built around this. Much like the a11ySnapshotPlugin, we'd likely solve this with a selector argument (that fallback to the entire page?) to ensure that we have the right information available when taking this action. So it might surface like:

clickElement({
    selector: '#my-element',
    options: {}, // see https://playwright.dev/docs/api/class-elementhandle#elementhandleclickoptions
});

One specific feature of the options bag for this API is that is accepts modifiers, which allow us to make the click with <Array<"Alt"|"Control"|"Meta"|"Shift">> modifiers pressed when enacting the click. I'd assume we could pair the sendKeys and sendMouse APIs to do the same there, but I've not checked that out yet.

Negatives
clickElement is much more abstract than having access to the entire series of events that make up a click event. Needing to have an element in question for testing brings this API much closer to the capabilities that we already have access to in the JS context. More research is needed to confirm whether falling back to the entire page with this API would seek to the lowest available element in the position sent before dispatching the event of not.

Positives
The pairs abilities like waiting for availability, clicking a visible part of the element, applying modifier keys, and waiting for subsequent network requests in a way that could be more ergonomic than the other options. The simplicity of a selector and an options bag leaves users with less API surface to keep in mind.

Questions/Research

• Do any of these interfaces surface pointer events? If we were getting this paired with mouse events by default by way of interacting with the browser runner then I'd be much more clearly sold on the benefits of leveraging one or more of the APIs? CONFIRMED, yes.
• Do the APIs cover different enough surface that we'd want to offer both of them to our users?
• How do we feel about the way these APIs are structured in the current proposal?
• What interactions with the mouse might I have missed that we should attempt to take into account here?

[Westbrook]

In the context of the Mouse API, it may be productive when doing things other than click to be enact more than one mouse instruction in a row. In this way you could directly leverage the above, like so:

await sendMouse({
    move: [9, 30]
});
sendMouse({
    down: [],
});

Or send an array of instructions to be taken sequentially:

await sendMouse([
    {move:[9,30]},
    {down:[]},
]);

The idea of batching the commands shifts me a little more towards the alternative command structure in that case:

sendMouse([
    {
        type: 'move' ,
        position: [9, 39];
    },
    {
        type: 'down',
    }
]);

Luckily, while doing this, the underlying pointer* events related to the mouse motion are dispatched as expected, which is quite nice.

[WickyNilliams]

Following discussion on slack, I thought I'd add my thoughts here for posterity...

I think it would be useful to have a command for clicking on an element (as well as specific coordinate), since a programatic click does not always match behaviour of an actual mouse interaction. One such example is someHtmlSelectElement.click() - this will not open the . And this is a nice convenience over calculating coordinates yourself

[vursen]

The idea to add a command for triggering the :hover state (described in #1758) actually relates to this discussion as the command can be implemented on top of the API proposed here.

[WickyNilliams]

I agree, should support both coordinates and a selector (or perhaps even passing in an element instance?).

[Westbrook]

I’m not sure you can pass an element reference from the client to server context, but that would be interesting!

[WickyNilliams]

It's done in visual snapshot plugin by giving the element an id and storing in an object on window, then it's retrieved server side by id. I think this is such a common use case, there should maybe be first class support for it. Selectors can't pierce shadow dom, so you need some other way to pass elements around eventually

[WickyNilliams]

I've been experimenting with my own mouse command...

I realised you don't need to pass an element through to the server, you can just calculate suitable coordinates from the element, in the browser, and only ever send x/y coordinates over.

So i was able to support both of these:

await mouse({ click: [0, 0] })
await mouse({ click: someElement }) // this calculates the center x/y coords behind the scenes with getBoundingClientRect()

[WickyNilliams]

OK, back with more things i've learnt in the process...

When you click an element, the mouse stays in the same position in subsequent tests! This causes weird issues if you have hover styles or mouseover events etc. I had to mouse mouse back to 0,0 after click to avoid this. Even this may have issues, but seems ok in my case for now