A faster way to find open shadow roots (or shadow hosts)

[TakayoshiKochi]

There are requests from web developers who need faster access to all trees under a document, e.g. to find out specific elements that matches some criteria. Now you have to run querySelector() recursively for each document and its descendant shadow roots. You don't have any faster way than iterating all elements to see if their .shadowRoot is null or not.

A popular use case that I have heard is for testing, and one possibility is to add a new interface to webdriver. But developers also demand some fast solution usable on the web, even if it's applicable to open shadow roots. As adding a new combinator to CSS selector was strongly opposed, it is not an option at least for now.

Other possibilities so far:

• A new pseudo class (e.g. :has-shadow-root) that matches a shadow host
• A new flag or filter for TreeWalker API for matching a shadow host or a shadow root.
• A new API that can accept limited set of selector (e.g. a simple selector) against all elements in all trees

First two options may achieve the same goal, though no one yet has submitted a concrete spec proposal. The last option was from Apple, but no standardization effort so far.

I once ran an experiment of performance, see WICG/webcomponents#78 (comment). From the results, any of the above once implemented native, would have significant speedup.

Background history: In the Shadow DOM V0 era Google proposed /deep/ combinator (later renamed to >>>) in CSS selector and found that the usage in stylesheet was problematic for performance, and restricted the usage only in querySelector() (aka snapshot or static profile). But it did not reach agreement with other vendors (see WICG/webcomponents#78). The rationale we had in selector was that it gives both performance and flexibility of CSS selectors, at the cost of implementation complexity.

References:
https://bugs.chromium.org/p/chromium/issues/detail?id=829713
WICG/webcomponents#78
w3c/csswg-drafts#640
https://www.chromestatus.com/feature/5045542597951488

[TakayoshiKochi]

I filed w3c/csswg-drafts#2908 for a new pseudo class at csswg.

[TakayoshiKochi]

For TreeWalker API, probably no one so far proposed anything concrete for Shadow DOM, as far as I know. As TreeWalker is based on node iteration, it may not be as fast as we expect, although we may be able to optimize nextNode() specifically only for when the shadow host flag is specified. To achieve the original motivation, a pseudo class is much simpler solution to implement.

The below is a dump of my thoughts, you can skip reading:

My first naive idea was to add a new flag (e.g. SHOW_SHADOW_HOST), in NodeFilter to match a shadow host even though the flag becomes non mutually-exclusive against existing ones (SHOW_ELEMENT).
The next idea was to add SHOW_SHADOW_ROOT flag which is not mutually-exclusive against SHOW_DOCUMENT_FRAGMENT either, and traversing algorithm needs to be updated to visit shadow root.
Probably extending the node traversal algorithm to support shadow trees are too much for what we can get from there.

[diervo]

@TakayoshiKochi would it be easier or have you consider start creating the abstraction in selenium/webdriver first rather than directly adding new things to the spec?

We could introduce new primitives like findAndExpandShadows() where internally we can implementing a traversal with whatever mechanisms we want.

We (salesforce) will be happy to drive the spec changes on that front since we already maintain and contribute to some of the implementations.

We believe adding something on the HTML/CSS might not be the right first start since we would want to gather the exact use cases first, without introducing any new shadow/WC introspection.

[TakayoshiKochi]

Hi @diervo I'm not familiar with selenium/webdriver, but I pinged w3c/webdriver#350 if anyone has interest. I appreciate if you could continue shaping the API there.

And yes, Treewalker API might not be a good point for extension, as I already commented above. But I hope w3c/csswg-drafts#2908 could help usages at runtime, not only testing.

[travisleithead]

After mulling this over, I wonder if something patterned after the existing document collections could work?
E.g., we have document.scripts, document.links (which finds all anchors with an href); something like document.shadowHosts (which could return all elements with a shadowRoot property that is non-null and open).
Though this would probably not work recursively if that is a necessary use case...

[annevk]

F2F discussion: it's unclear what the use cases for this API are and to what extent libraries have tried to work around it (or implemented it themselves). This needs more use cases to progress.

[freshp86]

F2F discussion: it's unclear what the use cases for this API are and to what extent libraries have tried to work around it (or implemented it themselves). This needs more use cases to progress.

Couple of example use cases from Chromium's code.

• Find placeholder strings/attributes in the DOM (at runtime) and replace them with localized strings, see here.
• Test that all icons in a page are properly rendered, see here.
• Disable all animations and transitions during automated tests, see here.
• Implement custom search, as done in chrome://settings page. /deep/ was previously used to find and remove all "search bubbles" when the user clears the search results. See example screenshot
  
  That code has now been updated to not use /deep/, but the workaround was heavily involved for something seemingly so simple. Basically we are now caching a reference to all "search bubbles" so that we can remove them later without having to search the DOM.

Overall, being able to query the entire DOM via JS using querySelector is very useful, and does not necessarily mean that the developer is doing something wrong (this is not about CSS penetrating shadow roots). By providing no such built-in way, developers are forced to keep implementing their own recursive search solutions in JS, which have poor performance and are cumbersome.

[WilcoFiers]

I just wanted to add a maybe more concrete use case. Accessibility testing tools currently are having a pretty hard time testing within shadow DOM, because there is no way to work out where all the shadow roots are. Axe-core (which I work on) was to a large extent rewritten, exactly because of this limitations. Other accessibility tools, such as HTML_Codesniffer have never supported shadow DOM, and are unlikely to do so without an efficient way to query the flat tree.

[annevk]

@WilcoFiers WebDriver or equivalent would be the way for accessibility tools to get access. It seems that was only recently fixed in the standard: w3c/webdriver#1616 by @jgraham. Not sure where implementations are at.

I think for the DOM Standard this can probably be closed at this point. We don't really want to add an API that favors one mode over another and the only pattern we've come up that works for closed shadow roots is the caller proving they can access them, at which point there is not much point querying for them.