Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need a callback for when children changed or parser finished parsing children #809

Open
rniwa opened this issue Apr 30, 2019 · 133 comments
Open

Comments

@rniwa
Copy link
Collaborator

rniwa commented Apr 30, 2019

There appears to be a strong desire for having some callback when the parser finishes parsing children or when a new child is inserted or an old child is removed.

Previously, children changed callback was deemed too expensive but I'm not certain that would necessarily be the case given all major browser engines (Blink, Gecko, and WebKit) have builtin mechanism to listen to children changes.

I think we should consider adding this callback once for all. As much as I'd like to make sure custom elements people write are good, developer ergonomics here is quite terrible, and people are resorting to very awkward workarounds like attaching mutation observers everywhere, which is arguably far slower.

@Jamesernator
Copy link

Jamesernator commented Apr 30, 2019

Not sure if this is possible but would it be possible to simply capture the value of childrenChangedCallback immediately after the element is constructed? Then one could easily enable an optimization that doesn't bother watching the children at all if the callback isn't actually provided.

(This would be similar to the iterator.next changes in for-of loops that .next is only captured once at the start of the loop).

EDIT: It looks like the other methods already work like this by capturing the methods off the prototype (TIL).

@rniwa
Copy link
Collaborator Author

rniwa commented Apr 30, 2019

All custom element callbacks are retrieved at the time of definition so such an optimization is already possible. See step 10 of https://html.spec.whatwg.org/multipage/custom-elements.html#dom-customelementregistry-define

@annevk
Copy link
Collaborator

annevk commented Apr 30, 2019

I support adding a children changed callback as that is indeed a primitive the DOM Standard provides and specifications and implementations use. (Firefox also has internal synchronous mutation observers, but nothing in the platform relies on them, though Firefox does currently use them for the output element.)

Before we add this however I'd like whatwg/dom#732 (review) solved. In particular, the primitive I referenced above isn't defined that well and some refactoring is needed. It'd be good to settle on the exact shape in private first before exposing it publicly.

(Note that this is effectively #550 btw.)

@domenic
Copy link
Collaborator

domenic commented Apr 30, 2019

As with #550, I do not support this if it is parser-specific. If it is just a CEReactions-time MutationObserver that only custom elements can install, then I'm open to investigating. In that case I think we should also seriously consider un-deprecating DOMNodeRemoved/DOMNodeInserted as an API shape, since that already exists in some browsers and we've needed to spec it properly for some time.

@justinfagnani
Copy link
Contributor

If it is just a CEReactions-time MutationObserver that only custom elements can install

I would love it to be a bit more than that, in that it observes projected children as well. I know that's covered slotchange, but currently you need to combine slotchange and a MO to implement "effective children changed".

@rniwa
Copy link
Collaborator Author

rniwa commented Apr 30, 2019

I don't think observing the assigned slot should be conflated with this API.

@rniwa
Copy link
Collaborator Author

rniwa commented Apr 30, 2019

A common ask that keeps coming up is the ability to know when a child node has been inserted & updated: e.g. #765, #619, #615

To do this, just knowing when a child is inserted or removed isn't quite enough due to async / out-of-order upgrading. Perhaps we need both childChangedCallback and childDefinedCallback the latter of which gets called when a child node is inserted & upgraded / defined.

Also see whatwg/dom#662

@caridy
Copy link

caridy commented May 1, 2019

@rniwa do you foresee this as something that can be observed in a custom element or a more low level API that can be applied to any element? It is not clear from the title or description.

@rniwa
Copy link
Collaborator Author

rniwa commented May 1, 2019

@rniwa do you foresee this as something that can be observed in a custom element or a more low level API that can be applied to any element? It is not clear from the title or description.

I think this needs to be a custom element reaction callback.

@caridy
Copy link

caridy commented May 1, 2019

I think this needs to be a custom element reaction callback.

That's probably limiting. I feel that this is in the same boat as the connect/disconnect reaction that we were asking for last week during the F2F, something a lot more generic.

Now, if we were to add a new reaction callback, will that callback be invoked even when the CE is not connected to the DOM? e.g.:

const ce = document.createElement('x-foo');
ce.appendChild(document.createElement('p'));

@rniwa
Copy link
Collaborator Author

rniwa commented May 1, 2019

I think this needs to be a custom element reaction callback.
That's probably limiting. I feel that this is in the same boat as the connect/disconnect reaction that we were asking for last week during the F2F, something a lot more generic.

For general nodes, you can just use MutationObserver. If you wanted it sooner, then that's a discussion that has to happen in whatwg/dom#533

Now, if we were to add a new reaction callback, will that callback be invoked even when the CE is not connected to the DOM?

You mean when it's not connected to a document? If so, then yes. There is no reason to restrict this childChangedCallback to the connected nodes.

@franktopel
Copy link

franktopel commented May 2, 2019

Remember the imo much more important part remains the childrenParsedCallback. Obviously that wouldn't make too much sense when the CE is not connected to the DOM. Currently CE only work in the upgrade case if they rely on children for setup.

Also, for a childChangedCallback we already have a working tool, MutationObserver. Why would we need an additional callback?

@rniwa
Copy link
Collaborator Author

rniwa commented May 2, 2019

Let first state that the time at which the parser had finished running isn't a great way to do anything because DOM is inherently dynamic. Any node can be inserted or removed after the parser had finished parsing its contents, not to mention that HTML parser could later insert more children via adoption agency, etc...

The reason we want to add childChangedCallback is ergonomics for the same reason we have attributeChangedCallback even though attribute changes can be observed via MutationObserver as well.

Now, custom elements have unique characteristics that child elements may be dynamically upgraded later so just knowing when a child is inserted may not be enough for a parent element to start interacting with a child element. This is a rather fundamental problem with the current custom elements API. We need some way for a parent to discover a child custom element when it becomes well defined.

@annevk
Copy link
Collaborator

annevk commented May 2, 2019

@rniwa how is whenDefined() not adequate for that?

@franktopel
Copy link

franktopel commented May 2, 2019

Any node can be inserted or removed after the parser had finished parsing its contents

I'm not interested in that case. Imagine you want to have an element <table is="data-table"> and you need to set it up - how are you ever going to do this without having guaranteed access to all children? Many web projects still generate static HTML code on the serverside using classic template engines like Thymeleaf etc. This statically generated content inside a webcomponent needs to be reliably accessible for the webcomponent at some point - and this point should definitely be a lifecycle callback, not the funky stuff that frameworks like Google AMP do, which led to the creation of https://github.com/WebReflection/html-parsed-element

In the course of the discussion back in the days I created a gist that sums up the problem and suggests a solution that served as a basis for the development of the aforementioned package.
https://gist.github.com/franktopel/5d760330a936e32644660774ccba58a7

@annevk
Copy link
Collaborator

annevk commented May 2, 2019

As already explained in multiple comments at the beginning of #551 that's not a pattern I want to encourage by adding API surface for it as it'll break down whenever someone dynamically uses such an element. I'm not sure why @rniwa mentioned the parser again. If we really want to reopen that discussion it should be in a separate issue as it's unlikely to get agreement, whereas children changed is an existing low-level primitive that would be vastly easier to get agreement on exposing.

@franktopel
Copy link

franktopel commented May 2, 2019

There appears to be a strong desire for having some callback when the parser finishes parsing children

This is exactly what this issue addresses, unfortunately it mixes it up with

or when a new child is inserted or an old child is removed.

which is a completely different problem. I really think there is a strong necessity to solve the first problem first, because the second problem can already be tackled with using a MutationObserver. While that may not be as convenient as having a childrenChangedCallback, it definitely works and does so without going ten extra miles just to find the right time via deduction the type of which

if ([this, ...this.parentNodes].some(el=> el.nextSibling) || document.readyState !== 'loading') {
  this.childrenAvailableCallback();
} else {
  this.mutationObserver = new MutationObserver(() => {
    if ([this, ...this.parentNodes].some(el=> el.nextSibling) || document.readyState !== 'loading') {
      this.childrenAvailableCallback()
      this.mutationObserver.disconnect()
    }
  }
  this.mutationObserver.observe(this, {childList: true});
}

does.

Please note that this is also explicitly mentioned in the entry post:

[...] and people are resorting to very awkward workarounds like attaching mutation observers everywhere

@justinfagnani
Copy link
Contributor

which is a completely different problem.

I don't think parser finishing and dynamic children are different problems at all. The parser adding nodes is just a sub-problem of dynamic child changes and a single API can handle both cases.

The fact that you can use MutationObservers after parsing is complete doesn't solve the current issue that developers have to piece together several APIs in order to know when their children change.

@rniwa
Copy link
Collaborator Author

rniwa commented May 2, 2019

I'm not interested in that case. Imagine you want to have an element table is="data-table" and you need to set it up - how are you ever going to do this without having guaranteed access to all children?

Components that don't support updating its appearance upon dynamic DOM tree change is outside the scope of custom elements API. We don't intend to address such a use case.

To answer @annevk's comment about why I brought up the parser again: I think there is legitimate scenarios in which the most natural solution authors think of is finishedParsingChildrenCallback. As far as I dissected the problem space, I think the most common complain there is that out-of-order upgrades makes it impossible to know when a parent can start interacting with children. I think this is an unique requirement that built-in elements don't have. So while I tend to agree we don't want to add that exact callback, we may need something like childDefinedCallback as I suggested above to address the underlying use cases of issues which motivated folks to request finished parsing callback.

@franktopel
Copy link

franktopel commented May 2, 2019

Components that don't support updating its appearance upon dynamic DOM tree change is outside the scope of custom elements API.

Can you please give reference on this? I assume such a heavy-weight decision is well-reasoned and -documented. What exactly is in scope of the custom elements API?

@annevk
Copy link
Collaborator

annevk commented May 3, 2019

@rniwa it seems to me that the combination of childrenChangedCallback, customElements.whenDefined(), plus some bookkeeping is all you need for that. Though perhaps if the bookkeeping gets too complicated there is some kind of shortcut we could offer if libraries all end up with something similar. (Seems like something user land should figure out the pattern first for though.) Also, @domenic has a point though that whatwg/dom#305 perhaps should be flushed out first given that we're likely not able to get rid of mutation events. Nevertheless, iIt might still make sense to offer a childrenChangedCallback given its similar timing and more idiomatic custom element API.

@franktopel it was decided early on that we wanted to encourage custom elements that behave equivalently to built-in elements and would therefore not add hooks that built-in elements do not have. It might be a little tricky to find a definitive reference for that, but that's best discussed separately from this thread in a new issue.

@justinfagnani
Copy link
Contributor

@rniwa it seems to me that the combination of childrenChangedCallback, customElements.whenDefined(), plus some bookkeeping is all you need for that. Though perhaps if the bookkeeping gets too complicated there is some kind of shortcut we could offer if libraries all end up with something similar. (Seems like something user land should figure out the pattern first for though.)

To give some color about what I've been doing for this situation: I generally fire events from children when they're ready to interact with an ancestor, and have the child wait for the parent to be ready to receive the event with customElements.whenDefined(). This seems to be simpler code usually and avoids tightly coupling to the exact structure of the DOM (works with grandchildren as well).

@WebReflection
Copy link

WebReflection commented May 6, 2019

The fact that you can use MutationObservers after parsing is complete

I think current work around is using MutationObserver on constructor, hence before the parsing is complete.

I generally fire events from children when they're ready to interact with an ancestor

That works only if children are Custom Elements.

A <sortable-list> might be a container for an UL and some LI, and knowing when the content is ready is crucial when it comes to setup.

Imagine the server producing the following based on some data:

<sortable-list>
  <ul>
    <li>a</li>
    <li>b</li>
  <ul>
</sortable-list>

If you define upfront the sortable-list component, and you don't want to use ShadowDOM, which shouldn't be mandatory to setup Custom Elements, the only way to do that is to use a hack via html-parsed-element or similar, or to use MutationObserver within the constructor so that the component will inevitably flick due asynchronous nature of the MO.

I think @rniwa here nailed the issue me, and others, are complaining about.

I think there is legitimate scenarios in which the most natural solution authors think of is finishedParsingChildrenCallback.

Anything else would result in probably nice to have, but not an answer to the real issue.

The fact built-in elements don't have such lifecycle is misleading, 'cause a <select> internally setup likely once, instead of N times synchronously per each discovered <option> with it.

We, users, simply don't get to know when that setup should happen, which is independent of the usage of MO in case the component should be able to react on added/removed nodes, 'cause these are not mutually exclusive scenarios.

@annevk
Copy link
Collaborator

annevk commented May 7, 2019

Did you test <select>? As far as I know that's not true. (Not even clear to me how that would work if you dynamically append to it.)

@WebReflection
Copy link

WebReflection commented Oct 13, 2023

I think the fact that several built-in elements on several browsers make use of such a functionality kind of proves it isn't an "extreme special case".

I don't think there's anything else to add in here ... this is close to the builtin extend discussion, where vendors can do it all, but exposed APIs to Web consumers are limited for no strong reasons.

@Danny-Engelman
Copy link

where vendors can do it all, but exposed APIs to Web consumers are limited for no strong reasons.

where vendors do it all for you, but the native API needs some extra work for consumers (replicating what vendor code does)

@annevk
Copy link
Collaborator

annevk commented Oct 13, 2023

@mfreed7 I would probably do it incrementally instead, that way it would be the same if the form came via the parser or a script (say, if the website was using the Navigation API). Any existing usage apart from maybe </script> is more indicative of a problem than good design in my experience.

(It's also not clear to me why input uses this. It doesn't even have an end tag, except in XML. That seems like something that can be changed.)

@justinfagnani
Copy link
Contributor

justinfagnani commented Oct 13, 2023

@annevk

Any existing usage apart from maybe </script>

<script>-like tags are exactly one of the use cases for a close tag signal though. I've been making declarative custom element libraries, and without knowing that the declaration is finished parsing you can get into some really bad situations where you have a broken element.

@bmeck
Copy link

bmeck commented Oct 24, 2023

Heya, just wanted to chime in that I was building a simple component that only shows 1 of its children at a time using display:none on dynamically generated <slot>s to avoid problems like in https://jsfiddle.net/tLc2znyh/ . I use manual slot assignment so I had to use a MutationObserver to mix wc.children and dynamic slot.assign() ( https://codepen.io/bradleymeck/pen/poxzXoE?editors=1001 ). Having a simpler way to do this would be great. I would note, in at least me scenario, even with a simplistic callback needing to re-layout due to slots is pretty wasteful and prone to some memory leakage in my tests (see layout() in codepen example) where I have to manually cross reference slot elements and their assignments so having better clarity on WHERE the node has been moved to would be ideal not just that it was added/removed.

@mfreed7
Copy link

mfreed7 commented Oct 30, 2023

And also, in the event the custom element creation was not parser-driven you'd have the exact same issue, but no solution, as you don't know when the append() calls will stop. So it's better to solve for that and then the parser case works too.

Coming back to this - I wonder if you could provide an example of how you'd "solve for that" here? Maybe using the two concrete use cases provided in this comment? One is for the <script> element, and the other is about entry animations for a chunk of DOM. (Note that neither are custom elements, nor have imperative append equivalents, per-se.)

@smaug----
Copy link

XTF (which with XBL was like web components on steroids) had a notification to tell when all the child nodes had been added to it during parsing. And that notification was very useful to implement new elements.

I don't yet have an opinion on whether some notification should come through MutationObserver or whether Custom Elements should have a callback. Children changed callback on a custom element might be rather slow if it was using CEReaction-type of timing during parsing - that would be basically the opposite why microtasks were created for MutationObserver. But then, on the other hand there is already attributeChangedCallback....

@mfreed7
Copy link

mfreed7 commented Dec 15, 2023

XTF (which with XBL was like web components on steroids) had a notification to tell when all the child nodes had been added to it during parsing. And that notification was very useful to implement new elements.

Thanks for the reference!

I don't yet have an opinion on whether some notification should come through MutationObserver or whether Custom Elements should have a callback. Children changed callback on a custom element might be rather slow if it was using CEReaction-type of timing during parsing - that would be basically the opposite why microtasks were created for MutationObserver. But then, on the other hand there is already attributeChangedCallback....

One point is that there seem to be use cases that don't involve custom elements, and it'd be nice to be able to solve those. So if there's a general solution (which I think MutationObserver represents), that would seem better to me. An async event would also do the trick.

@annevk
Copy link
Collaborator

annevk commented Dec 15, 2023

@mfreed7 I don't really understand the script use case. That seems like some kind of meta-programming it wasn't designed for. I think Opera at some point had beforeexecute events and such, but they had their own issues.

If it's for rendering I'd expect you maybe delay a bit and then just render what you have at some interval. Similar to incremental rendering in browsers.

@smaug---- how was XTF resilient against imperative creation of the same elements?

@mangelozzi
Copy link

On a side note, the argument of "What if the end tag is never enountered"... well then you have bigger problems, and code that depends on reaching the end does not fire. This does not seems like a realistic counter argument. To render HTML you need the HTML, that much is a given.

And secondly not wishing to provide it because could result in slower web pages, is like a country banning fire because people could get burnt. There are many ways to make a slow website. To not allow a feature that would help a lot just because it could be used the wrong way is tying developers hands and treating them like children.

@mfreed7
Copy link

mfreed7 commented Dec 15, 2023

@mfreed7 I don't really understand the script use case. That seems like some kind of meta-programming it wasn't designed for. I think Opera at some point had beforeexecute events and such, but they had their own issues.

If it's for rendering I'd expect you maybe delay a bit and then just render what you have at some interval. Similar to incremental rendering in browsers.

The use case (one of them, at least) is waiting for <script> tags to finish parsing (and therefore contain all of the script) and then moving the script element between documents. If there's no way to know when all of the script is there, it's brittle.

I'm not sure we have a list of the use cases the web is "designed for". We provide functionality, and creative developers build cool things with that functionality. Parsing <script>s and moving them between documents is certainly one part of that functionality that seems like fair game.

@keithamus
Copy link
Collaborator

JavaScript is not alone in the quirk of requiring the last byte to properly evaluate the file, many languages have references such as Yaml, Markdown, JSON Schema, consequently while <script> covers JS there is a realm of other languages which a website may wish to implement as a similar element to <script>, but would be need to resort in very awkward workarounds to find out where the end of the file is.

@smaug----
Copy link

@smaug---- how was XTF resilient against imperative creation of the same elements?

There were begin/end callbacks for the parser creation.

In the current element implementations in Gecko there is usually a flag which is set to true for non-parser created elements, and on parser created elements it is set to true when child nodes have been added to it.

@justinfagnani
Copy link
Contributor

It would be nice if a solution here also addressed #979

If a child-parsed callback was only called by parser created elements, then script-like elements could only execute on that callback. Addressing the "file"-like data use case seems like a superset of executable data though, where you may was to know when data is complete in insecure parsing contexts like innerHTML.

@silverwind
Copy link

silverwind commented Mar 6, 2024

To me it seems to reliably access children in connectedCallback, one has to check for both existing children (which is the case when the element is appended to the DOM via .innerHTML or similar methods like Vue does it), and for regular DOM content where the children will be available after a childList mutation. Related demos.

connectedCallback() {
  if (this.children.length) 
    // access this.children
  } else {
    const observer = new MutationObserver(() => {
      observer.disconnect();
      // access this.children
    }).observe(this, {childList: true});
  } 
}

Honstly, it seems way to much boilerplate for such a simple and common use case.

@patricknelson
Copy link

Just to clarify @silverwind it looks like your code is only for change events, and even then, only on the first change and doesn't cover the use case of indicating when the parser has completed parsing and appending children of a particular element on the DOM.

@silverwind
Copy link

silverwind commented Mar 6, 2024

Yes, in this limited case I'm only interested in getting the reference to a mandatory child element. It will not be removed/added again ever after.

@mfreed7
Copy link

mfreed7 commented Mar 28, 2024

The use case (one of them, at least) is waiting for <script> tags to finish parsing (and therefore contain all of the script) and then moving the script element between documents. If there's no way to know when all of the script is there, it's brittle.

I'm not sure we have a list of the use cases the web is "designed for". We provide functionality, and creative developers build cool things with that functionality. Parsing <script>s and moving them between documents is certainly one part of that functionality that seems like fair game.

Another use case (for builtin elements) is source selection for <video>. The conclusion on that issue seems to be that the best behavior is indeed to wait for the end tag before making a source selection. If that conclusion sticks, it'll be another instance of something a built-in element can do that a custom element cannot, because of the lack of the API this issue proposes.

@mangelozzi
Copy link

mangelozzi commented Mar 31, 2024

To me it seems to reliably access children in connectedCallback, one has to check for both existing children (which is the case when the element is appended to the DOM via .innerHTML or similar methods like Vue does it), and for regular DOM content where the children will be available after a childList mutation. Related demos.

connectedCallback() {
  if (this.children.length) 
    // access this.children
  } else {
    const observer = new MutationObserver(() => {
      observer.disconnect();
      // access this.children
    }).observe(this, {childList: true});
  } 
}

Honstly, it seems way to much boilerplate for such a simple and common use case.

Say a component has many children, is there any chance the parser may connect the component after parsing some of the children, i.e. not all of them. It appears to this code assumes that all the children exist, or none of them at the time the connectedCallback runs. (I am not sure if this is the case?).

@WebReflection
Copy link

Say a component has many children, is there any chance the parser may connect the component after parsing some of the children

if you define your component at the end of the page you are 100% sure you can access its children and everything else as it gets just "promoted" as CE ... if you define your component AOT (which is 90% of what libraries do instead of lazy dafining their components only when needed/encountered) you are 100% sure (last time I've checked) the connectedCallback will trigger instantly after attributeChangedCallback in case it was defined and the attribute is observed and encountered ... at that point the browser hasn't parsed yet the content of that element so you'll fallback with MutationObserver for children which guarantees nothing but usually it works "good enough".

The whole issue is to have something that never fails and that can be used in all circumstances and definition time, something that while streaming a document will signal via closedCallback that its ending has been encountered, something that even with lazy / late definitions will still trigger after the connectedCallback.

The problem raised in here is that in some streaming scenario the browser might already consider the element closed ... add people with "smart suggestions" like "don't need to close a <p> or a <li>" 'cause the browser does that for you, see early hacks on never closing a <body> because the browser does that for you (and the legend say it's faster in doing so) and so on ... so the issue remains and solutions are all awkward in a way or another, or really not applied via lazy custom elements definition, something I've explored ages ago, something that works and never suffers the issue itself but developers don't like it for other reasons (FOUC or other related shenanigans that might be indeed a concern specially with 100% ShadowDOM based components).

@Sleepful
Copy link

Sleepful commented Apr 3, 2024

I wonder if simply using the defer attribute on <script> helps?

I suppose it helps if the Component has not been registered already, but it might not help if it already registered.

Couldn't components have a similar defer quality?

@mangelozzi
Copy link

if you define your component at the end of the page you are 100% sure you can access its children and everything else as it gets just "promoted" as CE ... if you define your component AOT (which is 90% of what libraries do instead of lazy dafining their components only when needed/encountered) you are 100% sure (last time I've checked) the connectedCallback will trigger instantly after attributeChangedCallback in case it was defined and the attribute is observed and encountered ... at that point the browser hasn't parsed yet the content of that element so you'll fallback with MutationObserver for children which guarantees nothing but usually it works "good enough".

Often you have components that need to be loaded quickly, e.g. if they are above the fold. Putting them last on the page does not work in this case. It's kind of like putting your stylesheets just your closing </body>.

For now I make a function that handles slotted children, that function has to be idempotent.
When the component is connected:

  1. I setup a eventHandler for slotchange, that calls this function, this.addEventListener('slotchange', this.slotChangeHandler.bind(this))
  2. Set a zero wait timeout to call this function almost immediately, e.g. setTimeout(this.slotChangeHandler.bind(this))

This handles the situation when the children are added before the component can initialise, and when they are slotted after it has been initialised. A bit clunky but don't know of any better.

@nickcoury
Copy link

Updating on my case of streaming into a detached document and selectively moving nodes to the visible document. I found a much more efficient solution that also reliably identifies the closing tag by skipping the MutationObserver altogether and using a NodeIterator instead. Something similar might be possible for other use cases. Rough solution:

// Only works when streaming into an empty detached document. Otherwise additional logic is needed.
function elementIsClosed(element) {
  let currentElement = element;
  while (currentElement) {
    if (currentElement.nextSibling) return true;
    currentElement = currentElement.parentElement;
  }
  return false;
}

const iterator = new NodeIterator(detachedDocument, NodeFilter.SHOW_ELEMENT);
let element;
const response = await fetch("https://example.com");
const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();

while (true) {
  const { done, value } = await reader.read();
  if (done) return;
  detachedDocument.write(value);

  element = iterator.nextNode();
  while (element) {
    const isClosed = elementIsClosed(element);
    // Perform processing
    element = iterator.nextNode();
  }
}

detachedDocument.close();

// Process the root element if needed, document is closed so the root element is closed

This works for a few reasons:

  1. document.write() and NodeIterator are both synchronous, so each chunk is written and processed synchronously.
  2. Assuming the HTML is well-formed, the document and NodeIterator will both write and iterate depth-first. This allows the elementIsClosed() function to work reliably. If we find a nextSibling or a nextSibling of a parent, we know we encountered the close tag of the current element for something to be written after it.
  3. This assumption only holds in the detached document, not the visible document. It also only holds if we strictly remove nodes from the detached document and never add or move elements.
  4. It's possible to run similar logic on an open element that is streamed into the visible document, but needs to also keep track of the root node being streamed in the elementIsClosed() function and stop bubbling the parentElement at that point, since we expect nodes to exist outside of that root.
  5. If an element is moved to the visible document when it is open, it will continue to stream into the visible document. The NodeIterator in the detached document is live, so it will not find any of the children streaming element. The processing step won't find any new elements on subsequent chunks (at first). Once the streaming element closes, the next element will be written after it into the detached document, and iterator.nextNode() will again return that element. This can also be used to determine that the streamed element is closed, if needed.

In some simple tests, a slower device can iterate a ~5000 node document in ~10ms with some basic conditional checks like attribute lookups.

@WebReflection
Copy link

@nickcoury it's a great solution and thank sfor sharing but this also solves nothing and it's fully unrelated with Web Components, right?

@nickcoury
Copy link

Sorry, yes it's not directly related but still may be relevant. See my earlier comments for context. I originally commented with a similar use case looking for a MutationObserver close tag event. I haven't fully explored this for web components but it's possible part of the approach could be adapted.

I don't think there's an event when each chunk is written to check for completeness, though a new MutationObserver callback or a setInterval could be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests