Layer Changes / T-String All The Things

[ianjosephwilson]

After experimenting with the parser a lot I have been thinking it might be better to fully embrace t-strings by re-packing the Template back into a more "aware" Template. I have a "not very well integrated" prototype so I thought I'd start talking about it here and then maybe I could slowly make small pull requests and keep the tests up to date without sending a bunch of PRs out of no where. This mainly builds on the placeholder idea to use an integer index to lookup the original interpolations in the "values"/source template.

From a high level without completely drowning in details the long version could look something like this (psuedo code):

values_template = t'<div><img src={image_src}><span class="caption">{image_caption}</span></div>'
parser: NodeParser = get_parser()
struct_node: Node = parser.parse_template_into_node(values_template)
processor = get_processor()
structure_template: Template = processor.process_node(struct_node)
renderer = get_renderer(parser=parser, processor=processor)
html_string = str(renderer.render_html(values_template, stucture_template))

The cached version would look like this but would use values_template.strings to lookup the structure_template from the cache:

# Do all that stuff above behind the scenes if its not in the cache
html_string = str(renderer.render_html(values_template))

In the simplest possible scenario we'd do something like this (psuedo code):

# start walking left to right yielding strings and interpolations
q = [walk_template(structure_template)]
while q:
    parts_itr = q.pop()
    for part in parts_itr:
        if isinstance(part, str):
            yield part
        else:
            if is_html_content(part):
                # pause left to right walk and (descend a level or just extend the left to right in-place)
                q.append(parts_itr)
                q.append(walk_template(resolve_content(values_template, part))))
                break
            elif is_html_attrs(part):
                yield from resolve_attrs(values_template, part)
            elif is_component(part):
                # pause left to right walk and descend a level
                q.append(parts_itr)
                # resolve attrs for component and then call it, get back another template and start walking it
                q.append(walk_template(invoke_component(values_template, part)))
                break

Parser Changes

Remove placeholders and put in Interpolations.

• attributes
• components
• content

Node Changes

Nodes become more like an easier to manage 1-to-1 mapping of the structure of the values_template but in a hierarchy and with all the metadata established. We don't ever interpolate the actual values here we just pre-extract the placeholders and replace them with actual objects that still use the integer indices.

Processor Changes

Instead of interpolating into the node actually coerce the node tree back into a structured template. Almost the reverse of the parser where static content is flattened back down but with the information we gleaned from parsing maintained in Interpolations (or something similar).

• Dynamic attributes are condensed into a single interpolation
• Dynamic content is collected in a template
• Component tags are condensed into a single interpolation
• Static content is just flattened back out

Renderer Changes

Instead of using Node.__str__() to render the html we act directly on the structure template and descend through interpolations into nested/embedded templates.

Examples

These are just psuedo code right now but I could flesh out the details. The EXACT implementation might need to be adjusted because I'm not sure we'd want interpolations in interpolations or not. Maybe using custom tuples / slotted dataclasses or something would be better.

def t(*parts):
    return Template(*[part if isinstance(part, str) else Interpolation(part, '', None, part.format_spec) for part in parts])

process_template(t'<div>{content}</div>' ) == t('<div>', ContentSpec(...), '</div>')
process_template(t'<div {attrs}>{content}</div>' ) == t('<div', AttrSpec(), '>', ContentSpec(...), '</div>')
process_template(t'<div {attrs}>{content}<{component}></{component}><span>Static</span></div>' ) == t('<div', AttrSpec(), '>', ContentSpec(...), '', ComponentSpec(...), '<span>Static</span></div>')

[davepeck]

I like the simplicity of the current key public API: html(t: Template) -> Node, where the Node hierarchy is a 1:1 HTML tree and doesn’t try to encode substitution semantics.

I could definitely see adding a second render_html(t: Template) -> str fast path that skips building the Node tree and uses a more semantic compiled representation under the hood for performance.

I also agree we want a better cached intermediate representation in the processor. Holding on to Node with placeholder strings was… expedient but awkward, and something more explicitly semantic like what you’re proposing should be easier to reason about and optimize.

[ianjosephwilson]

The placeholder idea is genius in a way but there is something about having them "floating around" in the Nodes that can be confusing. I had a hard time trying to figure out what is happening and what should be happening. Also there are some book-keeping issues about where things were in the original template that can only be determined while parsing (mainly the > of a start tag if there were a lot of attribute interpolations). Furthermore when things go wrong with interpolation it makes things hard to debug because we are a few years into building out the html strings at that point.

I was able to make some progress on trying to get one of my prototypes shoe-horned into the current codebase and a new pattern did emerge. If a new "kind" of nodes existed that were "Template Nodes", TNodes, then:

• book-keeping information from parsing could be retained
• substitution rules could be well documented
• type checking validation would be more robust
• the "processing step" would be easier to evaluate and test
• you could tell if a "Node" was fully interpolated (not sure about children in this case maybe that would be something to determine?)

So there would be Node, Fragment, Text, Element, DocumentType and Comment and then at least:
TNode, TText, TElement, TComment. These would be structured versions of their fully interpolated counterparts that are ready for receiving interpolations.

So conceptually something like this:

doc_t: Annotated[Template, "html"] = get_doc_t()
t_node: TNode = parse_html(doc_t)
node: Node = html(t_node, doc_t.interpolations) # could do both at once like now but just for clarity
html_string: Annotated[str, "html"] = str(node)

[davepeck]

Yes! This is very similar to the direction I've been exploring to try and clean up the interpolation behavior, per our other discussions: a separate intermediate node hierarchy that has semantic substitutions, which hopefully makes much clearer exactly how it all ties together.

(It is maybe also strange that Node is emitted from the underlying HTMLParser -- you could imagine the final boss form of the Node hierarchy being frozen dataclasses, and the parser itself just emits a hierarchy of dicts or some simpler mutable type.)

[ianjosephwilson]

Yes! This is very similar to the direction I've been exploring to try and clean up the interpolation behavior, per our other discussions: a separate intermediate node hierarchy that has semantic substitutions, which hopefully makes much clearer exactly how it all ties together.

Yes that sounds great. I have taken a few stabs at it and got kind of far recently. There are a few "gray" areas around attributes and also how we determine "components" that I still don't fully understand. I was trying to avoid introspecting the interpolations as much as possible but when you check for a component end tag it seems unavoidable. It doesn't seem like a good idea to depend on expression other than for debugging information but do you think its viable to just check if interpolations[A].value == interpolations[B].value where the tags might be <{placeholder for A}...> .... </{placeholder for B}>? It seems like a rough spot. The anonymous closing tag seems like a better idea which I saw somewhere in there.

(It is maybe also strange that Node is emitted from the underlying HTMLParser -- you could imagine the final boss form of the Node hierarchy being frozen dataclasses, and the parser itself just emits a hierarchy of dicts or some simpler mutable type.)

Having the built out hierarchy of the template is really nice though for checking correctness, ie. did the attribute interpolations get put in the correct spots. I could see there being different flows just so we can use the less performant, but easily verified, version to check that the "optimized" versions are not doing something crazy.

[ianjosephwilson]

I've tried out a few things regarding the post parsing / pre processing format. This one kind of came out of the ashes and is very simple:

in template	example	stored
spread attribute	<div {attrs}>	(0,)
exact attribute interpolation	<img title={title}>	("title", 0)
attribute template	<img title="An image of {title}.">	("title", "An image of ", 0, ".")
attribute w/ value	<img title="The ocean.">	("title", "The ocean.")
attribute w/o value	<img title>	("title",)

It might be "too" tricky/fragile but we just flatten each attribute type into a tuple that uses indexes to point to the original interpolation, this is just meant to be iterated, for attr in attrs and easy to apply to match attr, like this:

def _substitute_interpolated_attrs(
    attrs_seq: tuple[tuple[int|str, ...], ...], interpolations: tuple[Interpolation, ...]
) -> dict[str, object|None]:
    new_attrs: dict[str, object | None] = LastUpdatedOrderedDict()

    for attr in attrs_seq: #tuple[tuple[int|str, ...], ...]
        match attr: #tuple[int|str, ...]
            case [int()]: # spread
                interpolation = interpolations[attr[0]]
                spread_value = format_interpolation(interpolation)
                for sub_k, sub_v in _substitute_spread_attrs(spread_value):
                    new_attrs[sub_k] = sub_v
            case [str()]: # bare attr
                new_attrs[attr[0]] = True
            case [str(), str()]: # static attr
                new_attrs[attr[0]] = attr[1]
            case [str(), int()]: # interpolated attr
                new_value = format_interpolation(interpolations[attr[1]])
                for sub_k, sub_v in _process_attr(attr[0], new_value):
                    new_attrs[sub_k] = sub_v
            case [str(), str()|int(), _, *rest]: # templated attr
                new_attrs[attr[0]] = "".join(
                    part
                    if isinstance(part, str)
                    else str(format_interpolation(interpolations[part]))
                    for part in attr[1:]
                )
            case _:
                raise ValueError(f"Unknown attr format {attr}")
    return new_attrs

We could make a more structured format that tries to keep all these things separate with a "kind" flag or try to revert to plain Template/Interpolations but the downside of Templates/Interpolations is that they have extra info we don't need here they we have to fill in all the time and they also are not comparable by default.

[ianjosephwilson]

Well I put a draft version of a structured format design in a PR to reference maybe: #76

[pauleveritt]

Just checking about this thread, has the discussion moved elsewhere (to PRs) and we should close this thread?

[ianjosephwilson]

I think the parser changes that we are trying to merge in now are the first part of this, ie. TNode, TElement, TAttribute, etc.

a separate intermediate node hierarchy that has semantic substitutions, which hopefully makes much clearer exactly how it all ties together.

@pauleveritt I looked at your projects a little bit and I'm starting to understand more of why the "realized" Node structure exists for analysis. ie. t-string -> TNode -> Node -> str(). Seems pretty obvious in retrospect with DOM in the name but oh well...

The second part was a plan to "repack" or "recompile" the templates into something that was faster to render. This isn't demo'ed yet and I'm not sure how compatible it would be with your middleware concepts I have been trying to grok. This would look more like this:
t-string -> TNode -> t-string "recompiled" -> str(). I think both could co-exist but I'm still trying to fully understand how it would be used. So in short this idea isn't fully realized. I think maybe I would clean this up and re-start an new discussion with a new PR proof-of-concept and we could re-discuss it then but we could leave this here for now if that's alright and I'll close it when I re-draft it.

[pauleveritt]

I confess, I've long had a deep, almost daily interest in a crazy idea: keeping intermediate representations around as JSON to cache in...SQLite. I'll spare you the explanation. In a nutshell: what's the least amount of work to remake a site from a change?

[pauleveritt]

Sounds good about letting you wrap up your work and start a new discussion, that's the right idea.