PROPOSAL: tdom-path: Relative path rewriting

[pauleveritt]

tl;dr A library that rewrites paths (static assets, links) in a Node to be relative to a target.

Problem

My theming work involves static assets that live in a directory as part of a component (layout, view, regular component.) Since my themes are intended to be framework-portable, I don't have access to all the helper functions (Jinja/Flask, Django, etc.) that you put in their interpolations. Plus, their approaches don't match the tooling-oriented DX we are looking for.

Proposal Overview

Let's start a package and a spec called tdom-path which tackles this. We also use it to incept what's needed in tdom itself for middleware-like utilities.

This effort is crucial to fulfill the tdom vision of "Big ecosystem of quality, interoperable themes, components, and tooling."

Scenario

Let's say I'm in src/storyville/components/header/. My logic/template is in header.py as Header. But it has some CSS and an image. Where do I put these files?

Then, later, when rendering, how do we calculate the correct href path? If I is an SSG (or there is some system that prepares static assets), how do I provide these files to a build step?

Goals

In some order of importance...a laundry list, but all things I've tackled at some point in the last 7 years.

Great DX via actual paths

Ideally, the static assets for Header would be in header/static/style.css for example. Header's t-string would include <link rel="stylesheet" href="./static/styles.css">. This is great for several reasons:

• Assets are local to the component, not over in some other garbage barge global static directory
• Smart editors can autocomplete the path reference, as it points to an actual path on disk
• Equally: squiggle if you got the path wrong
• Refactorings (e.g. file rename) could find all usages and fix
• Other static analysis tools could do things such as this, but possibly more

Portable

The moment we start writing url_for('static', filename="styles.css"), we lose the "big ecosystem" vision. We also lose what was just described: unless your editor/tooling is taught that specific frameworks syntax/meaning, you'll get no tooling support.

We could partially solve this by having some standard callable in a SOA (service oriented architecture) that called into per-framework implementation. But while that might be a fallback choice, we likely will lose other goals. Most of those were created before the days of Python tooling.

Locality

Having asset files local to the component fulfills a number of goals.

Dynamic and SSG

We have approaches for this now in Django and Flask (dynamic), in Sphinx and Pelican (static), etc. Ideally we come up with a solution that works in either case (that likely means generating relative paths.)

Part of a build system

This system should anticipate (though perhaps not solve) all the parts of the build process:

• Definition time when the template is first rendered (or even parsed)
• Application time, when the component is used as part of a render at a URL
• Build time, when an SSG (or static prep system) is optimizing static assets and writing to output directory

Good error messages

This is a very fiddly part of doing web development. Lots of magic, lots of mystery. We now have new powers, mainly through static analysis tooling and t-strings as an in-language feature. What new approaches can we do, beyond "pretty tracebacks"?

Optimized performance

Other systems take trips from string to parse then back to string, multiple times (e.g. WSGI middleware.) We are building a middleware ecosystem that anticipates handing around Node. That's already a win.

But we can go further. Can we collect component path information as part of the Node structure? Even crazier, could we put a closure in the parse representation that had the data needed for path-on-disk?

Can we collect information to make the build output generation part super-optimized?

Other paths

We are focused on paths for static assets. But we also have <a href> links to other resources. Perhaps in the header nav of a component. But perhaps in the body of a Markdown document. We'd like those to also be re-written. Even better, some systems (Sphinx) know when a target doesn't exist and you can emit a warning. Those systems should be able to plug into link generation to warn/fail early.

Site prefix

For SSGs, the build directory on disk might be deployed to the hosted site under a static prefix. We should anticipate that. (If we use relative paths everywhere, that isn't a problem.)

Convention over configuration

This is popular but I'm not a fan, as "convention" is a romance word for "impossible to static analyze". 😉 Perhaps we should make it possible to "just drop in a folder named static" but that should be a layer over a declarative, tooling-friendly system.

Pluggable

There might be better implementations. Or, people might prefer their battle-tested. If we design this as SOA with a Protocol or something, we could allow alternate implementations to be "registered."

Analysis

Let's look at how we might do this. We might first start with a survey of a bunch of systems, collecting common patterns and needs.

tdom context object

I've implemented some feature-complete versions of this, starting perhaps in 2019. Recently I implemented this on a tdom branch for context objects: html(t'<link href="static/styles.css">', context=this_context):

• This context was passed down through the rendering
• Components could optionally ask for it
• The component rendering lifecycle would look inside for custom factories, e.g. my old Hopscotch DI system
• I could also register middleware that would run before/after

This worked really well, but surfaced a number of issues:

• What is the type information of context if it is custom?
• Should there be well-known plug points?
• Stated differently, should we define a minimum rendering lifecycle?

Component information

This was my biggest problem. Once a component renders to a Node, I no longer have information about its path on disk. I can't then resolve file paths to static/styles.css. We could define an annotations on Node where "the system" could stash data. But perhaps TNode and parse time could do it. Perhaps even with a callable closure to make it flexible.

PurePath

My older system leaned heavily on PurePath which is mainly known through the Posix implementations. But it doesn't have to. What if we used the interface of PurePath but backed with our own concept of a tree? (That's what my thing did.) We then type into tons of tooling and expectations if static/styles.css is a variant of PurePath.

Two-layer and free threaded

My latest effort was a ChainMap with the system context as an immutable and the per-render as a writable that gets reset. It would be nice though if we made decisions that were free-threaded friendly. Can we make a render "bag of data" concurrent?

[pauleveritt]

@ianjosephwilson @davepeck Hey guys, sorry for the wall of text. I need this soon so I plan to work on a tdom-path that monkey patches my previous tdom context work.

[ianjosephwilson]

These are great ideas @pauleveritt but I feel like you are so far ahead right now!

This was my biggest problem. Once a component renders to a Node, I no longer have information about its path on disk. I can't then resolve file paths to static/styles.css. We could define an annotations on Node where "the system" could stash data. But perhaps TNode and parse time could do it. Perhaps even with a callable closure to make it flexible.

This is really helpful information. We have changed the current cache to not directly depend on the exact callable used, so potentially the callable used in a template the first time might be different later on. There is another cache for the callable info that probably does or could get this information and cache it. That is something we could maybe put on a Node but the node might be ambiguous if its from a fragment, furthermore the serialization has no idea its not a regular node. I think this is something that would have to be handled in the component itself somehow, as nodes are meant to be "done".

Do you mean sort of a "mount" point ? So the component is mounted in the HTTP path hierarchy of a web application and the asset urls inside it are resolved relative to that mount point regardless of where they are actually invoked within the application itself? Ie. FlatThemeContainer is mounted at /themes/flat-theme so whenever it is used it can include a <link href="static/styles.css"> that resolves to <link href="/themes/flat-theme/static/styles.css"> ?

[pauleveritt]

Thanks for taking the time to load this into your brain Ian. We're genuinely lucky to have you.

I wrote in my local notes that this idea needs to play nice with a route based system that might not like relative URLs. This point matches what you finish with. For my initial needs, the "resolves to" at the end would be a relative path. I can point to my most recent implementation if that would help.

If nodes are meant to be "done" than alas, middleware might be out the window?

[ianjosephwilson]

I guess there is a lot of branches between concepts and implementations right now that makes it confusing and every term already had 10 meanings even before any our discussions. I'm not sure where this idea fits in but I just am trying to understand what it is in my idea-space.

For middleware I think you might mean something like this might happen:

First: Somewhere the component is invoked:

container_result = '<{FlatThemeContainer}><div>with some children</div><link href="static/styles.css"></{FlatThemeContainer}>'

Then: This triggers some middleware from somewhere like this and that middleware adjusts the link's href:

def invoke_component_and_apply_middleware(component, sorted_middleware):
    # after calling, here component is FlatThemeContainer
    children_nodes = render_children_nodes() 
    component_and_children_nodes = component(children=children_nodes, **someargs)
    for middleware in sorted_middleware:
        if middleware.applies(component):
            component_and_children_nodes = middleware.after_invocation(component_and_children_nodes)
    return component_and_children_nodes

def invoke_component_and_apply_middleware(component, sorted_middleware):
    # OR before calling, here component is FlatThemeContainer
    children_nodes = render_children_nodes() 
    for middleware in sorted_middleware:
        if middleware.applies(component):
            children_nodes = middleware.before_invocation(children_nodes)
    component_nodes = component(children=children_nodes, **someargs)
    return component_nodes

[pauleveritt]

I apologize, I know I'm adding noise when you're trying to wrap up some fundamental re-thinking. I can make a prototype (I had it before) or I can write some code here in a comment. Which would you prefer?

[pauleveritt]

@ianjosephwilson Here we go, my first middleware package: https://t-strings.github.io/tdom-path/

• Decorate on a component
• Walks tree to find <link href=""> etc.
• Replaces the Element with a TraversableElement that uses an importlib.resources package spec to remember its path
• Later, the system finds such nodes and resolves (e.g. to relative path for current render)