Consider making Template and Interpolation generic at runtime

[sobolevn]

Feature or enhancement

Template and Interpolation will once be supported by type-checkers. And it might be worth adding generics support for these types.

Example

from string.templatelib import Template, Interpolation
from urllib.parse import quote_plus

domain = 'example.com'
query = 'python string formatting is too complex'
template = t'https://{domain}?q={query}'

def quote_url(interp: Interpolation[str]) -> str:
    return quote_plus(interp.value)

def format_url(template: Template) -> str:
    parts = []
    for part in template:
        match part:
            case str() as s:  # regular string
                parts.append(s)
            case Interpolation(str(), expression='query') as interp:
                parts.append(quote_url(interp))
            case Interpolation(value):
                parts.append(value)
    return ''.join(parts)

print(format_url(template))

Here we can see that Interpolation[str] can be useful in users' code.

We can also benefit from generic Template, where we can make it generic based on exact TypeVarTuple. Demo:

from typing import Any, TypeVarTuple, Generic, Unpack

Ts = TypeVarTuple('Ts', default=Unpack[tuple[Any, ...]])

class Template(Generic[*Ts]):
    def __new__(cls, *args: *Ts) -> Template[*Ts]: ...  # type: ignore[empty-body]
    
class Interpolation: ...
    
reveal_type(Template('a', 'b', Interpolation(), 'd'))
# Revealed type is "__main__.Template[Literal['a']?, Literal['b']?, __main__.Interpolation, Literal['d']?]"

https://mypy-play.net/?mypy=latest&python=3.13&gist=0dc13b3b926e1efb9783ab9b70d39ceb

This can potentially help type checkers to infer correct Template type.

Current state

Current typeshed definitions: https://github.com/python/typeshed/blob/main/stdlib/string/templatelib.pyi

Here's how Template is defined:

@final
class Template:  # TODO: consider making `Template` generic on `TypeVarTuple`
    strings: tuple[str, ...]
    interpolations: tuple[Interpolation, ...]

    def __new__(cls, *args: str | Interpolation) -> Template: ...
    def __iter__(self) -> Iterator[str | Interpolation]: ...
    def __add__(self, other: Template | str) -> Template: ...
    @property
    def values(self) -> tuple[Any, ...]: ...  # Tuple of interpolation values, which can have any type

Ideally, it should be generic on TypeVarTuple

Here's how Interpolation is defined:

@final
class Interpolation:
    value: Any  # TODO: consider making `Interpolation` generic in runtime
    expression: str
    conversion: Literal["a", "r", "s"] | None
    format_spec: str

    __match_args__ = ("value", "expression", "conversion", "format_spec")

    def __new__(
        cls, value: Any, expression: str, conversion: Literal["a", "r", "s"] | None = None, format_spec: str = ""
    ) -> Interpolation: ...

Ideally, it should be generic on value. Interpolation[str] means that value is str and Interpolation[tuple[int, int]] means that Interpolation has a value for type tuple[int, int]

Proposal

From runtime's part with only need to add Py_GenericAlias, that's it.

Is it too late for feature freeze? This is a minor detail, so I hope it is not :)

I have a PR ready.

CC @srittau @JelleZijlstra @lysnikolaou

Linked PRs

• gh-133970: Make PEP750 types generic #133976
• [3.14] gh-133970: Make PEP750 types generic (GH-133976) #134029

[srittau]

One alternative is to make Template generic over the interpolation value types instead. So __new__ would remain typed as it is, but type checkers could special case this (which they already have to do to support template strings) so that:

foo: str = ""
bar: int = 42
tmpl = t"x{foo}{bar}y"
assert_type(tmpl, Template[str, int])
assert_type(tmpl.values, tuple[str, int])

[sobolevn]

@srittau this would be also possible, if we just add Py_GenericAlias to Template. Exact typing would be determined later by type-checkers / typing council.

[JelleZijlstra]

Yes, I think we should do this, and it should be fine to do it during the beta phase as part of stabilizing the new feature.

We'll have to decide in typeshed what the actual generic parameters are. I think it makes the most sense to use a single type parameter for both Template and Interpolation, so that for example Template[str] holds a series of Interpolation[str] and each has a value of type str.

[sobolevn]

@JelleZijlstra ok, I will send a PR with Py_GenericAlias.

[AA-Turner]

cc @lysnikolaou @davepeck (edit: Lysandros was alredy tagged, sorry!)

[AA-Turner]

I think having Interpolation[T], where T is the type of i.value makes sense. For Template, my first reaction is that t'a {1} b {dict()} c {0.0}' could be type-hinted as Template[LiteralString, int, LiteralString, dict, ...]. This is very verbose though, and it would probably be annoying to type! Jelle's suggestion of a single parameter may be simpler in practice, though more restrictive for heterogeneous types.

# Revealed type is "__main__.Template[Literal['a']?, Literal['b']?, __main__.Interpolation, Literal['d']?]"

What do the question marks mean here? This isn't legal syntax.

A

[sobolevn]

@AA-Turner

What do the question marks mean here? This isn't legal syntax.

This is how mypy pretty-prints type when reveal_type is called. ? means that this type was inferenced, mostly.

[davepeck]

Considering the two suggested approaches:

1. Template[T] implies that Template.interpolations is typed as tuple[Interpolation[T], ...]: I worry that real-world templates are nearly always heterogeneous and we'll end up with big unions.

2. Template[A, B] implies Template.interpolations is typed as tuple[Interpolation[A], Interpolation[B]]: The types seem likely to get very large. Consider, say, building a full HTML page from a deeply nested collection of templates.

---

Walking through a simple example:

def user_image(user: User, classes: Sequence[str] = None):
    return t"<img src={user.image_url} alt={user.image_name} class={classes} />"

def user_details(user: User, attribs: Mapping = None, classes: Sequence[str] = None):
    return t"""
        <div {attribs}>
            <span>{user.name} ({user.star_count})</span>
            {user_image(user, classes)}
        </div>
    """

# What's the type of this `Template`?
template = user_details(
    some_user, 
    {"aria-label": user.name, "aria-hidden": False}, 
    ["rounded"]
)

In this example:

• {attribs} is a dict[str, str | bool]
• {user.name} is a str
• {user.star_count} is an int
• {user_image(...)} is a nested Template
• {classes} is list[str]

In approach (1) we probably end up with Template[object] in practice?

In approach (2) I suppose we get something like Template[dict[str, str | bool], str, int, Template[str, str, list[str]]]?

[JelleZijlstra]

I would think of this in terms of the type that you write if you accept a template. In your example, there might be a function render_html(template). If it can interpolate say str and int in a template, it would be typed as def render_html(template: Template[str | int]):. Type checkers can then tell you if you pass some object that isn't allowed as a template value.

[davepeck]

I would think of this in terms of the type that you write if you accept a template.

Right. With approach (1) and the example above, I suppose we'd type it as render_html(template: Template[int | str | bool | list | dict | Template]) if we wanted to be particular about what we accept. A real render_html() would probably have a much bigger union. I dunno, is this desirable?

(Sorry... maybe I'm just not thinking clearly about this?)

[sobolevn]

Any big union can be just replaced with Template[Any] :)
Which might be a TypeVar default for this case, if we agree on the typing.

[JelleZijlstra]

@davepeck If you don't want to be precise you can still write just Template, which will be equivalent to Template[Any]. But if you're writing a framework, it seems worth it to be precise about the types you accept.

[davepeck]

If you don't want to be precise you can still write just Template, which will be equivalent to Template[Any]. But if you're writing a framework, it seems worth it to be precise about the types you accept.

Hah, yeah, sorry for my cobwebs here. 😅

Templates are a form of collection, so it does seem to make sense to make them generic.

Of the options, Template[T] giving us interpolations: tuple[Interpolation[T], ...] feels the most natural to me. Making the TypeVar default Any also seems pretty clear.

So why the cobwebs? Mostly just the realization that (a) real-world template processing code will essentially always have large unions for T, and (b) most of what's interesting to capture about a template isn't expressible with our type system anyway. (For instance: "yes, render_html() accepts a dict value... but only at specific points in the HTML grammar".)

But both of those things are just fine!

Okay, cobwebs cleared. Assuming we're all on the same page, LMK and I'll put together a PR for the PEP.

---

PS: two minor things maybe worth mentioning:

1. Bytes templates (bt"...") have been discussed; maybe in the future we have Template[T, ST] where ST is str or bytes.

2. Approaches to hinting the grammar in the t-string itself have been discussed.Template["html"] has been one shorthand suggestion I've seen; in practice, it's probably more like Annotated[Template, "html"]

[TeamSpen210]

The big unions might be alleviated a bit by aliases, especially if checkers can display them unexpanded. Probably they'll be necessary anyway, since you'd want to specify sub-templates recursively. For the default, object might be better - templates are immutable and therefore covariant, so that'll accept any template, but require users to verify interpolation values before use. It's inconsistent with other generics though...

[sobolevn]

I merged the runtime generics PR, but I won't close the issue, we can continue talking about semantics here :)

[lysnikolaou]

I'm not sure about this. Yes, Interpolation being generic makes sense, but Template is much harder and both choices that have been presented here are far from ideal and will mostly go unused in practice. I think we need to think bigger about this and, while I'm not the right person to do it, we might also want to explore options for how our type system could evolve around t-strings and the new options they open up.

Tying ourselves to this specific design of making Template generic and then having to adhere to backwards-compatibility rules is not really something I'd like us to do. Maybe we should revert the PR and do it only for Interpolation for now?

[JelleZijlstra]

The runtime support is very simple; it just allows writing Template[whatever] creating a types.GenericAlias. It's up to type checkers and typeshed to decide what exactly will be allowed within the brackets, and we can decide that later without impacting backwards compatibility at the CPython level.

It's hard to do much with a vague "we need to think bigger", but it seems likely that any type system support for templates will involve some variation of Template + square brackets, whether those square brackets hold the types of the interpolations or something else. Having this support in CPython now means that even if we specify a type system feature later, people will be able to use it even in CPython 3.14.

I'd encourage you to open a discussion in the typing section on Discuss to brainstorm what this bigger thing would look like. What would be helpful would be examples of how templates are expected to be used and what kinds of things you'd like type checkers to be able to help you catch.

[sobolevn]

Probably any reference implementation for any of the type-checkers will also help a lot to decide what is best for the static typing.

[lysnikolaou]

The runtime support is very simple; it just allows writing Template[whatever] creating a types.GenericAlias. It's up to type checkers and typeshed to decide what exactly will be allowed within the brackets, and we can decide that later without impacting backwards compatibility at the CPython level.

Okay, I misunderstood the PR then and this looks good.

It's hard to do much with a vague "we need to think bigger", but it seems likely that any type system support for templates will involve some variation of Template + square brackets, whether those square brackets hold the types of the interpolations or something else. Having this support in CPython now means that even if we specify a type system feature later, people will be able to use it even in CPython 3.14.

I'd encourage you to open a discussion in the typing section on Discuss to brainstorm what this bigger thing would look like. What would be helpful would be examples of how templates are expected to be used and what kinds of things you'd like type checkers to be able to help you catch.

I know that this is vague and I apologize for that. It's just that I can't really do better than that because I don't know what that "bigger thinking" might look like. There's ideas about combining type checking with static analysis for DSLs, but I don't think that's in scope for this thread.

The recommendation to open a thread on Discuss is good. I also did a lightning talk in the language summit to get people to think about this stuff. I'll try to do it ASAP, but I'm not able to lead that discussion.