Container API: render components in isolation

[natemoo-re]

Body

Summary

Astro components are tightly coupled to astro (the metaframework). This proposal introduces a possible server-side API for rendering .astro files in isolation, outside of the full astro build pipeline.

Background & Motivation

Some of our own proposals, as well as many third-party tool integrations, are blocked until we expose a proper server-side rendering API for .astro components. Other frameworks have tools like preact-render-to-string or react-dom/server that make rendering components on the server straight-forward.

We've avoided doing this because... it's very hard! Part of the appeal of .astro components is that they have immediate access to a number of powerful APIs, context about your site, and a rich set of framework renderers. In the full astro build process, we are able to wire up all this context invisibly. Wiring up the context manually as a third-party is prohibitively complex.

Goals

• Provide an API for rendering .astro components in isolation
• Expose a familiar, user-friendly API
• Surface enough control for full astro parity, but abstract away internal APIs
• Enable third-party tools to consume .astro files on the server
• Enable unit testing of .astro component output
• Possibly unblock .mdx compiledContent/html output?
• Support Astro framework renderers (@astrojs/*) if possible

Example

import { AstroContainer } from 'astro/container';
import Component from './components/Component.astro';

const astro = new AstroContainer();
console.log(await astro.renderToString(Component, { props, slots }))

The container can be optionally constructed with some settings that are typically reserved for Astro configuration.

const astro = new AstroContainer({
  mode?: 'development' | 'production';
  site?: string;
  renderers?: Renderer[]
});

The second argument to renderToString optionally provides render-specific data that may be exposed through the Astro global, like props, slots, request, and/or params.

await astro.renderToString(Component, { props, slots, request, params })

See feat/container for an exploration of implementation.

[JReinhold]

I assume this will only work if the consumer has vite-plugin-astro applied? Is that Vite plugin already built to be able to work independently of the Astro CLI, our would that need work too?

[natemoo-re]

Great question! Yes, we would need to make sure that vite-plugin-astro is exposed publicly with a reasonable API. That's definitely in scope for this proposal.

The plugin is pretty close to being independent of the CLI, but it does currently need to be initialized with the following (internal) context:

interface AstroPluginOptions {
  settings: AstroSettings;
  logging: LogOptions;
}

Looking at the implementation, this shouldn't be too difficult to decouple. Ideally any plugin settings would be optional.

[JReinhold]

Is the return value of AstroContainer.renderToString meant to be added directly to the HTML with something like

const content = await astro.renderToString(Component, { props, slots, request, params })
document.getElementById('root').innerHtml = content;

?

Have you considered a more friendly API for rendering to the DOM, by taking in a DOM element that it would render straight too?

await astro.renderToDOM(
  Component,
  { props, slots, request, params },
  document.getElementById('root')
)

This might just be syntax sugar for the above, so it might not be too valuable. I don't know if there are special cases where it needs to be handled differently, that renderToDOM could then handle internally.

[JReinhold]

Server-side only would be a problem for Storybook, yeah.
I think I can see a workaround where our renderer calls an API that returns the rendered component, but that only works in dev. In build outputs (which are static) there wouldn't be a server to call. Theoretically you could maybe pre-render the components at build time, but they wouldn't be interactive, like reacting to prop changes, which is a big part of the Storybook workflow.

Personally I'd rather have a big slow bundle that works than none at all.
But I also don't know which APIs Astro uses under the hood to render a component, if it's even feasible to use them client-side (although the two examples you provide are usable on the client). I would even be okay if this was hidden behind a flag like DO_NOT_USE_THIS_IT_WILL_MAKE_YOUR_BUNDLE_THE_SIZE_OF_ALL_ASTRO_SITES_COMBINED.

[natemoo-re]

Hmm, still sitting with this! I hadn't realized that Storybook needed the ability to render Astro entirely on the client. Sounds like this Container API might be a step closer to unblocking your use case, but still doesn't deliver the full story.

We can ensure that this API wouldn't block Storybook, but my guess is that we might need another proposal or two to build on top of this to get things where you need them.

[kyr0]

Client-side rendering capabilities would be great but for my use-case it wouldn't be required. As long as I can render components via an endpoint on the server side, I can use this to render previews of a component and then use it in the WYSIWYG editor I'm working on to preview.

Therefore, because a component is passed by reference, it would be nice if I could dynamically import a component from an arbitrary path relative to the endpoint file location, so that in an endpoint impl. I could do:

const Component = await import('../../components/foobar.astro')

Alternatively, it would be great to just pass the path to the component as a string to the renderToDOM API.

On the other topics..:

• returning only HTML as a string is a nice thing already, but it would be helpful if it was possible to get metadata out of it too. I'm talking about the internal APIs having styles and scripts separated from rendered HTML. If I'm not mistaken, the idea was to just combine everything into one with renderToString -- but for my WYSIWYG use-case I'd need to tread <style> and <script> specially. This might also be a thing to consider from a security point of view. If this is too much -- no problem, this can be done in userland too by simply using https://github.com/WebReflection/linkedom to parse and extract

• this here would be a problem for me, as I need the HTML string but at the time being I wouldn't necessarily have a DOM node to inject it into:

await astro.renderToDOM(
  Component,
  { props, slots, request, params },
  document.getElementById('root')
)

So for me, it would be fine if the second argument was optional, but if there would still be a return value from renderToDOM.

For the return value, I'd really love something like this:

{
  html: string,
  styles: Array<string> // transpiled and processed CSS, in the right order so that it can safely be injected at the end of the html
  scripts: Array<string> // transpiled and built or simple inlined JS in the right order (can safely be injected at the end of the html and will be in the right order)
}

[bholmesdev]

@natemoo-re This goes beyond Storybook I believe. Client-based CMSes like Tina CMS and CloudCannon will likely try to use this for deploy previews as well. Regardless of support, we should have a really clear story for where this API is used!

[tylermercer]

FWIW, Storybook does support SSR, it's just that the API is a bit obtuse. But that could be something that could be smoothed over with an astro+storybook integration (which would depend on the proposed container API).

Using the container API client-side would probably still be necessary for TinaCMS etc as Ben pointed out.

[JReinhold]

This might be a stupid question, but Astro components are inherently static right? They don't need to be reactive to props changes, because they only ever run on build?

So if the props changes, it would be valid to just throw away the whole DOM tree and render a new string, we wouldn't need to special case it with keeping the Astro component state but with updated props?

(I hope you don't mind I'm spamming multiple messages, I think each question deserves its own thread)

[natemoo-re]

Correct! Astro components are inherently static, so I would probably handle updates one of two ways:

1. Throw away the old tree, set innerHTML to the incoming markup
2. Use something like micromorph to diff the old tree and the incoming markup.

[natemoo-re]

Also "spam" away! This is all super helpful feedback.

[matthewp]

I like it overall. I even like this just as an incremental improvement over internal APIs that we use for testing, even if this was just internal.

I think the Goals section could use some more use-cases. Testing is one that comes to mind for me, I don't know if this idea at all relates to the previous testing APIs that were discussed.

[natemoo-re]

Yes, this definitely unblocks the Unit Testing use-case and possibly even rendered MDX output for Content Collections

[JReinhold]

I'm thinking about components using the Astro global, and how that would work in an isolated context. You already talk about it in the description, describing the second parameter.

Would that allow all globals to be stubbed out, or only the subset described?
Eg. I can imagine both in testing and in playgrounds it would be valuable to stub out Astro.cookies, to listen to calls to cookies.set(), etc.

[natemoo-re]

Yes, all globals would be stubbed out but obviously some things like Astro.cookies would need to be noop'd

[lorenzolewis]

I'd be interested to see what Astro functions you'd imagine being usable while rendering in isolation vs. which ones aren't.

[matthewp]

Astro.cookies uses the request, so at least getting the cookies would still work. You might want to change the return signature (and maybe rename the method) so that it returns an object instead of just the HTML string. That way it could return a couple of things, maybe the cookies object itself being one.

[tanepiper]

I could see one use case that having access to this could be useful so I'd hope to see it - with Edge included (https://en.wikipedia.org/wiki/Edge_Side_Includes), you can render a fragment based on the current user such as an HTML fragment for an avatar and include this on each MPA page in page generation during the user session, instead of having to run an island first each time to check sessionStorage or call an API. This should still have access to client directives as we'd want to then enable a menu or update notifications for example

[lorenzolewis]

Possibly unblock .mdx compiledContent/html output?

If this means I could maybe have some Astro components pre-rendered in a reMARK plugin and then injected through the pipeline for a plain MD file and not only MDX then this would be amazing!

[natemoo-re]

Yes!

[JReinhold]

What can we expect in terms of components that import components from other frameworks, like React? Will that work out of the box? will the be interactive/hydrated or not? Or will it crash, or noop?

Is that what you are referring to here:

Support Astro framework renderers (@astrojs/*) if possible

[kyr0]

I guess it would work if the imported component (supported renderers) renders well on server side but not if it is client-only, right? Would be interesting to know the behaviour if it isn't rendered like client:only="preact" -- how would it hydrate, when renderToDOM just produces HTML? It would probably run into headaches what is to hydrate where? (out of place hydration); I think SolidJS solved that "addressing" of there to hydrate what using HTML comments and data-hk hydration keys, but it seems that this was quite hard nut to crack. It would also require a specific rehydration script on client side to pick the places where to hydrate and pass the server-side rendered state; and all that per renderer. I think this might be too much for the scope? Or maybe I'm missing something :))) But everything except client only would be great if it would render well.

[okikio]

Could we possibly build on this and create a spec for Astro components (of course, it'll have to be in a seperate RFC)?

[matthewp]

If there was interest in other implementations, that would be a reason for a spec.

[okikio]

Specing it out would allow for a standard syntax for Astro, and clear up any confusion on what parts of HTML & JSX are supported in Astro. I can imagine others using Astro's HTML like syntax for other usecases

[okikio]

Hold up would this allow React, Vue, etc... components to import Astro components? And would those components then be able to also import other React, Vue, etc... components through the Astro component?

Cause if true, we might just have a complete reveloutinary complete cross framework solution right here

[kyr0]

Same question #462 (comment) here? Also hydration is an important point here, if a component is client only

[okikio]

Yeah, but w/ portals, this should be possible to solve

[bholmesdev]

I have a similer sentiment to #462 (reply in thread). Bringing Astro rendering to the client feels like scope creep to me, and I think focusing on the server-side use case alone is best for an MVP. This could unlock endpoints that render Astro components as stringified HTML though. So rather than shipping all of Astro to the client, you can potentially fetch components as HTML from whatever backend you please.

[okikio]

Is there a difference between mode: "development" and mode: "production" a matter of optimization? If so, perhaps the API should have optimized: true option.

[Tc-001]

An app can behave differently in dev mode (like using a local api endpoint), so it is probably best for the user to explicitly set it. (because I assume it sets the import.meta.env.MODE value)

[JLarky]

I tested this and apart from the issue that someone noted (that scripts and styles are not set, so I just get html with a lot of "astro-XXX" class names and no styles for those 😅) I also wasn't able to make it work with new document collection api. I.e.

const { Content } = await doc.render();
renderToString(Content)

I really like the idea behind this API, but it might be too simplistic :)

My proposal would be something along these lines:

const result = renderTemplate(template)
const response = await resultToResponse(result) // if (response.statusCode === 404)
const {html, styles, scripts} = await resultToString(result)
// and probably something like resultToAsyncGenerator or resultsToReadableStream

[matthewp]

Page and component rendering a slightly different. If this was rendered as a page you would have the styles included. I think breaking this into multiple methods like:

container.renderPage(Component, args);

container.renderComponent(Component, args);

[kyr0]

@matthewp True, but Astro knows the styles and scripts also for components internally, right? I mean, if I'm not mistaken, Components carry their script and style dependency tree with them:
https://github.com/withastro/astro/blob/main/packages/astro/src/runtime/server/render/component.ts#L356

I was debugging that part of the codebase a few days ago and the return value of createAstroComponentInstance shows:

It would be amazing if the Container API would return what is available/known at the time and not limit the return value artificially to the rendered HTML only. Especially when rendering Astro components in isolation, I feel like it would be very helpful to get the styles and scripts that belong to that component, because when you build, for example, a component driven CMS for Astro or any other "I want to preview render a component" solution, you'd really be interested in the styles and scripts that belong to that very component you're rendering atm only. If you'd only get the styles when calling renderPage and then all the global ones..., well that would make isolated component preview rendering pretty problematic as one couldn't easily backtrack from "all styles of the page" to "belongs to this component".

[matthewp]

This needs to be figured out. Components don't carry their script/style dependencies with them. Astro figures that at at dev and build time and passes it into render. It's unclear if this layer would handle that or not.

[natemoo-re]

Yeah this is the main stumbling block with the proposal. Surfacing scripts and styles when rendering is critical for most real-world use cases. In my spike, there are some workarounds during dev but not during build. Haven't come up with an elegant solution yet.

[bholmesdev]

Really like this proposal overall! It feels aligned with the renderToString approach of popular UI frameworks like React, and covers the request and response use cases for testing. This should address the MDX compiledContent() discussion #419 as well 👏

[b3nten]

If this would let us render pages from requests other than GET that would help immensely.

[matthewp]

You can render pages from requests other than get already.

[b3nten]

How so?

[franco-onevillas]

How so?

Same question here!

[LopezNoah]

You can retrieve the method used for the request and then conditionally render the page using the given method. So, GET renders one thing but POST renders something else all in the same page route.

[tanepiper]

+1 on this - we've been looking at "Components as a Service" for working with headless content and Edge-Side Includes - instead of a page renderer.

I did a POC with it but because it's a page renderer it still needs all the possible available components. With this way we could instead render fragments on the fly instead in a function as we don't care about state.

[Tc-001]

🤔 This would also allow SPA-like navigation by generating API routes that can independently serve the page content without the layout

Still the issue with scripts not being unloaded after the navigation, but for simple docs sites and alike it would prob be enough.

[JLarky]

you don't have to use api to do that, just create separate pages that do not use layout :) like here https://github.com/JLarky/qgp.app/blob/main/src/pages/get_article/%5B...slug%5D.astro

[ematipico]

There's one part that is unclear to me, and the examples unfortunately don't help my brain.

If the APIs are meant to render astro files in isolation and we're outside the astro pipeline, that means that .astro files are not "well known" files, and they couldn't be imported with

import Foo from "Foo.astro"

Unless there's another pipeline - vite, webpack, etc - under the hood.

Am I wrong to assume that?

[tanepiper]

Good point, in terms of loading configuration, variables, plugins, etc I guess you would need to run it through some kind of vite/rollup setup?

I could think of examples where you want to possibly run a pre or post processor on things so you would still want to hook into the astro lifecycle.

[natemoo-re]

Right, this would either need a precompiled .astro file (already down-leveled to .js) or to be run inside an environment like Vitest that can use loader plugins.

[tanepiper]

When I was reading this proposal, I was thinking about the possibility of using some kind of snapshot functionality to be able to render the components inside, with some pre-existing state that can be used separately or merged with the passed state.

One question I couldn't resolve, and may be related to the above point above is how to include additional scripts and styles that are not included in global stypes.

One way is maybe to consider a post-processor that can slice the required fragment from inside a <body> tag on a page template, that can be configured with the required needs of that component - also include any required additional scripts and CSS in the fragment (I guess this might need manually configured unless it can be calculated somehow?) - this way a fragment is really a bundle of things outside the standard client context on the page at runtime.

I guess this could also be configurable regarding the selector? You could render a component in a specific template context and either still grab the body, or from a specific selector.

[dmmikkel]

Would be nice to be able to just pass in the Astro global object if you have one. I have a use case for rendering components to string inside the context of an Astro page request.

[Tc-001]

It would be nice if you could pass in the object you get when importing the astro.config file, but also it doesn't contain some things that would also be needed.

Maybe something like this?

import config from "../../astro.config.mjs"

const astro = new AstroContainer({
  ...config
  mode?: 'development' | 'production';
});

[Princesseuh]

This proposal has been accepted and moved into a Stage 2 proposal: #533

Thank you everyone for sharing your feedback on this first draft!