Universal blocks and block hydration

[michalczaplinski]

TL;DR

When you create a dynamic block, there is duplication of logic between PHP and JS code. I propose that we should try to achieve "universal blocks" that could be written once and run in both PHP and JS with the help of a compiler. Additionally, I explore how this idea intersects with block hydration, which is a way to add interactivity to server-rendered blocks. Finally, I show a basic demo of compiling blocks authored in Mitosis format to both Liquid templates and React components that can hydrate the server-rendered markup.

The challenges

1. Duplicate logic in PHP and JS when writing dynamic blocks (universal blocks)

When writing dynamic blocks, there is often necessary duplication of logic between the edit function of the block (written in javascript) and the function called by the render_callback on the server side, typically located in the index.php file of the block. Post Date core block is a simple example of a block with such duplication.

Unsurprisingly, everyone agrees that this duplication is bad. The users would like to see some API which would allow them to write a single template that could be rendered both on the client and the server. I'll refer to this idea as "universal blocks" throughout the post 🙂

There has already been a fair bit of discussion about this problem in many other places:

• Render dynamic blocks on the front end in JavaScript instead of PHP - the main github issue on the topic of "universal blocks". Various people suggest various strategies: Rendering on client only, using a templating language like Handlebars, using webcomponents...
• https://github.com/youknowriad/blocky - An experiment by @youknowriad where he explores the concept of "universal blocks" - very interesting. It allows you to create a block using a single JSON file where you define your "template" as well as any attributes and properties, like in a block.json file. AFAIK, this is also where he coined the term "universal blocks"! 😃
• https://github.com/mcsf/prototype-block-generator - An experiment by @miguelcsf to generate block HTML from a preact component tree and a template.
• https://twitter.com/markjaquith/status/1432461667187765251 - A thread with a discussion of making block creation easier.
• https://wptavern.com/a-world-where-some-block-development-is-merely-a-templating-system-with-no-build-process - An article about the above twitter thread
• https://github.com/markjaquith/BlockStub - An experiment in writing blocks with PHP which would dynamically generate React code.
• Live Q & A: How can we make building custom blocks easier? - a video of a panel discussion on the topic of "universal" blocks with Birgit, Helen, Mark and Riad.

2. Make it easier to create interactive blocks (block hydration)

Users would like to create blocks that use JS to create interactive experiences.

Woocommerce has been able to work with existing constraints to create interactive blocks that are being rendered only on the client using React, lazy-loading and some clever techniques. This is a great solution, but we should find a way to support their use case with an even better out-of-the-box experience.

Finding a solution to this challenge should involve figuring out how to hydrate the blocks generated in PHP. Why? For many blocks, only rendering on the client is not a good solution because it makes for a less-than-ideal UX (you have to wait for all client-side JS to load in order to render your block). So, ideally, we'd like to render the content on the server and then hydrate it a.k.a. "make interactive" using JS on the client. How? Well... that's the challenge 🙂.

Define dynamic blocks using a high-level block format and compile them to a format that Gutenberg understands

This is a solution proposed in blocky. I think this idea is worth exploring more. To recap: Riad proposed using plain JSON to define blocks, but we could push this idea even further and use a more sophisticated block format and a compiler!

But first, we gotta ask ourselves: "What are our constraints"?

• The system that we design should be completely opt-in. That is, the current Gutenberg API and ways to create blocks have to continue to work. This means that the system should work "on top of" the current Gutenberg API.
• We do not have a JS runtime, i.e. node on the server. We can only assume that the block creator has node installed and can use it only at build time (when authoring blocks).
• We cannot use custom PHP extensions beyond ones listed in https://make.wordpress.org/hosting/handbook/server-environment/#php-extensions. This means that running V8JS to server-render react components is out of the question.
• Bonus: It should be possible to re-use blocks outside of Gutenberg, e.g in Drupal. This use case has been mentioned by Riad in the Q&A and it ties into the discussion of "universal blocks" he explored in blocky. If we can make blocks in a Gutenberg-agnostic way, they could potentially be used in other contexts.
• Bonus: Our blocks should work with any framework on the frontend. This means that even though we create blocks with React, they don't have to be React components when they are used on the frontend. Maybe someone is using Vue or Preact in their WordPress site and would like their block to work with the rest of their site. Maybe the blocks could even be Web Component that could be used with any framework (but I'm getting ahead of myself here 😅).

Given those constraints, I've also briefly considered leveraging WebAssembly to compile the QuickJS javascript engine to WASM and run it using wasmer-php runtime. However, this turns out to also require a custom PHP extension so is a non-starter at the moment.

The demo

https://github.com/michalczaplinski/hydration-mitosis-demo

I've created a demo here which allows you to create Gutenberg blocks using Mitosis format and compile them to both a Liquid template AND a React component that hydrates the server-rendered HTML.

There is more information in the repo, including what I'd consider are limitations and weaknesses of this approach 🙂

Other ideas

There have been other ideas floated around with respect to "universal" blocks so I'd like to give them credit here as well.

Render blocks on the client only

I think this idea is probably a non-starter for SEO, performance and UX reasons, but I include it here for completeness as it does technically solve some challenges. It was proposed here.

Define blocks in PHP

This is the approach proposed by @markjaquith in https://github.com/markjaquith/BlockStub. I think this is an interesting idea, especially as it might be more approachable to PHP developers over more JS-focused approaches, but I believe it might be hard to scale beyond a "hello world" - I feel that writing JS inside of PHP might be the worst of both worlds though... You still have to write some JS for the interactive parts of your app but you forgo all the DX goodness that come with modern JS dev environment and just concatenate strings in PHP. On the plus side though, this approach would require no build step.

Conclusion and open questions

We are still very early with ideas for "universal blocks" and block hydration so I would 💕 some feedback here and any ideas that might have occurred to anyone. I also want to stress that the idea to use Mitosis is only an experiment and it is meant to serve as an example of the kind of direction that we could take here and NOT a suggestion to adopt it inside Gutenberg.

Those are some of the questions that I still have lingering in my mind:

• I wonder if there is a place for web components to play a role here, like originally suggested by @felixarntz . It would be great if we could compile our new block format to web components. This way, we could have framework-agnostic and browser-native block elements. The challenge here I think is finding a way to server-render web components in PHP - not sure if possible, but would love see this investigated more!
• How could we extend this work to enable creating blocks that mutate some state on the server? For example like a "checkbox" block or a "Comment Form" block? An approach similar to React Server Components is one idea. I have fiddled with an experiment that allows you to create server components in PHP with Gutenberg and I feel like it's not that promising. Perhaps a better way would be to explore something like Laravel Livewire .
• If we were to go this route, what direction would it make sense to explore with the new block format? Should we try to create our own format or adopt a format that already exists (Mitosis, Svelte, or something else)?

[michalczaplinski]

I'd like to note another approach I have considered but ultimately decided against, because I've stumbled across a PR to Jetpack where it was already tried 🙂.

The idea is to prerender React components to html templates at build time using React's renderToStaticMarkup. This approach works because at build time, the user has access to nodejs (JS on the server) and we can execute renderToStaticMarkup(block) where block is our shared block. You also have to pass some placeholders for your props that can later be "filled out" at request time on the server.

// This runs at build time. In the actual code we wouldn't save the 
// file manually but rather let webpack generate it for us. 

const markup = renderToStaticMarkup(
    <UpgradeNudge planName="#planName#" upgradeUrl="#upgradeUrl#" />
);

saveTemplate(markup, "_inc/blocks/upgrade.html");

// This runs on the server at request time.
// The actual code is a little more complicated to also handle i18n and some edge cases.

ob_start();
include JETPACK__PLUGIN_DIR . "_inc/blocks/upgrade.html";
$markup = ob_get_clean();
foreach ( $props as $key => $value ) {
	$markup = str_replace(
		"#$key#",   // <---- like `planName` in the code snippet above
		$value,
		$markup
	);
}

echo $markup;

However, this approach would only work for very simple blocks where the props do not determine the structure of the final HTML. If you have a component like this:

const DogsList = ({ dogs }) => (
    <ul>
        {dogs.map(({ name }) => (
            <li>
                {name} {name === "Max" ? "is NOT a good dog." : "is a good dog!"}
            </li>
        ))}
    </ul>
);

this approach would not work because you cannot know before request-time how many dogs there are and if Max is one of them 🙂.

In order to be able to generate a template that could handle such components we would need a full-blown compiler that could turn the { dogs.map(({ name }) => component } into an equivalent structure in PHP or into a template (like Mitosis does).

[michalczaplinski]

The proposal in OP (and the related demo) is one specific way of attacking the problem of block hydration.

I'd like to step back and note some more abstract steps or "building blocks" that I think going to be necessary for a full solution. This is a non-exhaustive list, of course 🙂

1. Use the same template on the client and server.
   Whatever this template is, be it JSX or PHP or Liquid (like in the OP demo) It doesn't even have to be the "same" as in "identical", but rather "compatible".

2. Identify the dynamic and static parts of the template.
   There are many approaches here. We use compilation in the OP, but others have used some type of light marking of the dynamic "holes" in the static templates (for example this was the approach tried in Jetpack that I've mentioned earlier)

3. Enable looping and conditionals in the template.
   Mitosis has the control flow components for that purpose. Each templating language has a flavour of those constructs and with JSX you use regular JS constructs to achieve this in userland.

4. Make the dynamic parts interactive on the client
   This is the actual "hydration" part of "block hydration" :)

[johnhooks]

@michalczaplinski to your knowledge have any of these projects defined a block AST?

[johnhooks]

This is my summary of the problem to see if I understand it correctly.

The power of JSX is that it is just JavaScript after transpiling. Though, this makes using it as a templating language for PHP very problematic because it practically requires transpiling JS to PHP.

Creating a new templating language or using an existing one would complicate the ecosystem of React based blocks.

The common target is serialized blocks.

Server rendering is required to generate markup for good SEO and quickly displaying properly formatted content on initial page load. The interactively has to be able to hydrate from this markup.

The solution needs to render in PHP, hydrate in JS, and remain compatible with the React block ecosystem.

[luisherranz]

The solution needs to render in PHP, hydrate in JS, and remain compatible with the React block ecosystem.

Good summary 🙂

[johnhooks]

Thanks for the confirmation I'm thinking of this correctly,

[johnhooks]

My thoughts on the problem.

The concept

• Create a JSX to PHP compiler, half of the job is already done. Robust AST can be generated from JSX.
• Start with a subset of JSX functionality.
• Use ESLint rules to help authors write complaint JSX.
• Start small, but eventually the functionality could become quite robust.

The process

1. Generate PHP templates from JSX that accept the block attributes as props
2. Assign as the render_callback of a block type.
3. Hydrate islands of blocks in the DOM, using the same function that generated it.

Pros

• Write views in JSX, the same way it is currently done.
• Define frontend interactivity in React.
• Retain the entire ecosystem of React and JS.

Cons

• Complex - There will be a mountain of edge cases.
• Confusion - The view would appear to be standard JSX, a developer unaware of the difference could easily mistake it for regular JSX and become frustrated. (This could be partially circumnavigate by helpful hints from the compiler and wp-scripts)
• Third Party React Libraries - One of the awesome powers of React is the expansive functionality provided by third party code. Ensuring it can be appropriately compiled will most likely require a complete JS to PHP compilation.

The questions this raises

Could the save function of a block type also be used to defined the client side interactivity, as well as, continue to fulfill its role as the save function in the editor?

Is loading React on the frontend an option?

Revision

However, there's a fundamental issue: the HTML saved in the database gets modified during WordPress rendering. - @luisherranz

The save function cannot be used to both save the serialized block to the database and rehydrate the block on the frontend of a site, the markup is past through filters on the server that can modify the final output. The changes applied by the filters would be removed as soon as the block rehydrates.

[johnhooks]

@gziolo oh yeah, I understand this is not going to be a quick fix. But I'm interesting and plan to try to help with this problem. @luisherranz I said it would be "hard" to accept but I might be leaning that direction, the wrinkle of filters is something I didn't consider. I'm still getting used to how "pluggable" WordPress is!

[johnhooks]

A templating system that compiles to React and PHP and uses directives for all interactivity. Do you think this could be possible? To write blocks in some other templating language and still use editor components and connect to the Redux store in the edit function? (Sorry, I'm going to keep framing this in a way to have a unified view authoring experience)

[gziolo]

We are currently exploring whether it would make sense to bring the concept of directives also to JS by leveraging it in edit and save in #48590. In general, we need to build lower-level primitives we can use as compilation targets, as this will inform the direction for the templating system people know from using Astro, Svelte, Solid, or Vue.

[johnhooks]

Thank you for helping me find the relevant Issues in the repo, it really helps. It's been hard to find a comprehensive explanation of the problem and the current solutions being considered.

[johnhooks]

I've only been in the WordPress space for two months. But the first thing I wanted to do after creating my first block was to add interactivity to the frontend. It has take a while to grasp why doing that is difficult, learn it is an acknowledged problem, and that work is being done to find a solution.

[johnhooks]

Correct me if I'm wrong. But I haven't seen a detailed Make blog post about this issue. Or detailed explanations of the proposed solutions. Do you think there is a more discoverable way to post this, rather than it being mostly contained in GitHub Issues and Discussions?

CC: @gziolo @michalczaplinski @luisherranz

[johnhooks]

I know this post exists, but it's pretty vague on the details.

https://make.wordpress.org/core/2022/04/27/exploration-to-enable-better-developer-and-visitor-experiences-with-blocks/

[gziolo]

Great timing! You can expect a very detailed update on Make Core blog this week that will cover many concepts that are essential for universal blocks.

[johnhooks]

I'm excited to read it!

[michalczaplinski]

Here's the blog post 🙂 https://make.wordpress.org/core/2023/03/30/proposal-the-interactivity-api-a-better-developer-experience-in-building-interactive-blocks/

[johnhooks]

Thank you! That was great, and I really appreciate everyone taking the time to walk me through it, even with the big announcement post just about to go out.

[johnhooks]

The solution needs to render in PHP, hydrate in JS, and remain compatible with the React block ecosystem.

Revision

The solution needs to render in PHP, remain modifiable by WordPress filters on the server, hydrate in JS, and remain compatible with the React block ecosystem.

[johnhooks]

Functional views are not a good fit for the use case of content that can be modified by a filter. React is the opposite concept. When the same things go in, same thing comes out. The system is broken if the output of the function can be modified. The same inputs no longer provide the same output.

So, a template DSL that compiles to JS templates isn't compatible with PHP filters, and WordPress heavily relies on them.
@luisherranz

I missed that on the first read through. I think I get it now.

[michalczaplinski]

It looks like people keep (re)discovering this discussion, so I'd like to clarify how my thinking about Universal blocks has evolved over the last two years:

A group of contributors has been working on the Interactivity API over the last year which is available in Wordpress 6.5. This API is a direct result of our thinking about how to make it easier to create interactive blocks (Challange 2. in the OP)

This API is available today, is compatible with all PHP filters, and works great with server-rendered content, unlike all of the React solutions suggested in this discussion.

That said, there is still a possible path to a universal block compiler in the long term. Such a compiler could, in principle, take as input something like UI state + template and return both the frontend and backend representations. Those could roughly be:

1. An interactive JS store using Interactivity API.
2. The server definition of the store (link).
3. PHP code that renders the HTML with directives.

However, no team is working on this at the moment and it might never happen. It largely depends on the adoption of the Interactivity API and the community feedback.

[ts-mz]

As per your last sentence - I wonder if having this kind (even development thereof) of universal block compiler would not also help increase the adoption of the interactivity API?

[michalczaplinski]

Hard to say 🙂