Cell prerendering + Prerender routes with parameters

[Tobbe]

This is an attempt at some kind of release notes. If it's too long, or too over-the-top, or just plain bad, please feel free to rewrite it however you like 🙂

Major new capability for Redwood's pre-rendering

With this release Redwood will now pre-render your Cells with data fetched from your GraphQL api at build-time. This means you can use Redwood as a proper Static Site Generator. Pre-render support for Cells gives you the possibility to generate static html pages for your blog posts, your e-commerce store front, your documentation, and much more! This gives you blazing fast pages for your users, and great SEO possibilities.

Read more in our docs https://redwoodjs.com/docs/prerender (when this PR is merged)

Conflicting files

It's possible, but highly unlikely, existing projects could have files conflicting with this feature. To make this feature work we've introduced a new file called XyzPage.routeHooks.{js,ts} located next to your page component file. So, if you already have files named like this and those files happen to export a function called routeParamters those functions will now be executed when RW tries to pre-render your pages.

---

Redwood uses ReactDOMServer.renderToString() when prerendering.
When using renderToString the React component tree is traversed once and then rendered to a static string. So what every component look like at their very first render is what will be prerendered to the final .html file. Basically, for every component that has some kind of loading state while it waits for data, you'll only ever going to get that loading state prerendered. Because of this you have to make sure all data is already fetched before trying to render (i.e. before calling renderToString). That way the component doesn't have to render the loading state, but can instead render it's full "data state" directly. So the full prerender process looks like a “waterfall”: fetch data (server) → render to HTML (server) → load code (client) → hydrate (client). This "waterfall" analogy is taken from reactwg/react-18#37 where you can read more about this problem (and also about how React 18 will help make this better).

More specifically for Redwood, what I'm trying to do here is make sure our Cells don't have to render their <Loading> component. Instead I want to have all data ready so they can render <Success> instead when we prerender.

The way I'm going about that is extracting all the gql queries from all cells and executing them before calling renderToString.

A few problems need to be solved

1. Find and extract all queries
2. Figure out what variables the queries need
3. Figure out what values those variables should have
4. Executing the queries with the same gql handler setup the user has in his/her app (same plugins, same directives etc)
5. Storing the execution result
6. Making the result available to the cells during prerender
7. Make the cells render <Success> with the stored query results

I've solved 1) and 2).

Problem 3) is more difficult. Very often the variable, and its value, comes from a path parameter. Like if you render this route <Route path="/blog-post/{id:Int}" page={BlogPostPage} name="blogPost" prerender /> at the url /blog-post/5 it's very likely you'd have a Cell that takes an id variable and expects the value to be 5. But you can't be sure. The path parameter id is passed to the BlogPostPage component. What that component does with it we have no control over. It could pass it on to some Cell (probably called BlogPostCell, but again, we don't know). But it could also do something entirely different with it, and use some other value for the id to the Cell.

Then we have user provided value. Like, there could be an input field on the page, and the value in that input could be needed as a variable value in a cell. Like say you have an address form, and the address the user provides is passed to a CurrentWeatherCell. How can we prerender that Cell?

A cell can also be used in multiple places on the same page, potentially with different variable values. So we'd have to be able to execute the cell query multiple times with different variable values and store all those results. So we can's just say "This is the Cell, and this is it's result data". It has to be more granular than that.

So I'm going to have to rewrite some of the stuff I've already written when I have a better plan for how to solve all the problems outlined above.

---

I tried just letting Apollo Client execute the gql query to see if I could intercept that to see exactly what it tried to fetch from the api side. But during the first render it doesn't even get that far. It never even reaches the ApolloLink execute() function in its execution path during first render. So that was a dead end. But if I ever do want to dig into how Apollo Client works again here are some notes: Look in the .cjs files in node_modules, those are what's executed when prerendering, not the regular .js files. Execution starts with InternalState.useQuery -> InternalState.useObservableQuery -> ApolloClient.watchQuery -> QueryManager.watchQuery -> new ObservableQuery. When running in the browser however, the .js files are used.

---

The idea I'm going with now is to render everything twice. The first pass we collect all queries and the variables that the page wants to execute. Then we manually run all those queries and store the results. Those results are then used in the second pass of rendering. This seems to work well so far. Still don't have any error handling, so need to add that.
From my original list of things to do, the only thing left is 4). Right now I'm using a very close approximation of what the user has, but not the exact same thing. One big thing left to do, that I forgot to add to my initial list, is to get all url path parameters. Tom has written about this problem before: https://community.redwoodjs.com/t/prerender-proposal/849

---

I'm getting pretty happy about the solution I've got going here.
Reading Tom's forum post it was suggested to have support for two prerender.js files. One at web/src/prerender.js and one at api/src/build/web/prerender.js. Since that proposal was written good things have happened with the framework. We've now got rw scripts (that you'd normally run with yarn rw exec). But we also have a special script in there - the seed.ts script for seeding the database. So to me it felt like a good fit to add a special prerender.ts file there as well. This file will be executed (just as if it had been run with yarn rw exec prerender) by yarn rw prerender and is expected to return a map with path parameters that will be used to generate the urls to prerender.
Example return value:

{
  blogPost: [ { id: 1 }, { id: 2 }, { id: 3 } ]
}

With that output, if you have <Route path="/blog-post/{id:Int}" page={BlogPostPage} name="blogPost" prerender /> in your Routes.tsx file three pages will be prerenderd: /blog-post/1, /blog-post/2 and /blog-post/3

I've also made it so that the project's own graphql handler() is used, ensuring all queries are executed exactly the same as when run live by the project.

---

Let's say you have a blog and you want to prerender the 10 latest blog posts (because no one ever looks past the front page anyway). This is what your prerender script could look like

import { db } from '$api/src/lib/db'

export default async function prerenderPathParameterValues() {
  const tenLatestPosts = await db.post.findMany({
    orderBy: {
      createdAt: 'desc',
    },
    take: 10,
  })

  return {
    blogPost: tenLatestPosts.map((post) => ({ id: post.id })),
  }
}

---

I've updated the test-project to also show post authors. The codemods for that were a pain to write, but now everything seems to be working on my machine at least.
The reason I've added this is to be able to test nested Cells. So now we first query for each post. And then, when the post renders its <Success> component, that component in turn will render an <AuthorCell>.
Prerendering still worked. I didn't have to make any changes at all to it. But it just renders "Loading..." for the nested cell. I'll see if I can make it render the <Success> state for that one as well. Danny calls this "the waterfall problem".

---

I've slightly updated the approach outlined above. To better follow best practices in the main "blog" part of the test-project I've made the post author<->post a proper relation. So I can now just nest my sql query to get the author of each post. So there is no need for a <AuthorCell>. Just need a simple <Author> component. So this instead of testing waterfall rendering became a great test to make sure we support relations/nested gql when prerendering. But then I also created a separate "waterfall" test page where I still have the nested cells, even though it's not technically needed in this case I still needed a test case for it, so I created a page (that's not linked to from the main blog layout) just for testing purposes that nests <AuthorCell> inside <WaterfallBlogPostCell>.

---

After discussing this implementation at a couple of Core Team Meetings we've decided to not use a /scripts/prerender.js file. Instead we'll colocate a XyzPage.renderData.js file next to the page components. So if you have web/src/pages/BlogPostPage/BlogPostPage.js you'll now need to add web/src/pages/BlogPostPage/BlogPostPage.renderData.js. This file should export a function called routeParameters. This function will receive the name of the route that's being rendered, and should return the path parameters to render that route for. So in the BlogPostPage example that function might look like this

import { db } from '$api/src/lib/db'

export async function routeParameters(routeName) {
  if (routeName === 'blogPost') {
    return (await db.post.findMany()).map((post) => ({ id: post.id }))
  }

  return []
}

Of course, if you know this page is only used for one route you don't have to look at the route name, so you can simplify the implementation to just this

import { db } from '$api/src/lib/db'

export async function routeParameters() {
  return (await db.post.findMany()).map((post) => ({ id: post.id }))
}

---

Getting naming right is difficult, so the discussions have continued. We've also realized this might tie into our future SSR support, as we might want to use the same file for that.

The top contenders for the file name are:
BlogPostPage.renderData.ts
BlogPostPage.serverData.ts
BlogPostPage.routeData.ts
BlogPostPage.routingData.ts
BlogPostPage.routeHooks.ts
BlogPostPage.routingHooks.ts
BlogPostPage.loader.ts

The function inside that file to return route parameters for pre-rendering:
routeParameters
getRouteParameters
routeParams
getRouteParams

Function for providing data when doing SSR
serverData
getServerData
renderData
getRenderData
routeData
getRouteData

In addition to the above suggestions/options we're still keeping an open mind about new suggestions coming in.

---

Left to do:

• Demo to core team
  • Everyone happy with the implementation I've gone with here? The /scripts/prerender.js file?
• Write tests for happy-path
• Test what happens when/if a gql query fails
  • Do we populate queryInfo.error? Should we? We don't and we shouldn't
• What happens if we have a page with nested cells, and the inner cell is only rendered after we've rendered <Success> in the first one?
  • Could we keep rendering and resolving queries until we don't find any more/new cell queries? Could that be a way to solve the waterfall problem? Yes! \o/
• Rip out the router updates from this PR and make a separate PR out of those Reorganize router utils #5659
• Update this PR branch, resolving merge conflicts
• Documentation
• Make author <-> user a proper relation in the test project
• Make a new <Author> component that's used in the AuthorCell success component
• Make BlogPost use <Author> instead of <AuthorCell> by nesting the query for author details in the main blogPost query
• Create a new page that's not linked from the main blog app that just tests nested cells where we can use <AuthorCell> nested in some other cell
• Add new prerender test for the testpage with nested cells
• Try prerendering only /blog-post/1
• Release notes
• Fix TS for $api imports in renderData.ts
• Decide on file- and function naming

[netlify]

✅ Deploy Preview for redwoodjs-docs ready!

Name	Link
🔨 Latest commit	4da78ad
🔍 Latest deploy log	https://app.netlify.com/sites/redwoodjs-docs/deploys/62e41c5e2f48e800097dc765
😎 Deploy Preview	https://deploy-preview-5600--redwoodjs-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[KrisCoulson]

with the path param name as the key, and the value for the parameter as the object value.

This is confusing. Maybe because we are describing an object or maybe its because "path param" but then use "parameter" right after but I think this could be clearer

[dac09]

Yes I agree.

[Tobbe]

Yeah, I wasn't super happy with the wording here... But Danny had a much better suggestion that I've now committed. Thanks Danny!

[KrisCoulson]

I think it might be work having an example with the db usage. Additionally, we should make it really clear that you need the $ to import the db. That is something that could be easily missed.

[Tobbe]

I added another example. One that uses the DB, and made a note about the import