1.0 Programmatic Routes

[KyleAMathews]

Gatsby currently is too magical when creating paths. It tries to auto-generate paths based on files' positions on the file system. So a file named my-sweet-blog-post.md becomes /my-sweet-blog-post/. Which is fun and works but often you want more control.

So with Gatsby 1.0, I'm planning that all paths will be created programmatically.

Within plugins and at the site level you can create a function called createRoutes. This gets called with a graphql function. With that you write queries to get data and then return an array of route objects with paths and the component responsible for that path.

Gatsby takes this route information and auto-generates a React Router config.

The beauty of static site generators is you know everything at build time. So you can calculate exactly what paths are needed.

I want to add support for purely client-side routes as well if you're loading data dynamically from an API but for server rendered stuff, we can calculate all routes ahead of time.

This means you're no longer limited to file-based routes and can easily do stuff like pagination or tag pages or a 1000 other things.

A simple example for a blog.

import _ from 'lodash'

exports.createRoutes = (graphql, cb) => {
  const paths = []
  graphql(`
    {
      allMarkdown(first: 1000) {
        edges {
          node {
            path
          }
        }
      }
    }
  `)
  .then(result => {
    const blogComponent = './pages/article-route.js'
    let routes = []
    // Create blog post routes.
    _.each(result.data.allMarkdown.edges, (edge) => {
      routes.push({
        path: edge.node.path,
        component: blogComponent,
      })
    })
    cb(null, routes)
  })
}

This is extra setup compared to what we have now but is still fairly straightforward and combined with the new GraphQL data layer, 1000x more flexible.

Also plugins and themes can provide default route creation for you so you can just install a blog theme and just start dropping markdown files in a content directory. Or install a pagination plugin and tell it to create /page/1, /page/2 (as many as needed) with 10 blog posts per page.

[KyleAMathews]

This has been prototyped and is part of the first 1.0 alpha. Very new though so please give feedback.

[lourd]

Hey @KyleAMathews, how's it going. First time caller, long time listener 😉 Gatsby looks like a really interesting project. Nice job! 😃

To repeat what I think you're proposing for clarity: A module that implements the createRoutes method can instruct Gatsby to render the html for every possible path reachable at build time. The prime use case is generating these pages programmatically based on I/O data – external APIs, file access, etc. Does that sound right?

Where do you think the cost of doing that becomes too big? For a blog made with Gatsby that has several thousand posts, Gatsby would generate several thousand different html files. Is that correct?

Where do you see the line between statically and dynamically generating a site?

[KyleAMathews]

With some optimizations even sites with many thousands of pages shouldn't take an unreasonable amount of time e.g. < 10 minutes.

It should be possible in the future to do incremental builds for only the subset of the site that's changed since the last build so for even huge huge sites, incremental builds would be fast.

So the main reason you should choose a dynamic site is if there's a logged-in personalized experience where either a) the number of users is really high meaning generating a page for each one of them would be prohibitively expensive or b) user data changes often e.g. a social network so you need live data.

But for most sites that don't change too often, static is great for all the perf benefits. I'm not entirely sure what the practical limits with Gatsby are but would definitely be interested in exploring those and pushing them back as far as possible!

[lourd]

Aside from the personalized experience, it doesn't seem like systems that need dynamic creation of data would work well.

For example, an anonymous bulletin board system where anyone can type in a note. How would that work? You type in your note and hit submit on the form... where does that data go? How does data flow to instruct Gatsby to render the new page?

[KyleAMathews]

Yeah, anytime data can change quickly and needs to be reflected immediately are problematic.

Solutions could be a) use a hosted service for the dynamic part e.g. Disqus is a common blog commenting solution and b) if incremental rebuilds can get down to a few seconds, dynamic data actually becomes possible!

But practically speaking I wouldn't build any sort of social network or anything with dynamic (esp user driven) data on a static site generator.

[ChristopherBiscardi]

@KyleAMathews For those situations Leo is considering using Relay 2's "GraphQL Extensions" support to do something like deferred queries that go to a comment server at runtime. I haven't figured out how this plays out in figuring out the static vs dynamic line, but will be investigating it more in depth in the coming months.

As for createRoutes, the approach is interesting. In Leo this is a "post-process" step so that you can derive pagination from the raw dataset. The benefit being that a "tags" page can just be a Router route with a gql query, whereas pagination really needs to be generated from the data. Will have to mull it over a bit since Gatsby "generates" RR-Routes whereas Leo places that in the "user's" hands.

[KyleAMathews]

Yeah how do you do code splitting with Leo? Since Gatsby knows what routes there are, the new stuff can automatically code split for you on a per-route basis.

[SachaG]

I think programmatic routes would be awesome (I used them a lot with Middleman, for things like discount codes for example: https://www.discovermeteor.com/blog/three-middleman-hacks-were-using-on-this-site/), but I also really like the current simplicity of the automatic routing.

So I think it'd be great to keep the current pages directory functionality, but also add a more explicity API for route management.

[KyleAMathews]

I'm hoping to keep most of the simplicity of the current arrangement. So far the plan is /pages remains and Javascript pages still automatically have routes created for them. For other common file => page stuff, there'll be themes which provide this sort of functionality out of the box. So for a blog theme, it'd include the routing code to auto-convert markdown files into blog post pages.

Basically the change here is whereas right now (markdown) => wrapper(page) => HTML, in 1.0 wrappers are completely under your control. It's still react components wrapping data but now the "wrappers" pull in data as they choose from the GraphQL schema you setup. This gives you a ton more flexibility and power. But also, once you've set things up or by installing a theme, the marginal cost of changing things around is still very low because you can drop in a new file or move a file and things just update.

[anthonysapp]

Agreed - on the original post. It does seem "too magical" to me. Even though you can easily do custom routes via the file system (awesome) the routes aren't hierarchical (it's always a flat structure). Is there a good way to hack this right now?

[KyleAMathews]

@anthonysapp the best would be to auto-generate files at the routes you want e.g. for tag pages for a blog, write a script which pulls out all the tags on the site and then writes out the info you want for a tag page (e.g. the title and path to each post with that tag) to json files. Not the most gainly of solutions but it gets the job done.

[KyleAMathews]

@anthonysapp also it is possible to nest _template.js's. See for example https://github.com/gatsbyjs/gatsby-starter-documentation/blob/master/pages/docs/_template.jsx

[anthonysapp]

@KyleAMathews thanks for the replies! I had looked at the docs examples with the nested template. That should probably work for the case I am thinking about. Just to confirm - essentially, this looks at the current route and checks the static "docPages" object defined in config to see if there's a child route, then pushes it into the router, correct? I could be reading it wrong. I plan to dig into this a bit deeper in the coming days.

[KyleAMathews]

The docPages object isn't relevant. That's just for the left sidebar navigation. How the _template stuff works is everything under a _template file is nested under it in the React Router routing table.

[KyleAMathews]

@SachaG it just occurred to me that it'd be possible to create a compatibility plugin for the wrapper functionality in 0.x. Itd create paths for markdown and other files and look for the wrapper components and inject the same props. Thata the cool thing about programmatic routes that any logic is possible.

[skipjack]

@KyleAMathews so is there a workaround in v1 to continue using filesystem based paths for markdown files? It seems for .jsx / .js files within /pages the routes are still "magically" generated, but not for markdown files. I understand the rationale behind taking it out by default but I think it would still be helpful to have a plugin or some other workaround that allows populating path dynamically based on the filesystem rather than hardcoding the path into every markdown file's frontmatter.

[JLongley]

@skipjack You just have to configure it yourself with the markdown transformer.

Take a look at the starter: https://github.com/Vagr9K/gatsby-advanced-starter/blob/master/gatsby-node.js

He creates routes by first creating a ‘slug’ property on the node in the onCreateNode function in gatsby-node.js, then using that property when setting the paths of the new routes in the createPages function.
You can use this logic to set the route to the path of the file by just storing the path of the file. something like this:

exports.onCreateNode = ({ node, boundActionCreators, getNode }) => { 
  ...
  const parsedFilePath = path.parse(fileNode.relativePath);
  ...
  createNodeField({ node, name: "slug", value: parsedFilePath });
}
...
exports.createPages = ({ graphql, boundActionCreators }) => {
...
  createPage({
    path: edge.node.fields.slug,
    ...

Make sense? You can customize the logic here to store your files in any directory structure you want, and massage your paths any way you want. In this starter, he uses branching logic to first check the markdown frontmatter to see if the slug is overridden, and then falls back to using the title of the post otherwise.

[skipjack]

@JLongley thanks for the detailed description and yeah I follow what you're saying (someone mentioned something similar to me on the discord chat when I first ran into this). I ended up moving in another direction after evaluating Gatsby and a few other SSG options though I may play around with it again at some point. Thanks for your help!

[KyleAMathews]

Shipped in v1!

[arggh]

@KyleAMathews Today I tried Gatsby. I picked the official blog starter, installed everything as I was told and fired up. After resolving some NPM issues with missing node-gyp I got Gatsby running, all good.

I then added a few old blog posts in markdown and they showed up perfectly with pretty URL's. Nice!

Then, like everybody does, I skipped most of the docs and wanted to add something, quickly. I wanted tags! So I pretty much copy-pasted code from the docs under section Adding Tags and Categories to Blog posts and restarted Gatsby.

No luck. Gatsby is complaining about missing path.

{ GraphQLError: Cannot query field "path" on type "frontmatter". Did you mean "date"?

Now, to somebody who knows Gatsby, this is probably an easy fix. However, having no idea whatsoever of the magic offered by Gatsby, I kind of thought path was something infered from the file structure. After all, the markdown posts in the starter repo didn't have any pathfields set and they worked?

So I began my quest of digging through the docs, which I spent a good 45 minutes reading. I couldn't find any mention about how the paths get formed, where they come from and how can I modify them. Maybe it's too late and I'm too blind, but I had to read the source code of few starter templates, read a blog post / tutorial on Gatsby from Auth0 and finally stumble across this issue to actually find the answer.

I didn't file an issue (almost did, twice), because I thought I'm just blind, stupid or too tired to understand.

So, maybe this should be explained in the tutorial, or did I miss something?

Edit: Now I found a mention, above one of the code samples it says "This example assumes that each markdown page has a “path” set in the frontmatter of the markdown file." in here @ docs

[KyleAMathews]

@arggh sorry you're having troubles! We highly recommend everyone go through the official tutorial which discusses these things in depth gatsbyjs.org/tutorial/

[arggh]

No troubles, I just had hard time finding what I'm looking for in the documentation with reasonable effort and I was hoping my feedback would maybe help you improve the docs!

I did browse through the tutorial, but since I was looking for help on a specific issue, I quickly searched each page for a reference to path. There wasn't any! path isn't mentioned in the tutorial, not once.

Then again, path is used in the code samples, at least the ones regarding Adding Tags and Categories in the documentation, but it is not explained where path is actually coming from.

Given that I had started my quest with Gatsby using the official blog starter, which contains sample posts in markdown which do not contain the path field in their frontmatter, it was even more confusing.

So, my suggestion (and feel free to not care) is to:

• explain clearly where the path is coming from when it's used in the documentation's code examples
• ...and/or use path field in frontmatter everywhere, at least in the official Gatsby starters (3), if the code samples in the documentation expect it to be present
• ...or have some other way of accomplishing the situation where strictly following the instructions in the documentation on how to add tags would actually work on the official Gatsby blog starter.

Just to be clear, I'm trying out Gatsby out of curiosity and trying to be helpful here! Thanks for building nice things!

[arggh]

These lines of code in the gatsby-starter-lumen pretty much explain what I wanted to know (without reading a ton of stuff):

// gatsby-node.js
let slug = fileNode.fields.slug;
if (typeof node.frontmatter.path !== 'undefined') {
   slug = node.frontmatter.path;
}