behind the scenes docs #7510

Moocar · 2018-08-21T08:09:06Z

I found myself having to relearn how the guts of Gatsby work every time I make a change. So I created some docs to stop myself having to learn from scratch each time.

I'm sure some of it could be moved into other parts of the docs (e.g ___NODE and frontmatter___date). But most isn't important for regular users. Just those looking to learn about how Gatsby works behind the scenes.

I've linked in quite a few places to lines of code on Github. I'm 50/50 on this approach. It's great to link from high level descriptions of the code straight to the code itself, but I'm worried these links will become stale. Open to suggestions here.

Lots more to document, but it's a start. Hopefully others knowledgeable with the code base can add to it over time.

I've also used the new gatsby-remark-graphviz to display graphs, which has the benefit of being able to source control graphs via a language vs an image drawing application.

Was causing gatsby build to freeze on createPages

Moocar · 2018-08-27T09:51:50Z

Closing this PR as I've updated the docs significantly. Will reopen when they're pushed.

m-allanson

Woah this is great stuff @Moocar!

I started reviewing this but tbh it's mainly nitpicking. It probably makes more sense to merge this in, then followup with any changes if that sounds good to you?

m-allanson · 2018-09-04T15:24:23Z

docs/docs/behind-the-scenes.md

+title: Behind the Scenes
+---
+
+Curious how Gatsby works under the hood? This pages in this section describe how a Gatsby build works from an internal code/architecture point of view. It should be useful for anyone who needs to work on the internals of Gatsby, or for those who are simply curious how it all works, or perhaps you're a plugin author and need to understand how core works to track down a bug? Come one, come all! 


Typo

- This pages in this section + The pages in this section

m-allanson · 2018-09-04T15:25:01Z

docs/docs/schema-generation-behind-the-scenes.md

+
+During the schema generation phase, we must generate what's called a `ProcessedNodeType` in Gatsby. This is a simple structure that builds on top of a [graphql-js GraphQLObjectType](https://graphql.org/graphql-js/type/#graphqlobjecttype). Our goal in the below steps is to infer and construct this object for each unique node type in redux.
+
+The flow is summarized by the below graph. It shows the intermediate transformations that are performed by code in the Gatsby [schema folder](https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby/src/schema), finally resulting in the [ProcessedNoteType](https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/schema/build-node-types.js#L182). It uses the example of building a `File` GraphQL type.


Typo

- resulting in the [ProcessedNoteType] + resulting in the [ProcessedNodeType]

m-allanson · 2018-09-04T15:28:55Z

docs/docs/schema-generation-behind-the-scenes.md

+  "resolve" [ label = "ProcessedNodeType.resolve()", shape = box ];
+  "gqlType" [ label = "gqlType (GraphQLObjectType)\l{\l    fields,\l    name: `File`\l}\l ", shape = box ];
+  "parentChild" [ label = "Parent/Children fields\lnode {\l    childMarkdownRemark { html }\l    parent { id }\l}\l ", shape = "box" ];
+  "objectFields" [ label = "Object node fields\l  node {\l    relativePath,\l    accessTime   }\l  }\l  ", shape = "box" ];


Looks like there's a spare closing bracket here:

m-allanson · 2018-09-04T15:30:16Z

docs/docs/schema-generation-behind-the-scenes.md

+digraph graphname {
+  "pluginFields" [ label = "custom plugin fields\l{\l    publicURL: {\l        type: GraphQLString,\l        resolve(file, a, c) { ... }\l    }\l}\l ", shape = box ];
+  "typeNodes" [ label = "all redux nodes of type\le.g internal.type === `File`", shape = "box" ];
+  "exampleValue" [ label = "exampleValue\l{\l    relativePath: `bogs/my-blog.md`,\l    accessTime: 8292387234\l}\l ", shape = "box" ];


less bogs, more blogs :)

- relativePath: `bogs/my + relativePath: `blogs/my

My favorite typo

m-allanson · 2018-09-04T15:32:47Z

docs/docs/schema-generation-behind-the-scenes.md

+
+#### 2. Create a "GQLType"
+
+This step is quite complex, but at its most basic, it infers GraphQL Fields by constructing an `exampleObject` that merges all fields of the type in Redux. It uses this to infer all possible fields and their types, and construct GraphQL versions of them. It does the same for fields created by plugins (like in step 2). This step is explained in detail in [GraphQL Node Types Creation](/docs/schema-gql-type).


update step number?

- (like in step 2) + (like in step 1)

m-allanson · 2018-09-04T16:03:18Z

docs/docs/schema-gql-type.md

+
+#### Foreign Key reference (`___NODE`)
+
+If not a mapping field, it might instead end in `___NODE`, signifying that its value is an ID that is a foreign key reference to another node in redux. Check out the [Create a Source Plugin](/docs/create-source-plugin/#create-source-plugin) for how this works from a user point of view. Behind the scenes, the field inference is handled by [inferFromFieldName](https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/schema/infer-graphql-type.js#L204).


+ Check out the [Create a Source Plugin] - Check out [Create a Source Plugin]

m-allanson · 2018-09-04T16:07:11Z

docs/docs/schema-gql-type.md

+}
+```
+
+To resolve `file.childMarkdownRemark`, we filter over all the node's (that we're resolving) `children`, until we find one of type `markdownRemark`, which is returned. Remember that in redux, `children` is a collection of IDs. So as part of this, we lookup the node by ID in redux too.


- all the node's + all the nodes

I think this one is correct as is, but the brackets between node's and children confuse it. I'll rearrange the sentence.

m-allanson · 2018-09-04T16:11:19Z

docs/docs/schema-gql-type.md

+
+#### parent field
+
+When a node is created as a child of some node, that fact is stored as a `parent` field on the redux node whose value is the ID of the parent. The parent GraphQL field resolver simply looks up the resolving node's `parent` field, and then looks up the parent in the redux `nodes` namespace map. It also creates a [page dependency](TODO have this in own section), as is done by child resolvers.


I found this paragraph quite confusing. I understood it as the parent field is added to the parent node, which doesn't seem right?

m-allanson · 2018-09-04T16:51:57Z

docs/docs/schema-sift.md

+
+- field key is`elemMatch`? Change to `$elemMatch`. Recurse on value object
+- field value is regex? Apply regex cleaning
+- field value is globl, use [minimatch](https://www.npmjs.com/package/minimatch) library to convert to Regex


- globl + glob

m-allanson · 2018-09-04T16:53:56Z

docs/docs/schema-sift.md

+
+The `resolve()` function calls `run-sift.js`, and provides it with the following arguments:
+
+- GraphQLArgs (as js object). Within a filter. E.g `wordcount: { paragraphs: { eg: 4 } }`


There's a few eg's instead of eq's following this:

- eg: 4 + eq: 4

…l-docs

Moocar · 2018-09-05T05:13:24Z

@m-allanson Sounds good! Yeah, reviewing 1300+ lines of documentation is no easy feat. Thanks for the initial pass. I agree we should just merge it and fix typos/clarifications etc on the fly.

m-allanson

👌

calcsam

Per earlier conversation, I'll 👍 this so you can merge when you're ready @Moocar

Moocar · 2018-09-06T11:20:09Z

@calcsam Thanks :). But looks like I might need elevated privileges to merge it.

fk · 2018-09-10T11:17:57Z

🎉 ❤️

Moocar added 23 commits August 15, 2018 20:24

add gatsby-remark-graphviz package

a0393d3

replace README SVG with image to guarantee it will display

30dc68b

Replace README rendered image with rendered svg

d9b9caa

formatting/linting

13b8d03

use engine render option instead of lang

0c2f3c4

fixed bug where top level promise callback was never resolved

d0fbe30

Was causing gatsby build to freeze on createPages

Merge branch 'master' into gatsby-remark-graphviz

f9c79ec

fixed bug where top level promise callback was never resolved

998ff5c

Was causing gatsby build to freeze on createPages

add detail about prismjs to remark-graphviz

6754bbe

add schema behind the scenes

37e77d9

add error handling to viz.js execution

dfad957

Merge branch 'gatsby-remark-graphviz' into internal-docs

7da1c04

add first graph

0b714c2

moar behind the scenes docs

884a132

add more behind the scenes sections

1b7f329

separated plugins/apis docs

87567fe

APIs running diagram

c7fea22

disable unfinished docs

5e6d9d1

more internal docs

879fe11

Merge branch 'master' into internal-docs

18455da

more docs updates

27c212e

add graphviz dependency

916d64b

fixed overview: true

dd58c75

Moocar requested review from KyleAMathews and kkemple August 21, 2018 08:12

Moocar added the type: documentation label Aug 21, 2018

Moocar added 2 commits August 21, 2018 18:22

formatting

b84db49

All caps heading

4572b98

Moocar closed this Aug 27, 2018

Moocar reopened this Aug 28, 2018

m-allanson added type: documentation An issue or pull request for improving or updating Gatsby's documentation and removed type: documentation labels Aug 28, 2018

Moocar and others added 11 commits August 30, 2018 12:21

finished schema connections doc

436166d

more linking

0e0cf51

typo

8b13cd9

Merge branch 'master' into internal-docs

541964d

Docs for query behind the scenes

ba666a2

revert accidental sites changes

f09800a

add graph for how query execution works

f2497f9

Merge branch 'master' into internal-docs

52e9c4e

added query extraction graphs

27d9b65

Added File Type stuff

20f6d5b

Merge branch 'master' into internal-docs

98b862d

m-allanson previously approved these changes Sep 4, 2018

View reviewed changes

Moocar added 3 commits September 5, 2018 13:39

typos

2c7d512

Merge branch 'master' into internal-docs

41262cb

Merge branch 'internal-docs' of github.com:Moocar/gatsby into interna…

2aac2df

…l-docs

Moocar dismissed m-allanson’s stale review via 2aac2df September 5, 2018 04:04

m-allanson previously approved these changes Sep 5, 2018

View reviewed changes

Moocar added 2 commits September 6, 2018 11:34

More behind the scenes docs

4be9afd

Merge branch 'master' into internal-docs

58fd41e

Moocar dismissed m-allanson’s stale review via 58fd41e September 6, 2018 02:14

Moocar added 2 commits September 6, 2018 13:09

add internal data bridge docs

a0aeb71

filled in page creation docs

f5e53ba

calcsam approved these changes Sep 6, 2018

View reviewed changes

Moocar merged commit ca7d0ed into gatsbyjs:master Sep 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

behind the scenes docs #7510

behind the scenes docs #7510

Moocar commented Aug 21, 2018 •

edited

Loading

Moocar commented Aug 27, 2018

m-allanson left a comment

m-allanson Sep 4, 2018

m-allanson Sep 4, 2018

m-allanson Sep 4, 2018

m-allanson Sep 4, 2018

Moocar Sep 5, 2018

m-allanson Sep 4, 2018

m-allanson Sep 4, 2018

m-allanson Sep 4, 2018

Moocar Sep 5, 2018

m-allanson Sep 4, 2018

m-allanson Sep 4, 2018

m-allanson Sep 4, 2018

Moocar commented Sep 5, 2018

m-allanson left a comment

calcsam left a comment

Moocar commented Sep 6, 2018

fk commented Sep 10, 2018


		During the schema generation phase, we must generate what's called a `ProcessedNodeType` in Gatsby. This is a simple structure that builds on top of a [graphql-js GraphQLObjectType](https://graphql.org/graphql-js/type/#graphqlobjecttype). Our goal in the below steps is to infer and construct this object for each unique node type in redux.

		The flow is summarized by the below graph. It shows the intermediate transformations that are performed by code in the Gatsby [schema folder](https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby/src/schema), finally resulting in the [ProcessedNoteType](https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/schema/build-node-types.js#L182). It uses the example of building a `File` GraphQL type.


		#### 2. Create a "GQLType"

		This step is quite complex, but at its most basic, it infers GraphQL Fields by constructing an `exampleObject` that merges all fields of the type in Redux. It uses this to infer all possible fields and their types, and construct GraphQL versions of them. It does the same for fields created by plugins (like in step 2). This step is explained in detail in [GraphQL Node Types Creation](/docs/schema-gql-type).


		#### Foreign Key reference (`___NODE`)

		If not a mapping field, it might instead end in `___NODE`, signifying that its value is an ID that is a foreign key reference to another node in redux. Check out the [Create a Source Plugin](/docs/create-source-plugin/#create-source-plugin) for how this works from a user point of view. Behind the scenes, the field inference is handled by [inferFromFieldName](https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/schema/infer-graphql-type.js#L204).


		#### parent field

		When a node is created as a child of some node, that fact is stored as a `parent` field on the redux node whose value is the ID of the parent. The parent GraphQL field resolver simply looks up the resolving node's `parent` field, and then looks up the parent in the redux `nodes` namespace map. It also creates a [page dependency](TODO have this in own section), as is done by child resolvers.


		The `resolve()` function calls `run-sift.js`, and provides it with the following arguments:

		- GraphQLArgs (as js object). Within a filter. E.g `wordcount: { paragraphs: { eg: 4 } }`

behind the scenes docs #7510

behind the scenes docs #7510

Conversation

Moocar commented Aug 21, 2018 • edited Loading

Moocar commented Aug 27, 2018

m-allanson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Moocar commented Sep 5, 2018

m-allanson left a comment

Choose a reason for hiding this comment

calcsam left a comment

Choose a reason for hiding this comment

Moocar commented Sep 6, 2018

fk commented Sep 10, 2018

Moocar commented Aug 21, 2018 •

edited

Loading