Parent/child link created from remote file node not persisted in cache

[Mrtenz]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

In my application I'm processing remote images using createRemoteFileNode from gatsby-source-filesystem. In onCreateNode I check if the node type of the created node is GhostPage (in this case), and download the feature_image (url) for that page. The newly created file node is then set on the GhostPage node using createParentChildLink.

The problem is that when running Gatsby with the nodes (supposedly) cached, the child files are not set on the GhostPage nodes, resulting in the following error:

Cannot query field "childFile" on type "GhostPage".

I'm able to reproduce this issue with a simple Yaml file and attaching it to the Yaml node as well, so it doesn't seem to be related to the Ghost CMS plugin. See minimal reproduction repo for an example.

From my understanding Gatsby only runs onCreateNode when the nodes changed, am I misunderstanding how I'm supposed to attach remote file nodes to another node, or is this a bug in Gatsby/gatsby-source-filesystem? I tried using node.file___NODE = remoteFileNode.id too, but that results in the same issue.

Reproduction Link

https://github.com/Mrtenz/gatsby-remote-file-node-issue

Steps to Reproduce

1. Create a remote file node and attach it to another node in onCreateNode
2. Run Gatsby
3. When it's done building, run Gatsby again without clearing the cache

Expected Result

The remote file node should be attached to the other node, and the application should run fine.

Actual Result

The remote file node is not attached to the other node, resulting in:

Cannot query field "childFile" on type "GhostPage".

Environment

System:
    OS: Linux 5.14 Arch Linux
    CPU: (16) x64 AMD Ryzen 7 3800X 8-Core Processor
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 14.18.1 - /tmp/yarn--1635866615094-0.2457912608307613/node
    Yarn: 1.22.17 - /tmp/yarn--1635866615094-0.2457912608307613/yarn
    npm: 6.14.15 - ~/.nvm/versions/node/v14.18.1/bin/npm
  Languages:
    Python: 3.9.7 - /usr/bin/python
  Browsers:
    Firefox: 93.0
  npmPackages:
    gatsby: ^4.0.2 => 4.0.2
    gatsby-source-filesystem: ^4.0.0 => 4.0.0
    gatsby-transformer-yaml: ^4.0.0 => 4.0.0

Config Flags

No response

[LekoArts]

Hi!

You're running into two things:

1. createParentChildLink is intended for another use case. It's purpose is to create derived nodes from a parent and not referencing related nodes. It's used in transformers primarily: https://www.gatsbyjs.com/docs/how-to/plugins-and-themes/creating-a-transformer-plugin/#creating-the-transformer-relationship
2. Since you're working in onCreateNode the direct mutation of the node won't no longer work with v4: https://www.gatsbyjs.com/docs/reference/release-notes/migrating-from-v3-to-v4/#dont-mutate-nodes-outside-of-expected-apis - so you're running into that change

You can fix it like this:

// gatsby-node.js
const { createRemoteFileNode } = require('gatsby-source-filesystem');

// Random cute picture
const FILE_URL = 'https://cdn.shibe.online/shibes/530e562bdd24954292fb3ce25aeb2d6180fb5d6e.jpg';

module.exports.onCreateNode = async ({ node, actions, store, cache, reporter, createNodeId }) => {
  const { createNode, createNodeField } = actions;

  // As an example, attach remote file node to `DataYaml`
  if (node.internal.type === 'DataYaml') {
    const fileNode = await createRemoteFileNode({
      url: FILE_URL,
      parentNodeId: node.id,
      store,
      cache,
      reporter,
      createNode,
      createNodeId
    });

    if (fileNode) {
      createNodeField({ node, name: 'localFile', value: fileNode.id })
    }
  }
};

exports.createSchemaCustomization = ({ actions }) => {
  const { createTypes } = actions

  createTypes(`
    type DataYaml implements Node {
      childFile: File @link(from: "fields.localFile")
    }
  `)
}

This is explained here: https://www.gatsbyjs.com/docs/how-to/plugins-and-themes/creating-a-source-plugin/#optimize-remote-images

[Mrtenz]

@LekoArts Thank you for the prompt reply. I started this project in Gatsby 4 so didn't check the migration guide for v3 to v4, but it makes sense that that wasn't an intended feature in Gatsby. I considered doing something with createNodeField and setting up a custom GraphQL resolver, but using @link is much easier for sure.

Did some testing with your example however, and it seems that the remote file node isn't being persisted to the cache? I ran the following query twice (with restarting in between) after implementing your changes in the minimal reproduction repo:

{
  allFile {
    nodes {
      name
    }
  }
  allDataYaml {
    nodes {
      childFile {
        name
      }
    }
  }
}

The first time it returns this as expected:

{
  "data": {
    "allFile": {
      "nodes": [
        {
          "name": "foo"
        },
        {
          "name": "530e562bdd24954292fb3ce25aeb2d6180fb5d6e"
        }
      ]
    },
    "allDataYaml": {
      "nodes": [
        {
          "childFile": {
            "name": "530e562bdd24954292fb3ce25aeb2d6180fb5d6e"
          }
        }
      ]
    }
  },
  "extensions": {}
}

Both the Yaml file (foo) and image (530e562bdd24954292fb3ce25aeb2d6180fb5d6e) exist. After running it for a second time though, it returns this:

{
  "data": {
    "allFile": {
      "nodes": [
        {
          "name": "foo"
        }
      ]
    },
    "allDataYaml": {
      "nodes": [
        {
          "childFile": null
        }
      ]
    }
  },
  "extensions": {}
}

The image node doesn't exist, and childFile returns null. Am I missing something here? How do I make sure the remote file nodes are being recreated?

[LekoArts]

Whoops, a small typo crept into the code. It should be this:

    const fileNode = await createRemoteFileNode({
      url: FILE_URL,
	  parentNodeId: node.id,
      store,
      cache,
      reporter,
      createNode,
      createNodeId
    });

Otherwise the files will be garbage collected

[Mrtenz]

Thanks, everything seems to work perfectly now. And the build time is reduced from 3 minutes to 30 seconds, which is great!

[DennisKraaijeveld]

@LekoArts Maybe I should open a separate thread, but it is a similar question. How would you add a parent/child link in this setup, when using the buildObjectType api?

// Add a "fields.localImages" field to Shopify Product integration fields. It
// contains a list of File nodes that lets `gatsby-transformer-sharp` process
// the images.
exports.onCreateNode = async ({
  node,
  actions: { createNode, createNodeField },
  createNodeId,
  reporter,
  getCache,
  store,
}) => {
  // Look for Shopify Product integration nodes.
  if (
    node.internal.type ===
    "PrismicHomepageDataBodySmallProductGridItemsProductgridMax3IntegrationType"
  ) {
    const images = node.images
    console.log(images)

    // For each image, use createRemoteFileNode to download the image and save
    // in a new node.
    const imageNodes = await Promise.all(
      images.map(async (image) => {
        if (image.src) {
          return await createRemoteFileNode({
            url: image.src,
            parentNodeId: node.id,
            createNode,
            createNodeId,
            reporter,
            getCache,
            store,
          })
        }
      })
    )

    // We will reference each downloaded image node by ID.
    const imageNodeIds = imageNodes.map((node) => node.id)

    // Add a field called "fields.localImages" to the Shopify Product
    // integration node.
    createNodeField({
      node,
      name: "localImages",
      value: imageNodeIds,
    })
  }
}

// Tell Gatsby that `fields.localImages` is a list of File nodes.
exports.createSchemaCustomization = ({ actions, schema }) => {
  const shopifyProductType = schema.buildObjectType({
    name: "PrismicHomepageDataBodySmallProductGridItemsProductgridMax3IntegrationTypeFields",
    fields: {
      localImages: {
        type: "[File]",
        extensions: { link: {} },
      },
    },
  })

  actions.createTypes(shopifyProductType)

[Ruben-deBruijn]

Probably this issue has been resolved already for a long time, but i did it like this:

    schema.buildObjectType({
      name: 'MyType',
      interfaces: ['Node'],
      fields: {
        image: {
          type: 'File',
          extensions: {
            link: {
              from: "fields.image"
            }
          }
        },
        images: {
          type: '[File]',
          extensions: {
            link: {
              from: "fields.images"
            }
          }
        }
      }
    })