gatsby-config/node.js doesn't support ESM import/export

[janosh]

Summary

Node v14 is out and with it the removal of the Experimental Modules warning! 🎉

I installed it, added "type": "module" to package.json, used export default in gatsby-config.js and took the default starter for a spin. This is what I got:

Error: [ERR_REQUIRE_ESM]: Must use import to load ES Module: gatsby-config.js
require() of ES modules is not supported.
require() of gatsby-config.js from node_modules/gatsby/dist/bootstrap/get-config-file.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename gatsby-config.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from package.json.

Sounds like Gatsby, specifically gatsby/dist/bootstrap/get-config-file.js, stands in the way of using ESM import/export syntax. I think it's time to make gatsby's config and node APIs ESM-ready. 🚀

Basic example

export default {
  siteMetadata: {
    title: `Gatsby Default Starter`,
    author: `@gatsbyjs`,
  },
  plugins: [
    `gatsby-plugin-react-helmet`,
    // ...
  ],
}

Motivation

Would be really cool if we could finally use a single import/export system all throughout our Gatsby projects! As the release announcement states:

It is our belief that the current implementation offers a future proof model to authoring ESM modules that paves the path to Universal JavaScript.

If that doesn't get you excited, you must be in the wrong line of work. 😄

[herecydev]

It's worth highlighting the following parts of the release docs:

However, the ESM implementation in Node.js remains experimental
Users should be cautious when using the feature in production environments

So I expect there's still some time before this will be implemented.

[janosh]

Fair enough. Although it's probably a good idea to start looking into this now.

As always, corporate users should wait to upgrade their production deployments until October when Node.js is promoted to LTS. However, now is the best time to start testing applications with Node.js 14, and try out new features.

[LekoArts]

Thank you for opening this!

As they say:

As per our stability index: “The feature is not subject to Semantic Versioning rules. Non-backward compatible changes or removal may occur in any future release.” Users should be cautious when using the feature in production environments.

So I don't think this is ready yet for prime-time in Gatsby as we can ship more impactful features than trying to keep up with breaking changes from them. We want to support it of course once it reached a stable status.

We're marking this issue as answered and closing it for now but please feel free to comment here if you would like to continue this discussion. We also recommend heading over to our communities if you have questions that are not bug reports or feature requests. We hope we managed to help and thank you for using Gatsby!

[thecodingwizard]

Node v15 lists ECMAScript module support as stable. Will Gatsby be supporting this anytime soon? Libraries such as xdm can't be used with Gatsby without this.

[tujoworker]

@thecodingaviator did you try it actually? Because I think you should be able to import ESM into CJS. You would require it from gatsby-node.js to run it on a certain Gatsby API.

[thecodingwizard]

For xdm specifically, I've tried a variety of techniques to import it into Gatsby (including using the esm package) and none of them worked. Did anybody have better success importing esm libraries into gatsby-node.js?

[kimbaudi]

I'm facing a similar issue as @thecodingwizard where I want to use the latest unist-util-visit v3.0.0 (which is ESM only) in Gatsby. I've also tried using the esm package and also adding "type": "modules" into my Gatsby package.json, but it doesn't work.

[ghost]

@LekoArts @KyleAMathews Node 16 is out, this is no longer experimental and should be reopened.

[wdavidw]

I'm facing a similar issue as @thecodingwizard where I want to use the latest unist-util-visit v3.0.0 (which is ESM only) in Gatsby. I've also tried using the esm package and also adding "type": "modules" into my Gatsby package.json, but it doesn't work.

Same problem here. Several gatsby-remark plugins are created inside the ./plugins folder of my project. They fail to require the latest unist-util-visit. I tried to convert them to .mjs and .cjs files without much success. The code work with unist-util-visit version ^2.0.3 and fail with ^3.0.1. I believe the problem is in how gatsby load local plugins more than in the unist-util-visit. This is an extract of the error:

 ERROR #11321  PLUGIN

"gatsby-transformer-remark" threw an error while running the createSchemaCustomization lifecycle:

Must use import to load ES Module: /home/user/website/node_modules/unist-util-visit/index.js
require() of ES modules is not supported.
require() of /home/user/website/node_modules/unist-util-visit/index.js from
/home/user/website/plugins/blog-md-url/index.js is an ES module file as it is a .js file whose nearest parent package.json contains
 "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename /home/user/website/node_modules/unist-util-visit/index.js to end in .cjs, change the requiring code to use import(),
 or remove "type": "module" from /home/user/website/node_modules/unist-util-visit/package.json.




  Error: [ERR_REQUIRE_ESM]: Must use import to load ES Module: /home/user/website/node_modules/unist-util-visit/index.js
  require() of ES modules is not supported.
  require() of /home/user/website/node_modules/unist-util-visit/index.js from /home/user/website/plugins/blog-md-
  url/index.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in th
  at package scope as ES modules.
  Instead rename /home/user/website/node_modules/unist-util-visit/index.js to end in .cjs, change the requiring code to use import(
  ), or remove "type": "module" from /home/user/website/node_modules/unist-util-visit/package.json.
  
  - errors:363 new NodeError
    node:internal/errors:363:5
  
  - loader:1126 Object.Module._extensions..js
    node:internal/modules/cjs/loader:1126:13
  
  - loader:989 Module.load
    node:internal/modules/cjs/loader:989:32
  
  - loader:829 Function.Module._load
    node:internal/modules/cjs/loader:829:14
  
  - loader:1013 Module.require
    node:internal/modules/cjs/loader:1013:19
  
  - v8-compile-cache.js:159 require
    [gatsby]/[v8-compile-cache]/v8-compile-cache.js:159:20
  
  - index.js:9 Object.<anonymous>
    /home/user/website/plugins/blog-md-url/index.js:9:15
  
  - v8-compile-cache.js:192 Module._compile
    [gatsby]/[v8-compile-cache]/v8-compile-cache.js:192:30
  
  - loader:1138 Object.Module._extensions..js
    node:internal/modules/cjs/loader:1138:10
  
  - loader:989 Module.load
    node:internal/modules/cjs/loader:989:32
  
  - loader:829 Function.Module._load
    node:internal/modules/cjs/loader:829:14
  
  - loader:1013 Module.require
    node:internal/modules/cjs/loader:1013:19
  
  - v8-compile-cache.js:159 require
    [gatsby]/[v8-compile-cache]/v8-compile-cache.js:159:20
  
  - create-schema-customization.js:45 
    [gatsby]/[gatsby-transformer-remark]/create-schema-customization.js:45:28
  
  - Array.forEach
  
  - create-schema-customization.js:44 Object.module.exports [as createSchemaCustomization]
    [gatsby]/[gatsby-transformer-remark]/create-schema-customization.js:44:11

Let us know if we need to open a new issue.

[LekoArts]

Please open a feature request for ESM support here: https://github.com/gatsbyjs/gatsby/discussions/categories/ideas-feature-requests

[paulmelnikow]

I found a couple of related and more specific feature requests (#31026 #31599) though I think it still makes sense to create a new feature request for this feature.

[Ir1d]

hi @kimbaudi have you found a solution for this? We're facing similar issues

[kimbaudi]

@Ir1d - I haven't found a solution to using import/export in gatsby-config/node.js. The closest I got to using import/export in gatsby-config/node.js is by using the esm npm package to require serializable js files and convert to es modules to use in gatsby-config/node.js. For example, I had config/site.js, which contained serializable javascript like so:

// site.js
const site = {
  title: `my title`,
  description: `my description`,
  url: `https://example.com`,
};

export default site;

and I use esm npm package to convert site.js from esm to cjs:

// esm.js

require = require('esm')(module)

const {default: site} = require('./site')

module.exports = {
  site
}

this way, I can use site (which is esm) in gatsby-config.js:

const {site} = require('./src/config/esm')

module.exports = {
  siteMetadata: site,
  ...
}

However, this no longer works if you try to convert .js/jsx to .ts/tsx files. I know you can convert .jsx files to .tsx in Gatsby, but esm npm package cannot read .ts files so I am unable to update my Gatsby sites to use TypeScript at the moment. I'm probably better off using Next.js at this point if I want to continue using my config files

[tujoworker]

@kimbaudi and @Ir1d I haven’t used it by my self, but this plugin looks promising, doesn’t it?

https://www.gatsbyjs.com/plugins/gatsby-plugin-ts-config/

[Ir1d]

We tried using esm's require, but it isn't of much success

[kimbaudi]

@tujoworker I'm aware of the plugin, but haven't used it myself.

@Ir1d esm require only works if the es module you are requiring is serializable. If esm's require doesn't work for you, it probably means you are requiring non-serializable js such as react components or something else I'm unaware of.

[Ir1d]

Thanks @kimbaudi. I'm trying to load unist-util-visit which comes from unified and mdx, similar to #31599

[kimbaudi]

yeah, that probably won't work since unist-util-visit references react components, which isn't serializable

[karlhorky]

Yeah, Gatsby not supporting ESM will become a big problem in the next weeks as people upgrade to the latest remark packages, which are full ESM.

@LekoArts would recommend the team reopening this one - it should be an issue, not a feature request.

cc @wooorm

[Ir1d]

indeed. afaik, no one has ever been able to make gatsby work with esm module unist-util-visit

[karlhorky]

One thing that I imagine would work would be to compile the gatsby-config.js and gatsby-node.js and all dependencies (including node_modules) first with webpack / esbuild, and then use that resulting file with Gatsby. 🤯

But setting that up... 😬

[Ir1d]

It requires complex configuration and I don't think that's what remark wanted when they decided to upgrade to esm..

[tujoworker]

Not «that» complex, see the this doc: https://support.gatsbyjs.com/hc/en-us/articles/1500000294121-Using-ES6-Module-Syntax-in-Gatsby-API-Files-on-Gatsby-Cloud

Sure, I agree, having that build in, would be very nice!

[karlhorky]

Not «that» complex, see the this doc: support.gatsbyjs.com/hc/en-us/articles/1500000294121-Using-ES6-Module-Syntax-in-Gatsby-API-Files-on-Gatsby-Cloud

This doesn't work - just tried it with the new pure ESM packages (for example, rehype-autolink-headings@6.0.0). You end up with the same errors (not being able to require ESM) as before.

Also, this was not my suggestion.

[karlhorky]

This isn't a "nice to have" or "has a workaround" - Gatsby just doesn't work with new versions of many packages.

[Ir1d]

@tujoworker thx, but that method doesnt work with remark ecosystem. i've tried that before.

[tujoworker]

Okay! There is at least some activity going on: https://twitter.com/wardpeet/status/1422843507283316736?s=21
Also, could this help? await import(dep)

[karlhorky]

Nope, dynamic import using await import() doesn't seem to work either. Top-level await in the gatsby-config.js file is not supported:

module.exports = {
  plugins: [
  ...
    {
      resolve: `gatsby-plugin-mdx`,
      options: {
        rehypePlugins: [
          [await import('rehype-autolink-headings'), { behavior: 'wrap' }],
                 ^^^^^^
  SyntaxError: Unexpected token 'import'

Without the top-level await, it would need to have some way of accessing the value in the .then callback, which it doesn't appear that Gatsby supports (eg. returning a promise-returning function as a value from gatsby-config.js)

[tujoworker]

You can return an function as the configuration, inside gatsby-config.js. I have done this at least. Not sure if it can be async 🤞

[karlhorky]

Yeah probably doesn't work with promises. I don't even see anything in the docs about the option to return a function.

[karlhorky]

Ah, the reason I can't find it in the docs is because returning a function in gatsby-config.js is only for themes:

@DSchau: Exporting a function from gatsby-config.js is reserved for themes. Are you trying to create a theme?

#16013 (comment)

And yeah, that doesn't support promises / async functions:

@sidharthachatterjee: Themes do currently accept a function (as you've mentioned) but we expect it to the sync.

#19644 (comment)

[bytrangle]

Has anyone found a way to make Gatsby work with pure ESM package like unist-util-visit ^3.0?

[karlhorky]

@bytrangle No known way currently. Gatsby is only compatible with packages that are NOT pure ESM. Downgrade to older versions for use with Gatsby.

The Gatsby team is apparently "not against" moving to ESM (tweet from @wardpeet), but I'm guessing it will still be some time before you can use this in your projects.

If you absolutely need to use the new versions now, you may want to try experimenting with bundling / precompiling the dependency to CommonJS, although this may be complex.

[karlhorky]

@wardpeet @LekoArts maybe this can be reopened - this will be a growing issue for Gatsby users as time goes on.

[LekoArts]

Please follow & comment on #31599 - it's on our radar but for now we have other things that are higher priority for us + we can't directly act on it without prior planning.

[karlhorky]

Ok, will do, maybe can you lock this thread then?

[LekoArts]

Please go to #34613 and you can use TypeScript with Gatsby. By the time you're reading this it's also possible that the feature already was shipped in Gatsby 4. If that is the case the RFC will be marked as done.

You can also expect ESM support (through this) very soon, please follow #31599 for progress. Again, if the feature is shipped it'll be noted in the discussion.