Support ESM usage with Gatsby

[wdavidw]

Problem

Parts of the Node.js ecosystem are moving their packages to ESM and importing/using them together with Gatsby currently doesn't work. Users should be able to use ESM-only packages.

Solution

Please visit ESM in Gatsby files to learn how to use ESM with Gatsby.

[Ir1d]

Hi. @wdavidw have you found solution on this? We are facing similar situations

[wdavidw]

No I haven't found any solution. I have tried using esm, eg const visit = require('esm')(module)('unist-util-visit').default without much success.

[Ahacad]

unifiedJS seems decided to go full ESM: syntax-tree/unist-util-visit#23 (comment), when would ESM be supported by Gatsby? A lot of plugins of gatsby relies on remark and the lack of support is causing issues.

[Ir1d]

@LekoArts @wardpeet Excuse me, do you have any idea on this? Thanks.

[karlhorky]

No known way to do this currently. Gatsby is only compatible with packages that are NOT pure ESM. Downgrade to older versions for use with Gatsby.

The Gatsby team is apparently "not against" moving to ESM (tweet from @wardpeet), but I'm guessing it will still be some time before you can use this in your projects.

If you absolutely need to use the new versions now, you may want to try experimenting with bundling / precompiling the dependency to CommonJS, although this may be complex.

[karlhorky]

Update: I got the update to rehype-slug@5.0.0 in my gatsby-config.js working by installing esm and patch-package and:

1. Modifying the gatsby-config.js as follows (to allow require() with the pure ES Modules)

   +require = require('esm')(module);
   module.exports = {
     plugins: [
       {
         resolve: `gatsby-plugin-mdx`,
         options: {
           rehypePlugins: [
   -          require('rehype-slug'),
   +          require('rehype-slug').default,

2. Try running the Gatsby dev server and see what breaks. It will throw an error either like this:

     Error: Must use import to load ES Module: /Users/k/p/project/node_modules/rehype-slug/index.js
     require() of ES modules is not supported.
     require() of /Users/k/p/project/node_modules/rehype-slug/index.js from /Users/k/p/project/gatsby-config.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
     Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /Users/k/p/project/node_modules/rehype-slug/package.json.
   

   ...or like this:

     Error: /Users/k/p/project/gatsby-config.js:1
     Error: Must use import to load ES Module: /Users/k/p/project/node_modules/rehype-slug/index.js
   

3. Make a note of the location of the pure ES Modules package so that you can locate the package.json file corresponding to the pure ESM package and manually remove the line with "type": "module", like this:

   -  "type": "module",

4. Run yarn patch-package <pkg name> --exclude '^$', which will record the changes that you have made in the package.json file (to be reapplied by patch-package every time that you run yarn or npm install). Examples:

   # For a top-level dependency
   yarn patch-package rehype-slug --exclude '^$'
   
   # For a transitive dependency
   yarn patch-package rehype-slug/unist-util-visit --exclude '^$'

5. Run the Gatsby dev server again, and repeat step 3 and 4 for every pure ESM package.

At that point, I could start the Gatsby dev server with the new pure ESM dependency versions.

cc @Ir1d @kimbaudi @wardpeet @LekoArts

---

Caveat: In my specific case, I needed to also patch one of the dependencies (hast-util-heading-rank) because I was getting an undefined node.tagName, but I think this is unrelated to this problem - probably related to a different issue.

[ownchoice]

That was quite a lot of work (around 10 patches) and feels gimmicky, but it works! Thank you.

[karlhorky]

Totally, it's not a great solution, but it works for now, until Gatsby supports ESM.

I want to try out these two solutions below, they seem simpler:

• Support ESM usage with Gatsby #31599 (comment)
• Support ESM usage with Gatsby #31599 (comment)

[simplenotezy]

Thanks for sharing. This was the exact reason why I found this discussion in the first place. Would be great to have fixed.

[natapg]

@karlhorky were you able to update remark and remark-html with gatsby 4.9? I've migrated to TS my gatsby-node.ts and I get this error after changing to import remark from 'remark'

Error in ".../.cache/compiled/gatsby-node.js": require() of ES Module
.../node_modules/remark/index.js from
.../.cache/compiled/gatsby-config.js not supported.
Instead change the require of index.js in .../.cache/compiled/gatsby-config.js to a dynamic import()
which is available in all CommonJS modules.

[karlhorky]

For Gatsby 4.9 and TypeScript configs, see the post below: #31599 (comment)

[wardpeet]

I'm slowly but surely looking into a strategy to make this possible. I can't put a date on it but just wanted to share that we're looking into it. We will first start with some utils and then move to plugin support.

[wdavidw]

@wardpeet if you need feedbacks, I am finishing to migrate the csv package to esm. I spent some time figuring out the right architecture. Our stack is js + .d.ts files for the source, using package.exports to expose modules in both cjs and esm versions (we also have a browser version), Mocha.js + Should.js, with tests written in CoffeeScript and Typescript.

[diauweb]

We are currently using a solution that compile ES module using esm then forcing node to require it by catching ERR_REQUIRE_ESM
This way will not need any extra changes to external dependencies from npm
It is still a workaround whatever but it works
https://gist.github.com/diauweb/f9f7be271ecc7bc0aecaf3dc891e18d2

[karlhorky]

Interesting approach!

And I guess this is how you use it from code:

OI-wiki/gatsby-oi-wiki@3f782b2

[natapg]

@karlhorky were you able to use it with remark and remark-html?

[FezVrasta]

I started getting this error all of a sudden, with a yarn.lock file and yarn --frozen-lockfile installation...

 ERROR #98123  WEBPACK

Generating development JavaScript bundle failed

[BABEL] /Users/federicozivolo/Projects/popper-website/.cache/polyfill-entry.js: React Refresh Babel transform should only be enabled
 in development environment. Instead, the environment is: "production". If you want to override this check, pass {skipEnvCheck:
true} as plugin options. (While processing:
"/Users/federicozivolo/Projects/popper-website/node_modules/gatsby/node_modules/react-refresh/babel.js")

My website is broken and there's no way for me to fix it, I tried to upgrade the deps but it got even worse since I get the ESM incompatibilities issue reported here.

[Lukespacewalker]

@FezVrasta this may help you.
This is workaround I found on how to use ESM with gatsby 4 + yarn 2 (but PnP must be disable)
edit .yarnrc.yml to disable PnP as ESM is not currently compatible with yarn 2 PnP

yarnPath: .yarn/releases/yarn-3.1.1.cjs
nodeLinker: node-modules
enableGlobalCache: true

esmRequire.js put this file in the root folder (same place as package.json)

 const esm = require('esm')
 const fs = require('fs')
 const Module = require('module')
 
 // Node: bypass [ERR_REQUIRE_ESM]
 const orig = Module._extensions['.js']
 Module._extensions['.js'] = function (module, filename) {
   try {
     return orig(module, filename)
   } catch (e) {
     // if (e.code === 'ERR_REQUIRE_ESM') { <-- I comment this because in my system this error is not throw.
       const content = fs.readFileSync(filename, 'utf8')
       module._compile(content, filename)
    // }
   }
 }
 
 const _esmRequire = esm(module, {
   cjs: true,
   mode: 'all',
 })
 
 // don't pollute Module
 Module._extensions['.js'] = orig
 
 
 module.exports = function esmRequire(id) {
   return _esmRequire(id)
 }

Then use it in gatsby-config.js like this

const esmRequire = require('./esmRequire')
module.exports = {
   ....... omitted code ......
    plugins: [
        {
            resolve: 'gatsby-plugin-sass',
            options: {
                sassOptions:{
                    includePaths: ['src/styles']
                },
                implementation: require('sass')
            }
        },
        {
            resolve: `gatsby-plugin-mdx`,
            options: {
                extensions: ['.mdx', '.md'],
                remarkPlugins: [esmRequire("remark-slug").default]
            }
        },
        ....... omitted other plug-ins ......
   ]
   ....... omitted code ......
}

[barbalex]

I guess this is why I have not managed to run tests using https://playwright.dev/ and keep getting errors concerning .mjs, not being able to find modules, modules not being defined in ES module scope when trying to run tests and not being able to set "type": "module" in package.json.

I spent a few hours trying to get it work. But am giving up.

I would appreciate if someone could tell me if this could be because of this issue or if I am simply doing something wrong?

[tujoworker]

Gatsby does support by default ESM with .js via Babel - this issue is about config and dependencies in served as ESM only.

.mjs is a native Node.js thing.

Running Playwright end-to-end tests on “serverd” pages should work without touching anything Gatsby related.

[karlhorky]

With the new TypeScript configs from Gatsby 4.9, it's possible to patch only a single Gatsby internal file with all of the ESM packages, much easier!

#34613 (reply in thread)

[phil-lgr]

Hi @karlhorky thanks for the link

I have local plugins in TS that are not under site/plugins/

function constructParcel(siteRoot) {
  return new _core.Parcel({
    entries: [`${siteRoot}/${gatsbyFileRegex}`, `${siteRoot}/plugins/**/${gatsbyFileRegex}`],
    defaultConfig: require.resolve(`gatsby-parcel-config`, {
      paths: [siteRoot]
    }),
    mode: `production`,
    targets: {
      root: {
        outputFormat: `commonjs`,
        includeNodeModules: false,
        sourceMap: false,
        engines: {
          node: `>= 14.15.0`
        },
        distDir: `${siteRoot}/${COMPILED_CACHE_DIR}`
      }
    },
    cacheDir: `${siteRoot}/${PARCEL_CACHE_DIR}`
  });
}

any idea on how to include those files in the parcel compilation?

Thanks

[karlhorky]

The way I did it in the link above was to use patch-package to patch the line with includeNodeModules in the file called node_modules/gatsby/dist/utils/parcel/compile-gatsby-files.js

Not sure how it works with a local module, but maybe just refer to the name of your local package?

You could also consider using something like Yarn Workspaces, which will then treat your local plugin as a normal module within node_modules

[DevSide]

I'm actually sad that Typescript is ready before native NodeJS ESM.

[raae]

Is this on the roadmap yet?

[LekoArts]

We've started work on this, you can follow the milestone: https://github.com/gatsbyjs/gatsby/milestone/15 (no ETA!)

We'll first make gatsby-config.mjs and gatsby-node.mjs work correctly, afterwards gatsby-config.ts and gatsby-node.ts will follow.

Once we have a version published you can try out you'll find all details in the umbrella discussion - you can then also post your feedback there.

[axe312ger]

Yes pls. Can't wait to load ESM in my TS loaders

[tyhopp]

Hi! A couple questions for everyone we'd like your feedback on:

1. Do you use primarily ESM features (e.g. import/export modules, import.meta, etc.), CJS features (e.g. require, module.exports, __filename, __dirname, etc.) or both in your gatsby-config.ts and gatsby-node.ts files? If you can, please share a gist!
2. What do you expect to work in your gatsby-config.ts and gatsby-node.ts files? All ESM and CJS features? One or the other? A subset of either?

[nkuehn]

concering 2.:
I want to be able to use 3rd party ESM-only but also CJS distributed libraries in gatsby-node.js (or ts / cjs for that matter) without inofficial hackery . No emotions attached what specific import / export / require / include form this implies as long as it does not require messing with babel config or the like.

Notable example: All the "unified" markdown processing ecosytem is ESM-only now but many other key libraries are CJS-only distributions. If I want to add the heading structure of an MDX source into the GraphQL using MDX v2 plugins I need a powerful markdown processor. Currently I have to lock them to old pre-ESM versions.

gatsby-config.js is not as critical, it's not really dependent on powerful 3rd party libraries in our case.

[karlhorky]

2. I would suggest that gatsby-config.ts and gatsby-node.ts should support the same module features that the official TypeScript compiler (tsc) supports, instead of creating some new custom level of support, which will lead to confusion and pain.

[LekoArts]

Please see #37069, we now have instructions on how you give gatsby-node.mjs and gatsby-config.mjs with gatsby@next a try.

[tik9]

Does esm work in a js gatsby project? I get the following error when starting develop, I npm i esm and added gatsby-node.esm.js with content of gatsby-node.js:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /Users/thome/tire/.cache/tmp-86565-2GrsbTzxTLRk. Loading extensionless files is not supported inside of "type":"module" package.json contexts. The package.json file /Users/thome/tire/package.json caused this "type":"module" context. Try changing /Users/thome/tire/.cache/tmp-86565-2GrsbTzxTLRk to have a file extension. Note the "bin" field of package.json can point to a file with an extension, for example {"type":"module","bin":{"tmp-86565-2GrsbTzxTLRk":"./.cache/tmp-86565-2GrsbTzxTLRk.js"}}

[ZeldOcarina]

I am getting this error too. Is there any workaround @LekoArts?

[tyhopp]

The error doesn't look familiar since we don't use esm internally. We do expect the name of the Gatsby node file to be either gatsby-node.js or gatsby-node.mjs in https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/utils/import-gatsby-plugin.ts#L40-L51 and https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/bootstrap/resolve-js-file-path.ts, so one guess is it might be an issue with the name gatsby-node.esm.js. Does it work if you change it to gatsby-node.js?

Alternatively you can switch to gatsby-node.mjs and drop the esm package if you want to use ESM packages directly.

[luckpot207]

ERROR

Error in "/root/smutt_hole/Gatsby_ellipsis_system/node_modules/gatsby-source-hubspot/gatsby-node.js": require() of ES Module
/root/smutt_hole/Gatsby_ellipsis_system/node_modules/gatsby-source-hubspot/node_modules/node-fetch/src/index.js from
/root/smutt_hole/Gatsby_ellipsis_system/node_modules/gatsby-source-hubspot/gatsby-node.js not supported.
Instead change the require of index.js in
/root/smutt_hole/Gatsby_ellipsis_system/node_modules/gatsby-source-hubspot/gatsby-node.js to a dynamic import() which is
available in all CommonJS modules.

Error: [ERR_REQUIRE_ESM]: require() of ES Module /root/smutt_hole/Gatsby_ellipsis_system/node_modules/gatsby-source-hubspot/no
de_modules/node-fetch/src/index.js from /root/smutt_hole/Gatsby_ellipsis_system/node_modules/gatsby-source-hubspot/gatsby-node
.js not supported.
Instead change the require of index.js in /root/smutt_hole/Gatsby_ellipsis_system/node_modules/gatsby-source-hubspot/gatsby-no
de.js to a dynamic import() which is available in all CommonJS modules.

[axe312ger]

Well, the error message clearly tells you the limitations:

You need to use a dynamic import OR switch your whole file to use the new syntax. Renaming to .mjs should help :)

[luckpot207]

ya , i have done
but not resolved

[piorus]

I'm trying to set up ESM on my gatsby website but it fails with

> my-gatsby-site@1.0.0 develop
> gatsby develop

node:internal/errors:490
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /home/peter/sites/my-gatsby-site/.cache/tmp-71701-szMR8p4vhwap. Loading extensionless files is not supported inside of "type":"module" package.json contexts. The package.json file /home/peter/sites/my-gatsby-site/package.json caused this "type":"module" context. Try changing /home/peter/sites/my-gatsby-site/.cache/tmp-71701-szMR8p4vhwap to have a file extension. Note the "bin" field of package.json can point to a file with an extension, for example {"type":"module","bin":{"tmp-71701-szMR8p4vhwap":"./.cache/tmp-71701-szMR8p4vhwap.js"}}
    at new NodeError (node:internal/errors:399:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:79:11)
    at defaultGetFormat (node:internal/modules/esm/get_format:121:38)
    at defaultLoad (node:internal/modules/esm/load:81:20)
    at nextLoad (node:internal/modules/esm/loader:163:28)
    at ESMLoader.load (node:internal/modules/esm/loader:605:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:457:22)
    at new ModuleJob (node:internal/modules/esm/module_job:64:26)
    at #createModuleJob (node:internal/modules/esm/loader:480:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:434:34) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

I decided to try debugging it through a fresh gatsby instance, but it also failed there (with the same error).

what I did

gatsby new
What would you like to call your site?
✔ · My Gatsby Site
What would you like to name the folder where your site will be created?
✔ sites/ my-gatsby-site
✔ Will you be using JavaScript or TypeScript?
· JavaScript
✔ Will you be using a CMS?
· No (or I'll add it later)
✔ Would you like to install a styling system?
· styled-components
✔ Would you like to install additional features with other plugins?
MDX

then, I:

• added "type": "module" to package.json
• renamed gatsby-config.js -> gatsby-config.mjs
• changed module.exports into export default.

What else needs to be done in order to successfully launch build and develop commands? I feel that I'm missing something obvious.

[luckpot207]

so you didn't fix this issue?

[piorus]

no, and I'm afraid that I'm running out of ideas what might be wrong

[piorus]

@luckhole007 (and everyone else with a similar issue) I just managed to fix it, and as I suspected, it was something really silly. In order to make it work, you shouldn't be adding "type": "module" to package.json. It breaks everything.

The only things that are needed:

• rename gatsby-config.js -> gatsby-config.mjs
• adjust mjs file(s) so it supports ESM syntax (replace module.exports into export default, etc)

[axe312ger]

This only works with Gatsby v5. Sure you are using v5?

[luckpot207]

ya
I using gatsby v5, now

[luckpot207]

Hi could you help me? Let me know the gatsby-cli version with Node 16.0.0 On Sun, Mar 5, 2023 at 9:17 PM Smutt Bond ***@***.***> wrote:

…

Hi Could you contact me now via Discord or Telegram? SmuttChain#9020 17722492462 I want know Thanks On Mon, Mar 6, 2023 at 2:52 AM Smutt Bond ***@***.***> wrote: > Thank you very much > I want to meet u via Discord > plz add friend > SmuttChain#9020 > > On Mon, Mar 6, 2023 at 1:24 AM Piotr Rusin ***@***.***> > wrote: > >> @luckhole007 <https://github.com/luckhole007> (and everyone else with a >> similar issue) I just managed to fix it, and as I suspected, it was >> something really silly. In order to make it work, you shouldn't be adding >> "type": "module" to package.json. It breaks everything. >> >> The only things that are needed: >> >> - rename gatsby-config.js -> gatsby-config.mjs >> - adjust mjs file(s) so it supports ESM syntax (replace >> module.exports into export default, etc) >> >> — >> Reply to this email directly, view it on GitHub >> <#31599 (reply in thread)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AVPPG6BO3D66FZPKJJFWMYTW2S435ANCNFSM45S7BTIA> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> >

[luckpot207]

Awesome, thanks!

…

On Wed, Mar 8, 2023 at 11:48 AM Benedikt Rötsch ***@***.***> wrote: Well, the error message clearly tells you the limitations: You need to use a dynamic import OR switch your whole file to use the new syntax. Renaming to .mjs should help :) — Reply to this email directly, view it on GitHub <#31599 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVPPG6HDNL7RBB2DP74CERLW3BBXJANCNFSM45S7BTIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[flogy]

Hi there,

I have implemented some private Gatsby themes to simplify developing new projects. As of Gatsby v5.3 it is possible to use ESM syntax in gatsby-config.mjs. However, when integrating Gatsby themes that are built as ESM ("type": "module" in package.json) running the project fails with the following error:

ES Modules may not assign module.exports or exports.*, Use ESM export syntax, instead: ./.cache/public-page-renderer.js

After some investigation I found that the public-page-renderer.js and create-content-digest-browser-shim.js files of Gatsby are still in CJS syntax, which leads to said error. After manually transforming them to ESM syntax everything worked.

So in case anyone else gets this error, here is a patch to fix it:

diff --git a/node_modules/gatsby/cache-dir/create-content-digest-browser-shim.js b/node_modules/gatsby/cache-dir/create-content-digest-browser-shim.js
index edc673f..cbd390c 100644
--- a/node_modules/gatsby/cache-dir/create-content-digest-browser-shim.js
+++ b/node_modules/gatsby/cache-dir/create-content-digest-browser-shim.js
@@ -1 +1 @@
-exports.createContentDigest = () => ``
+export const createContentDigest = () => ``
diff --git a/node_modules/gatsby/cache-dir/public-page-renderer.js b/node_modules/gatsby/cache-dir/public-page-renderer.js
index 6a91ada..e2ee03a 100644
--- a/node_modules/gatsby/cache-dir/public-page-renderer.js
+++ b/node_modules/gatsby/cache-dir/public-page-renderer.js
@@ -1,9 +1,11 @@
 const preferDefault = m => (m && m.default) || m
 
+let renderer = () => null;
+
 if (process.env.BUILD_STAGE === `develop`) {
-  module.exports = preferDefault(require(`./public-page-renderer-dev`))
+  renderer = preferDefault(require(`./public-page-renderer-dev`))
 } else if (process.env.BUILD_STAGE === `build-javascript`) {
-  module.exports = preferDefault(require(`./public-page-renderer-prod`))
-} else {
-  module.exports = () => null
+  renderer = preferDefault(require(`./public-page-renderer-prod`))
 }
+
+export default renderer;

Best,
Florian