Import d3 causes webpack dependency errors

[nwshane]

Adding the line import * as d3 from 'd3' to any JS file in my src code causes the following errors:

ERROR  Failed to compile with 2 errors

These dependencies were not found:

* child_process in ./~/xmlhttprequest/lib/XMLHttpRequest.js
* fs in ./~/xmlhttprequest/lib/XMLHttpRequest.js

To install them, you can run: npm install --save child_process fs

Could this be a bug caused by Gatsby's webpack config? I don't understand what's going on here: Why is XMLHttpRequest throwing an error when it tries to require built in node modules like fs and child_process, and why would these errors be triggered by including d3?

More info

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: 'NathanShane.me',
    author: 'Nathan Shane',
    description: require('./package.json').description
  },
  plugins: [
    'gatsby-plugin-react-helmet',
    'gatsby-plugin-catch-links',
    'gatsby-transformer-sharp',
    'gatsby-plugin-offline',
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/src/pages/projects`,
        name: 'projects',
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/src/pages/blog`,
        name: 'blogPosts',
      },
    },
    {
      resolve: 'gatsby-transformer-remark',
      options: {
        plugins: [
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 590
            }
          },
          'gatsby-remark-copy-linked-files'
        ]
      }
    }
  ]
}

gatsby-node.js:

const {resolve} = require('path')

exports.modifyBabelrc = ({ babelrc }) => {
  return {
    plugins: babelrc.plugins.concat([
      ['babel-plugin-root-import']
    ]),
  }
}

exports.createPages = ({boundActionCreators, graphql}) => {
  const {createPage} = boundActionCreators

  return graphql(
    `
    {
      allMarkdownRemark(limit: 1000) {
        edges {
          node {
            fileAbsolutePath
            frontmatter {
              slug
            }
          }
        }
      }
    }
    `
  ).then((result) => {
    if (result.errors) {
      return Promise.reject(result.errors)
    }
    const nodes = result.data.allMarkdownRemark.edges

    const filterByPathIncludes = (testStr, nodes) => (
      nodes.filter(
        ({node: {fileAbsolutePath}}) => (fileAbsolutePath.includes(testStr))
      )
    )

    const projects = filterByPathIncludes('/pages/projects/', nodes)

    projects.forEach(({node}) => {
      const {slug} = node.frontmatter
      createPage({
        path: `/projects/${slug}`,
        component: resolve('./src/templates/project.js'),
        context: {
          slug
        }
      })
    })

    const blogPosts = filterByPathIncludes('/pages/blog/', nodes)

    blogPosts.forEach(({node}) => {
      const {slug} = node.frontmatter
      createPage({
        path: `/blog/${slug}`,
        component: resolve('./src/templates/blogPost.js'),
        context: {
          slug
        }
      })
    })
  }).catch((error) => {
    console.log(error)
  })
}

package.json:

{
  "name": "nathan_shane_site",
  "description": "Nathan Shane's portfolio website",
  "repository": "https://github.com/nwshane/nwshane.github.io.git",
  "license": "MIT",
  "scripts": {
    "dev": "gatsby develop",
    "build": "gatsby build",
    "start": "gatsby serve"
  },
  "dependencies": {
    "babel-plugin-root-import": "^5.1.0",
    "d3": "^4.10.2",
    "gatsby": "^1.8.11",
    "gatsby-link": "^1.4.1",
    "gatsby-plugin-catch-links": "^1.0.8",
    "gatsby-plugin-offline": "^1.0.9",
    "gatsby-plugin-react-helmet": "^1.0.6",
    "gatsby-remark-copy-linked-files": "^1.5.7",
    "gatsby-remark-images": "^1.5.10",
    "gatsby-source-filesystem": "^1.4.12",
    "gatsby-transformer-remark": "^1.7.6",
    "gatsby-transformer-sharp": "^1.6.5",
    "normalize.css": "^7.0.0",
    "ramda": "^0.24.1",
    "styled-components": "^2.1.2",
    "webfontloader": "^1.6.28"
  }
}

Repo and branch in which error is occuring: https://github.com/nwshane/nwshane.github.io/tree/blog

Gatsby version: 1.9.21
NodeJS version: 8.4.0
OS version: Mac OS X 10.12

[KyleAMathews]

Probably you installed the d3-node module by accident which then webpack is choking on trying to import as it's trying to import a bunch of node only modules.

[jquense]

its interesting that the error is so misleading, maybe we can fix it. e.g. if the target is not Node in webpack we shouldn't recommend installing core node modules.

[nwshane]

@KyleAMathews I don't think that's the case. The d3 documentation itself recommends using import * as d3 from "d3";, which is what I'm doing. I don't have d3-node in my package.json dependencies and I'm not requiring it anywhere.

If you have any ideas about how to debug this issue, that'd be very helpful.

[revolunet]

looks like this xmlhttprequest module is not webpack friendly :/

related : driverdan/node-XMLHttpRequest#127

[nwshane]

@revolunet Why is xmlhttprequest getting required by d3 in the first place?

[revolunet]

no idea , could be some other module

[haroldtreen]

Some findings:

• d3 requires d3-request code
• d3-request requires xmlhttprequest code
• xmlhttprequest requires child-process and fs code
• d3-request builds two version - one for node and one min.
• The default export is the node version.
• Changing the main module of d3-request from build/d3-request.node.js --> build/d3-request.min.js fixes the import.
• Using the modifyWebpackConfig to include: node: { fs: 'empty', child_process: 'empty' } fixes the issue.

Strange though because this issue seems to not appear when using a very plain webpack.config.js. Trying to find out what webpack differences are triggering the error.

[haroldtreen]

The plain webpack was version 3. Using the gatsby webpack breaks it.

It would be interesting to talk to d3-request and find out why the node version if the main module...

[haroldtreen]

Known d3-request issue. That module is expecting bundlers to use the module field of package.json instead of main.

d3/d3-request#24

[haroldtreen]

d3-request deleted a field called browser that used to direct webpack to the correct non-node index file. Now d3-request expects the module field to be used.

Commit: d3/d3-request@d635b89#diff-b9cfc7f2cdf78a7f4b91a753d10865a2L19

Solution:

Add this to the webpack config:

{
   resolve: {
      packageMains: [
        'module', // adds check for 'module'
        'webpack',
        'browser',
        'web',
        'browserify',
        ['jam', 'main'],
        'main',
    ]
   }
}

See packageMains config documentation

[hermionewy]

I added the packageMains, but still not working. I use Webpack1. Anyone know what to do to get rid of the XMLHttpRequest and fs error?

[mattjstar]

yep, getting the same issue as @hermionewy when using @haroldtreen 's solution with packageMains. iTerm output:

$ gatsby develop
success delete html files from previous builds — 0.010 s
success open and validate gatsby-config.js — 0.003 s
info One or more of your plugins have changed since the last time you ran Gatsby. As
a precaution, we're deleting your site's cache to ensure there's not any stale
data
success copy gatsby files — 0.013 s
success onPreBootstrap — 0.004 s
success source and transform nodes — 0.015 s
success building schema — 0.064 s
success createLayouts — 0.024 s
success createPages — 0.004 s
success createPagesStatefully — 0.011 s
success onPreExtractQueries — 0.001 s
success update schema — 0.044 s
success extract queries from components — 0.057 s
success run graphql queries — 0.013 s
success write out page data — 0.003 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.000 s

info bootstrap finished - 1.505 s

error There was an error compiling the html.js component for the development server.

See our docs page on debugging HTML builds for help https://goo.gl/yL9lND


  WebpackError: Unexpected token import






  - reactProdInvariant.js:34 Compilation.<anonymous>
    ~/react/lib/reactProdInvariant.js:34:1

  - develop-static-entry.js:3 Compilation.next
    .cache/develop-static-entry.js:3:1



  - reactProdInvariant.js:30 Compilation.<anonymous>
    ~/react/lib/reactProdInvariant.js:30:1

  - develop-static-entry.js:3 Compilation.next
    .cache/develop-static-entry.js:3:1

  - React.js:127 ExtractTextPlugin.<anonymous>
    ~/react/lib/React.js:127:1


  - React.js:78 Object.async.forEachOf.async.eachOf
    ~/react/lib/React.js:78:1

  - React.js:51 Object.async.forEach.async.each
    ~/react/lib/React.js:51:1

  - React.js:79 ExtractTextPlugin.<anonymous>
    ~/react/lib/React.js:79:1


error Command failed with exit code 1.

I was able to get it to work with this, but I'm afraid that will have unintended consequences down the line.

exports.modifyWebpackConfig = ({ config, stage }) => {
  config.merge({
     node: { fs: 'empty', child_process: 'empty' },
  })

  return config;
};

[jquense]

No i think that's the right way to handle this @mattjstar we should make those the defaults for v2 ( #2641)

[richardwestenra]

I'm getting a similar error with fs being required by dotenv, which I presume showed up after reinstalling my node modules a couple of days ago, but I just noticed it today:

 ERROR  Failed to compile with 1 errors 

This dependency was not found:

* fs in ./~/dotenv/lib/main.js

To install it, you can run: npm install --save fs

@mattjstar's fix works for me too.

EDIT: No it didn't - it stopped the error messages from appearing, but then dotenv stopped working.

[KyleAMathews]

@richardwestenra is this error happening in the browser? If so, know that dotenv is only useful in node.js code so shouldn't be required in the browser.

[richardwestenra]

@KyleAMathews Nope, this error is happening in the terminal, confusingly.

I've tried moving it to devDependencies, as well as setting

  config.merge({
    target: 'node'
  });

in the webpack config, but have had no success 😕

[KyleAMathews]

Webpack is only for browser code so unless you're requiring dotenv in browser code modifying the webpack config isn't going to help with node code. Confusing I know :-)

[richardwestenra]

ah okay, that helps narrow things down a bit. Does that mean it's unlikely to be a Gatsby-related issue, then? I'm pretty much stumped for ideas.

I'm using dotenv to populate Contentful/Workable API keys used in plugins in gatsby-config.js, having initialised dotenv at the start of that file.

[KyleAMathews]

Perhaps try rm -r node_modules && yarn? :-)

But yeah, this is either an install problem or a dotenv problem. Node.js is somehow failing to require the built-in fs module.

[richardwestenra]

Have tried that several times, and it doesn't look like dotenv has published any releases in a year. I'll keep looking. Thanks for the assistance! :)

[richardwestenra]

Okay I figured it out! I'm unsure whether it's relevant to others' issues posted above, but I'll post details of my solution here in case it helps anyone.

It turns out, the issue showed up when I started including siteMetadata.title from gatsby-config in my index layout file. I'm not sure what the siteMetadata prop is supposed to be used for - I found a reference to page components having access to it with props.data.site.siteMetadata, but that props.data isn't showing up for me. Anyway it seemed like using that property for the site title would likely be best practice, so I included it in the layout template with import { siteMetadata } from '../../gatsby-config', then used it to populate <title> with React Helmet. However I guess this caused gatsby-config started to be included in Webpack browser code, which doesn't have access to fs, so it started throwing errors.

Is there a recommended way of including gatsby-config's siteMetadata prop in browser code? If not, what is its purpose?

[KyleAMathews]

Yes, via GraphQL. See https://www.gatsbyjs.org/tutorial/part-four/#our-first-graphql-query

[richardwestenra]

ah okay, great, thanks! I didn't spot that, but I guess it'll be included in the documentation when the GraphQL docs page is done.

Thanks again for your help!

[parvezk]

You can also follow this approach to fix the XMLHTTPRequest issue with module bundling:
webpack/webpack-dev-server#66

[kwelch]

I am seeing the same fs error post-installing dotenv. Does anyone have version 1.9.223 working with dotenv?

[ohenrik]

@haroldtreen where should i add this? If i add packageMains to resolve i get an error telling me that packageMains is unknown.

[jhlabs]

I am facing exactly the same error as @richardwestenra when installing the gatsby-plugin-mailchimp and using dotenv. Is there a way to resolve it in that case?

[kakadiadarpan]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub issues, we have to clean some of the old issues as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Gatsby version and check if that solves the issue. Let us know if that works for you by adding a comment 👍

[kakadiadarpan]

This issue is being closed because there hasn't been any activity for at least 30 days. Feel free to open a new one if you still experience this problem 👍