Netlify wont build. Building gatsby-image and production js / css bundles now takes substantially longer time.

[ArthurHwang]

Summary

Every deploy to netlify since I believe since around 6/27 has massively grown in time to build and get hit with timeout everytime.
Before whatever that happened, my deploys used to build pretty quickly for a 3000 ish page with 4000 image transformation site.

Relevant information

Here is a screenshot from before this started happening of a usual deploy:

https://imgur.com/SINW61K

Here it is now with timeout:

https://imgur.com/jqrlja1

notice how much longer it takes to build javascript / css bundles

also, I noticed since this has been happening. The build process of gatsby image generation has also significantly taken longer.

In the screenshot above, it took 1668 seconds to complete this step. Before with an non-cached build it would take 600 seconds for near 4000 images.

The building production js / css step used to take sub 40 seconds.

Environment (if relevant)

  System: 
    OS: Linux 4.4 Ubuntu 18.04.1 LTS (Bionic Beaver)
    CPU: (12) x64 Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
    Shell: 5.4.2 - /usr/bin/zsh
  Binaries:
    Node: 11.15.0 - /usr/bin/node
    npm: 6.9.1 - /usr/bin/npm
  Languages:
    Python: 2.7.15 - /usr/bin/python
  npmPackages:
    gatsby: ^2.11.7 => 2.11.7
    gatsby-image: ^2.2.1 => 2.2.3
    gatsby-paginate: ^1.1.0 => 1.1.0
    gatsby-plugin-breadcrumb: ^2.1.1 => 2.1.2
    gatsby-plugin-exclude: ^1.0.0 => 1.0.0
    gatsby-plugin-manifest: ^2.2.0 => 2.2.0
    gatsby-plugin-netlify: ^2.1.0 => 2.1.0
    gatsby-plugin-netlify-cache: ^1.1.0 => 1.2.0
    gatsby-plugin-no-sourcemaps: ^2.1.0 => 2.1.0
    gatsby-plugin-offline: ^2.2.0 => 2.2.0
    gatsby-plugin-react-helmet: ^3.1.0 => 3.1.0
    gatsby-plugin-react-helmet-canonical-urls: ^1.2.0 => 1.2.0
    gatsby-plugin-remove-fingerprints: 0.0.2 => 0.0.2
    gatsby-plugin-remove-trailing-slashes: ^2.1.0 => 2.1.0
    gatsby-plugin-robots-txt: ^1.4.0 => 1.4.0
    gatsby-plugin-sharp: ^2.2.1 => 2.2.1
    gatsby-plugin-sitemap: ^2.2.1 => 2.2.1
    gatsby-plugin-styled-components: ^3.1.0 => 3.1.0
    gatsby-plugin-tslint: 0.0.2 => 0.0.2
    gatsby-plugin-typescript: ^2.1.0 => 2.1.0
    gatsby-plugin-typography: ^2.3.0 => 2.3.0
    gatsby-source-filesystem: ^2.1.1 => 2.1.1
    gatsby-source-wordpress: ^3.1.1 => 3.1.2
    gatsby-transformer-sharp: ^2.2.0 => 2.2.0
  npmGlobalPackages:
    gatsby-cli: 2.7.7

File contents (if changed)

`gatsby-config.js`: 
  plugins: [
    `gatsby-plugin-react-helmet`,
    `gatsby-plugin-styled-components`,
    `gatsby-image`,
    `gatsby-plugin-sitemap`,
    `gatsby-transformer-sharp`,
    `gatsby-plugin-typescript`,
    `gatsby-plugin-remove-trailing-slashes`,
    {
      resolve: "gatsby-plugin-no-sourcemaps"
    },
    {
      resolve: `gatsby-plugin-sitemap`,
      options: {
        output: `/sitemap.xml`
      }
    },
    {
      resolve: `gatsby-plugin-breadcrumb`,
      options: {
        sitemapPath: `/sitemap.xml`
      }
    },

    {
      resolve: `gatsby-plugin-react-helmet-canonical-urls`,
      options: {
        siteUrl: `https://www.bestattorney.netlify.com`
      }
    },
    {
      resolve: "gatsby-plugin-robots-txt",
      options: {
        policy: [{ userAgent: "*", allow: "/" }]
      }
    },
    {
      resolve: "gatsby-plugin-exclude",
      options: { paths: ["/**/folder-options", "!/app/demo/*"] }
    },
    {
      resolve: `gatsby-plugin-sharp`,
      options: {
        useMozJpeg: false,
        stripMetadata: true,
        defaultQuality: 90,
        pngCompressionSpeed: 10
      }
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `images`,
        path: `${__dirname}/src/images`
      }
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `images`,
        path: `${__dirname}/src/data`
      }
    },
    {
      resolve: `gatsby-plugin-typography`,
      options: {
        pathToConfigModule: `src/components/utilities/typography`
      }
    },
    {
      resolve: "gatsby-source-wordpress",
      options: {
        baseUrl: "www.bestattorney.com/blog",
        protocol: "https",
        hostingWPCOM: false,
        useACF: false,
        verboseOutput: false,
        perPage: 100,
        searchAndReplaceContentUrls: {
          sourceUrl: "https://source-url.com",
          replacementUrl: "https://replacement-url.com"
        },
        concurrentRequests: 100,
        includedRoutes: [
          "**/categories",
          "**/posts",
          "**/pages",
          "**/media",
          "**/tags",
          "**/taxonomies",
          "**/users"
        ],
        excludedRoutes: ["**/posts/1456"],
        normalizer: function({ entities }) {
          return entities
        }
      }
    },

    {
      resolve: "gatsby-plugin-netlify-cache",
      options: {
        cachePublic: true
      }
    },
    // gatsby-plugin-netlify HAS TO BE THE LAST
    {
      resolve: `gatsby-plugin-netlify`,
      options: {
        headers: {}, // option to add more headers. `Link` headers are transformed by the below criteria
        allPageHeaders: [], // option to add headers for all pages. `Link` headers are transformed by the below criteria
        mergeSecurityHeaders: true, // boolean to turn off the default security headers
        mergeLinkHeaders: true, // boolean to turn off the default gatsby js headers
        mergeCachingHeaders: true, // boolean to turn off the default caching headers
        transformHeaders: (headers, path) => headers, // optional transform for manipulating headers under each path (e.g.sorting), etc.
        generateMatchPathRewrites: true // boolean to turn off automatic creation of redirect rules for client only paths
        // For redirects:
        // createRedirect({fromPath: "/old-url", toPath: "/new-url", isPermanent: true})
      }
    }
  ]
}

package.json:

{ 
  "dependencies": {
    "@types/reach__router": "^1.2.4",
    "@types/react": "^16.8.22",
    "@types/react-dom": "^16.8.4",
    "@types/react-helmet": "^5.0.8",
    "@types/react-slick": "^0.23.4",
    "@types/styled-components": "^4.1.16",
    "babel-plugin-styled-components": "^1.10.1",
    "gatsby": "^2.11.7",
    "gatsby-image": "^2.2.1",
    "gatsby-paginate": "^1.1.0",
    "gatsby-plugin-breadcrumb": "^2.1.1",
    "gatsby-plugin-exclude": "^1.0.0",
    "gatsby-plugin-manifest": "^2.2.0",
    "gatsby-plugin-netlify": "^2.1.0",
    "gatsby-plugin-netlify-cache": "^1.1.0",
    "gatsby-plugin-no-sourcemaps": "^2.1.0",
    "gatsby-plugin-offline": "^2.2.0",
    "gatsby-plugin-react-helmet": "^3.1.0",
    "gatsby-plugin-react-helmet-canonical-urls": "^1.2.0",
    "gatsby-plugin-remove-fingerprints": "0.0.2",
    "gatsby-plugin-remove-trailing-slashes": "^2.1.0",
    "gatsby-plugin-robots-txt": "^1.4.0",
    "gatsby-plugin-sharp": "^2.2.1",
    "gatsby-plugin-sitemap": "^2.2.1",
    "gatsby-plugin-styled-components": "^3.1.0",
    "gatsby-plugin-tslint": "0.0.2",
    "gatsby-plugin-typescript": "^2.1.0",
    "gatsby-plugin-typography": "^2.3.0",
    "gatsby-source-filesystem": "^2.1.1",
    "gatsby-source-wordpress": "^3.1.1",
    "gatsby-transformer-sharp": "^2.2.0",
    "lodash": "^4.17.11",
    "polished": "^3.4.1",
    "prop-types": "^15.7.2",
    "react": "^16.8.6",
    "react-dom": "^16.8.6",
    "react-helmet": "^5.2.1",
    "react-icons": "^3.6.1",
    "react-lazyload": "^2.6.2",
    "react-media": "^1.9.2",
    "react-slick": "^0.24.0",
    "react-typography": "^0.16.19",
    "slick-carousel": "^1.8.1",
    "styled-components": "^4.3.2",
    "styled-components-modifiers": "^1.2.1",
    "tslint": "^5.18.0",
    "tslint-config-prettier": "^1.18.0",
    "tslint-loader": "^3.5.4",
    "tslint-plugin-prettier": "^2.0.1",
    "tslint-react": "^4.0.0",
    "typescript": "^3.5.2",
    "typography": "^0.16.19",
    "typography-theme-wordpress-2011": "^0.16.19"
  },
  "devDependencies": {
    "prettier": "^1.18.2"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "scripts": {
    "type-check": "tsc --noEmit",
    "lint": "tslint --project .",
    "build": "gatsby build",
    "develop": "gatsby develop -o",
    "format": "prettier --write src/**/*.{js,jsx}",
    "start": "npm run develop",
    "serve": "gatsby serve -o",
    "test": "echo \"Write tests! -> https://gatsby.dev/unit-testing\""
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/gatsbyjs/gatsby-starter-default"
  },
  "bugs": {
    "url": "https://github.com/gatsbyjs/gatsby/issues"
  }
}

gatsby-node.js:

require("dotenv").config({
  path: `.env.${process.env.NODE_ENV}`
})

const path = require("path")
const createPaginatedPages = require("gatsby-paginate")

// Implement the Gatsby API “onCreatePage”. This is
// called after every page is created.
exports.onCreatePage = async ({ page, actions }) => {
  const { createPage } = actions

  // page.matchPath is a special key that's used for matching pages
  // only on the client.
  if (page.path.match(/^\/app/)) {
    page.matchPath = "/admin/*"

    // Update the page.
    createPage(page)
  }
}

exports.createPages = ({ graphql, actions }) => {
  const { createPage } = actions

  return new Promise((resolve, reject) => {
    graphql(
      `
        {
          allWordpressPost(limit: ${process.env.BLOG_LIMIT}) {
            edges {
              node {
                id
                slug
                title
                content
                excerpt
                date
                modified
                categories {
                  name
                }
                author {
                  name
                }
                featured_media {
                  localFile {
                    childImageSharp {
                      fluid {
                        src
                        srcSet
                        sizes
                        aspectRatio
                      }
                    }
                  }
                }
              }
            }
          }
        }
      `
    ).then(result => {
      if (result.errors) {
        console.log(result.errors)
        reject(result.errors)
      }

      createPaginatedPages({
        edges: result.data.allWordpressPost.edges,
        createPage: createPage,
        pageTemplate: "./src/components/elements/blog/PostsTemplate.tsx",
        pageLength: 5,
        pathPrefix: "/blog"
      })

      result.data.allWordpressPost.edges.forEach(({ node }) => {
        createPage({
          path: "blog/" + node.slug,
          component: path.resolve(
            "./src/components/elements/blog/PostTemplate.tsx"
          ),
          context: {
            slug: node.slug,
            imageSharp: node.featured_media,
            title: node.title,
            author: node.author.name,
            categories: node.categories,
            modified: node.modified,
            content: node.content,
            excerpt: node.excerpt
          }
        })
      })
      resolve()
    })
  })
}

gatsby-browser.js: N/A
gatsby-ssr.js: N/A

[grubersjoe]

Try using babel-plugin-styled-components@1.10.6. Upgrading to this version fixed the build performance issues for me.

See here for more details: https://github.com/styled-components/babel-plugin-styled-components/releases/tag/v1.10.6

Also upgrade Gatsby to the latest version. They seem to have increased build time for lots of people too by using babel-loader for node_modules since 2.11, but the latest patch releases improve that.

[wardpeet]

Have you tried gatsby version 2.11.8? It should have a fix for slow builds

[danboyle8637]

@ArthurHwang Here is what I just did with my site that was timing out. I would be interested in knowing if it works for you.

#15275

In essence... I ran:

npm --depth 9999 update

[ArthurHwang]

@grubersjoe thanks for that i actually ended up downgrading to 1.10.0 and it fixed my issue just now before I read your comment. I am just glad that an actual reason was found for this issue.

I will upgrade to 1.10.6 latest version and see what happens then report back.

Thank you again.

[zappys]

Had a similar issue. Upgrading to babel-plugin-styled-components@1.10.6 solved it for me.

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[wardpeet]

Thank you for opening this, @ArthurHwang

We're marking this issue as answered and closing it for now but please feel free to reopen this and comment if you would like to continue this discussion. We hope we managed to help and thank you for using Gatsby! 💜