Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue: warning Query takes too long + Build breaks on netlify #30256

Closed
quansenB opened this issue Mar 15, 2021 · 28 comments
Closed

Issue: warning Query takes too long + Build breaks on netlify #30256

quansenB opened this issue Mar 15, 2021 · 28 comments
Labels
stale? Issue that may be closed soon due to the original author not responding any more. topic: DX Developer Experience (e.g. Fast Refresh, i18n, SSR, page creation, starters) topic: performance Related to runtime & build performance type: bug An issue or pull request relating to a bug in Gatsby

Comments

@quansenB
Copy link

quansenB commented Mar 15, 2021

Description

I'm getting the Warning Query takes too long on very small queries, for example:

export const pageQuery = graphql`
  query impressumPageQuery {
    markdownRemark(frontmatter: { templateKey: { eq: "impressum-page" } }) {
      html
    }
  }
`;

This is not a problem for building my site locally but is on netlify which enforces a RAM limit (NPM error code 137).

I have no idea why such a query would be so big. At first I thought it might be related to the new gatsby-plugin-image but it also appears on some sites with very small queries, like the above.

Steps to reproduce

  1. Fork project: https://github.com/quansenB/Website-Faltermeier
  2. Build on netlify

Expected result

Should be built without Error and query takes too long warning

Actual result

Getting npm ERR! code 137 (out of memory)

Environment

System:
    OS: macOS 10.15.7
    CPU: (4) x64 Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 14.15.4 - /usr/local/bin/node
    npm: 6.14.10 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 89.0.4389.82
    Edge: 89.0.774.54
    Firefox: 85.0.2
    Safari: 14.0.3
  npmPackages:
    gatsby: ^3.0.4 => 3.0.4
    gatsby-cli: ^3.0.0 => 3.0.0
    gatsby-plugin-google-tagmanager: ^3.0.0 => 3.0.0
    gatsby-plugin-image: ^1.0.1 => 1.0.1
    gatsby-plugin-manifest: ^3.0.0 => 3.0.0
    gatsby-plugin-minify-html: ^1.0.4 => 1.0.4
    gatsby-plugin-netlify: ^3.0.0 => 3.0.0
    gatsby-plugin-netlify-cms: ^5.0.0 => 5.0.0
    gatsby-plugin-offline: ^4.0.0 => 4.0.0
    gatsby-plugin-postcss: ^4.0.0 => 4.0.0
    gatsby-plugin-preact: ^5.0.0 => 5.0.0
    gatsby-plugin-preload-fonts: ^2.0.0 => 2.0.0
    gatsby-plugin-react-helmet: ^4.0.0 => 4.0.0
    gatsby-plugin-remove-serviceworker: ^1.0.0 => 1.0.0
    gatsby-plugin-robots-txt: ^1.5.5 => 1.5.5
    gatsby-plugin-sharp: ^3.0.1 => 3.0.1
    gatsby-plugin-sitemap: ^3.0.0 => 3.0.0
    gatsby-plugin-svgr: ^3.0.0-beta.0 => 3.0.0-beta.0
    gatsby-remark-images: ^4.0.0 => 4.0.0
    gatsby-remark-relative-images-v2: ^0.1.5 => 0.1.5
    gatsby-source-filesystem: ^3.0.0 => 3.0.0
    gatsby-transformer-remark: ^3.0.0 => 3.0.0
    gatsby-transformer-sharp: ^3.0.0 => 3.0.0
  npmGlobalPackages:
    gatsby: 3.0.4

@quansenB quansenB added the type: bug An issue or pull request relating to a bug in Gatsby label Mar 15, 2021
@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Mar 15, 2021
@quansenB quansenB changed the title Issue: warning Query takes too long Issue: warning Query takes too long + Build breaks on netlify Mar 15, 2021
@mrkaluzny
Copy link
Contributor

I have the same error. The other problem is that gatsby-plugin-image seems to be generating the images when query is running. My builds are now 25min+ instead of previous 5-6min. Any help?

"dependencies": {
    "babel-preset-gatsby": "^1.0.0",
    "classnames": "^2.2.6",
    "gatsby": "^3.0.4",
    "gatsby-plugin-advanced-sitemap": "^1.4.5",
    "gatsby-plugin-brotli": "^2.0.0",
    "gatsby-plugin-disqus": "^1.2.2",
    "gatsby-plugin-gdpr-cookies": "^1.0.3",
    "gatsby-plugin-google-tagmanager": "^3.0.0",
    "gatsby-plugin-image": "^1.0.1",
    "gatsby-plugin-netlify": "^3.0.0",
    "gatsby-plugin-netlify-cms": "^5.0.0",
    "gatsby-plugin-purgecss": "^6.0.0",
    "gatsby-plugin-react-helmet": "^4.0.0",
    "gatsby-plugin-root-import": "^2.0.5",
    "gatsby-plugin-sass": "^4.0.2",
    "gatsby-plugin-sharp": "^3.0.1",
    "gatsby-remark-autolink-headers": "^3.0.0",
    "gatsby-remark-copy-linked-files": "^3.0.0",
    "gatsby-remark-images": "^4.0.0",
    "gatsby-remark-prismjs": "^4.0.0",
    "gatsby-remark-relative-images": "^2.0.2",
    "gatsby-source-filesystem": "^3.0.0",
    "gatsby-transformer-remark": "^3.0.0",
    "gatsby-transformer-sharp": "^3.0.0",
    "lodash": "^4.17.15",
    "lodash-webpack-plugin": "^0.11.4",
    "netlify-cms-app": "^2.14.29",
    "node-sass": "^5.0.0",
    "parcel-bundler": "^1.9.4",
    "prismjs": "^1.22.0",
    "prop-types": "^15.6.0",
    "react": "^17.0.1",
    "react-confetti": "^6.0.0",
    "react-dom": "^17.0.1",
    "react-helmet": "^6.1.0",
    "react-markdown": "^5.0.2",
    "react-slick": "^0.28.1",
    "react-svg": "^12.1.0",
    "react-transition-group": "^4.4.1",
    "slick-carousel": "^1.8.1",
    "universal-cookie": "^4.0.2",
    "uuid": "^8.3.2"
  },

@bradgarropy
Copy link
Contributor

Same here. I updated to Gatsby v3 and my build went from 3 minutes to 30 minutes with a timeout on Netlify. I see many of these messages.

10:36:39 AM: warning Query takes too long:
10:36:39 AM: File path: /opt/build/repo/src/templates/post.js
10:36:39 AM: URL path: /posts/to-be-loved-by-you
10:36:39 AM: Context: {
10:36:39 AM:     "slug": "to-be-loved-by-you"
10:36:39 AM: }

The website is open source. In particular, the build logs are complaining about the page query in my post.js template.

const query = graphql`
    query($slug: String!) {
        markdownRemark(frontmatter: {slug: {eq: $slug}}) {
            html
            frontmatter {
                slug
                title
                date(formatString: "MMMM D, YYYY")
                tags
                image {
                    publicURL
                    childImageSharp {
                        gatsbyImageData(
                            width: 750
                            placeholder: BLURRED
                            formats: [AUTO, WEBP, AVIF]
                            quality: 100
                        )
                    }
                }
            }
        }
    }
`

@LekoArts LekoArts added topic: page creation and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Mar 17, 2021
@creotip
Copy link

creotip commented Mar 18, 2021

i have the same problem.

@quansenB
Copy link
Author

quansenB commented Mar 18, 2021

This was assigned topic: scaling builds, so just for information: This is a rather small site. Like 15 pages total, and the queries in question query just some markdown and one image max.

@vladar
Copy link
Contributor

vladar commented Mar 18, 2021

We've heard reports that AVIF transformations are slow. Can you double-check by temporarily disabling any AVIF usages and build without it? Just to confirm if it is the actual source of the issue.

@lomby92
Copy link

lomby92 commented Mar 18, 2021

I will try it tomorrow, thanks for the suggestion

@mrkaluzny
Copy link
Contributor

@vladar That's definitely the case, another point could be the DOMINANT_COLOR option.
As a solution I had to switch from Netlify to Gatsby Cloud for build processes (Netlify would finish builds but threw errors when deploying).

When it comes to images, could you also disable warnings on FULL_WIDTH images? It seems kind of redundant ;)

@bradgarropy
Copy link
Contributor

@vladar I took your advice and simplified my gatsbyImageData query from this:

childImageSharp {
    gatsbyImageData(
        width: 750
        placeholder: BLURRED
        formats: [AUTO, WEBP, AVIF]
        quality: 100
    )
}

Down to this.

childImageSharp {
    gatsbyImageData(width: 750)
}

But, I saw no changes or improvements in the build time. All of the query takes too long messages were still present.

9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/bury-my-bones-heart-wrenching-music-video
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "bury-my-bones-heart-wrenching-music-video"
9:21:47 PM: }
9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/casey-donahews-nye-livestream
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "casey-donahews-nye-livestream"
9:21:47 PM: }
9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/evan-felker-finds-sobriety
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "evan-felker-finds-sobriety"
9:21:47 PM: }
9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/country-bars-closing
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "country-bars-closing"
9:21:47 PM: }

@lomby92
Copy link

lomby92 commented Mar 19, 2021

I haven't done the test without AVIF yet.

As I have both Gatsby cloud build and Netlify build on the same site, I can report the differences:

Gatsby Cloud: success run page queries - 0.347s - 7/7 20.19/s
Netlify: success run page queries - 167.913s - 7/7 0.04/s

Note that on Gatsby Cloud the Query takes too long warning does not happen

@vladar
Copy link
Contributor

vladar commented Mar 19, 2021

@lomby92 Image processing in Gatsby Cloud is heavily optimized. So this could explain the difference if the source of the issue is AVIF in your case. Curious to see your results without AVIF to confirm.

@bradgarropy This happens locally for you too? If I run it locally - will I see the same issue? Or does it only happen on Netlify?

@vladar
Copy link
Contributor

vladar commented Mar 19, 2021

We may be seeing this sharp issue in play here: lovell/sharp#2597 (not 100% sure yet)

There is a work in progress to improve it though.

@lomby92
Copy link

lomby92 commented Mar 19, 2021

@vladar thanks for the info about the Gatsby Cloud optimization.

After a local test I was able to reproduce the warning disabling the AVIF format.

Without AVIF

success run page queries - 1.361s - 8/8 5.88/s
success write out requires - 0.003s
success Building production JavaScript and CSS bundles - 31.376s
success Rewriting compilation hashes - 0.004s
success Building HTML renderer - 3.590s
success Building static HTML for pages - 0.150s - 8/8 53.50/s
success onPostBuild - 0.016s
info Done building in 42.339797475 sec

With AVIF

success run page queries - 31.345s - 8/8 0.26/s
success write out requires - 0.011s
success Building production JavaScript and CSS bundles - 34.760s
success Rewriting compilation hashes - 0.003s
success Building HTML renderer - 4.159s
success Building static HTML for pages - 0.214s - 8/8 37.44/s
success onPostBuild - 0.020s
info Done building in 81.095961358 sec

Transforming images into AVIF format uses all my CPU 6c/12t at 100%. Also, in this case I can see the warnings as expected.

Conclusions

As you can see, the AVIF setting lead to a longer build, it's confirmed

@quansenB
Copy link
Author

quansenB commented Mar 19, 2021

I can confirm that the build works without AVIF. Though I'm still getting the Query takes to long warning.

I'm now also getting an seemingly unrelated error, which also doesn't appear locally. (The build passes though).
I'm dropping it here, just in case it helps:

error (node:1307) [DEP_WEBPACK_COMPILATION_NORMAL_MODULE_LOADER_HOOK] DeprecationWarning: Compilation.hooks.normalModuleLoader was moved to NormalModule.getCompilationHooks(compilation).loader
1:58:38 PM: (Use `node --trace-deprecation ...` to show where the warning was created)
1:58:55 PM: error (node:1307) [DEP_WEBPACK_COMPILATION_ASSETS] DeprecationWarning: Compilation.assets will be frozen in future, all modifications are deprecated.
1:58:55 PM: BREAKING CHANGE: No more changes should happen to Compilation.assets after sealing the Compilation.
1:58:55 PM: 	Do changes to assets earlier, e. g. in Compilation.hooks.processAssets.
1:58:55 PM: 	Make sure to select an appropriate stage from Compilation.PROCESS_ASSETS_STAGE_*.

@bradgarropy
Copy link
Contributor

@vladar This only happens on Netlify, locally things run just fine!

@ascorbic
Copy link
Contributor

I'm going to add a note to the docs warning about the build time performance implications of AVIF support. It's the reason we don't enable it by default. I think the issue here is that Netlify containers don't appear to have enough memory to handle AVIF builds on your site, whereas Gatbsy Cloud is specifically optimised to handle it.

@bradgarropy
Copy link
Contributor

That sounds great @ascorbic!
Alghtouh I'm still seeing the query takes too long issue even without AVIFs included.

@ehowey
Copy link
Contributor

ehowey commented Mar 23, 2021

Just hopping in to confirm that I am seeing this error as well, build is still completing in my case but I am testing it on just my personal website. As mentioned above I was also seeing the deprecation warnings about Webpack [DEP_WEBPACK_COMPILATION_NORMAL_MODULE_LOADER_HOOK]. I can confirm I am not using AVIF.

@vladar
Copy link
Contributor

vladar commented Mar 23, 2021

@ehowey can you share a reproduction project for Query takes too long without AVIF? I'll try to figure out what's going on.

@ehowey
Copy link
Contributor

ehowey commented Mar 31, 2021

@vladar It seems to have gone away now - at least in the most recent builds on Gatsby v3.2. I'll pop back in here if I see it again.

Since that time I have done some optimizations to my GraphQL queries for the images so I may have even solved it myself. One thing I had happening early in my transition to Gatsby v3 and gatsby-plugin-image was queries that were just gatsbyImageData without any options passed to it, so maybe that was a problem?

@ascorbic
Copy link
Contributor

@ehowey Were the source images very large? That's the only scenario where I could see having it without options could make it slow, as it would default to the full size with constrained layout.

@bradgarropy
Copy link
Contributor

@vladar After updating to v3.2.0 I'm still experiencing the same Query takes too long messages.
https://app.netlify.com/sites/dailytexascountry/deploys/6065198d1239431a00670966

@ehowey
Copy link
Contributor

ehowey commented Apr 1, 2021

@ascorbic could be - sometimes I take images straight from Unsplash and don't bother downsizing them because Gatsby has usually handled that for me.

@ardiewen
Copy link

Just wanted to comment here that I'm experiencing this with local builds on both a 2017 macbook pro (4 core / 16gb ram) and a remote Ubuntu development server with 4 cores / 8gb ram. No issues with Gatsby Cloud, and we don't build on Netlify anymore (but we do deploy there for Netlify functions).

We have image fragments with formats: [AUTO, WEBP, AVIF] and we have quite a few images (400-500) that are in high quality.

When I gatsby clean && gatsby develop there's no issues with building the development bundle and proceeding as intended. If however, I don't run gatsby clean it will usually hang on the run page queries step.

When I gatsby clean && gatsby build, I will usually hang on one of the following:

MBP:

success run page queries - 6.833s - 1/1 0.15/s

Ubuntu:

success Building production JavaScript and CSS bundles - 40.024s
success Rewriting compilation hashes - 0.001s

MBP Info:

  System:
    OS: macOS 11.2.3
    CPU: (4) x64 Intel(R) Core(TM) i5-7267U CPU @ 3.10GHz
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.15.3 - ~/.nvm/versions/node/v14.15.3/bin/node
    Yarn: 1.22.10 - ~/Projects/WORKING_DIRECTORY/node_modules/.bin/yarn
    npm: 6.14.9 - ~/.nvm/versions/node/v14.15.3/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 89.0.4389.128
    Firefox: 86.0
    Safari: 14.0.3
  npmPackages:
    gatsby: ^3.3.0 => 3.3.0
    gatsby-omni-font-loader: ^1.3.0 => 1.3.1
    gatsby-plugin-image: ^1.3.0 => 1.3.0
    gatsby-plugin-mailchimp: ^5.2.2 => 5.2.2
    gatsby-plugin-manifest: ^3.2.0 => 3.3.0
    gatsby-plugin-netlify: ^3.2.0 => 3.3.0
    gatsby-plugin-no-sourcemaps: ^2.2.0 => 2.9.0
    gatsby-plugin-react-helmet: ^4.2.0 => 4.3.0
    gatsby-plugin-react-svg: ^2.1.2 => 2.1.2
    gatsby-plugin-remove-serviceworker: ^1.0.0 => 1.0.0
    gatsby-plugin-segment-js: ^3.6.0 => 3.6.0
    gatsby-plugin-sharp: ^3.3.0 => 3.3.0
    gatsby-plugin-sitemap: ^3.2.0 => 3.3.0
    gatsby-plugin-styled-components: ^4.2.0 => 4.3.0
    gatsby-plugin-webpack-bundle-analyser-v2: ^1.1.16 => 1.1.22
    gatsby-source-filesystem: ^3.2.0 => 3.3.0
    gatsby-source-wordpress: ^5.3.0 => 5.3.0
    gatsby-transformer-sharp: ^3.3.0 => 3.3.0

Ubuntu VM Info

  System:
    OS: Linux 4.15 Ubuntu 18.04.2 LTS (Bionic Beaver)
    CPU: (4) x64 Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
    Shell: 5.4.2 - /usr/bin/zsh
  Binaries:
    Node: 14.16.1 - ~/.nvm/versions/node/v14.16.1/bin/node
    Yarn: 1.22.10 - ~/Projects/WORKING_DIRECTORY/node_modules/.bin/yarn
    npm: 6.14.12 - ~/.nvm/versions/node/v14.16.1/bin/npm
  Languages:
    Python: 2.7.17 - /usr/bin/python
  npmPackages:
    gatsby: ^3.3.0 => 3.3.0
    gatsby-omni-font-loader: ^1.3.0 => 1.3.1
    gatsby-plugin-image: ^1.3.0 => 1.3.0
    gatsby-plugin-mailchimp: ^5.2.2 => 5.2.2
    gatsby-plugin-manifest: ^3.2.0 => 3.3.0
    gatsby-plugin-netlify: ^3.2.0 => 3.3.0
    gatsby-plugin-no-sourcemaps: ^2.2.0 => 2.9.0
    gatsby-plugin-react-helmet: ^4.2.0 => 4.3.0
    gatsby-plugin-react-svg: ^2.1.2 => 2.1.2
    gatsby-plugin-remove-serviceworker: ^1.0.0 => 1.0.0
    gatsby-plugin-segment-js: ^3.6.0 => 3.6.0
    gatsby-plugin-sharp: ^3.3.0 => 3.3.0
    gatsby-plugin-sitemap: ^3.2.0 => 3.3.0
    gatsby-plugin-styled-components: ^4.2.0 => 4.3.0
    gatsby-plugin-webpack-bundle-analyser-v2: ^1.1.16 => 1.1.22
    gatsby-source-filesystem: ^3.2.0 => 3.3.0
    gatsby-source-wordpress: ^5.3.0 => 5.3.0
    gatsby-transformer-sharp: ^3.3.0 => 3.3.0

@pfortes
Copy link

pfortes commented Apr 18, 2021

I had some posts with the "Query takes too long" messages.

it seams like having HTML tags in my mdx files seamed to be the culprit of my long querys, i dont know if the classes or the dimension atributes where afecting.

Changing images from something like
<img className="alignnone size-medium wp-image-574" alt="logo" src="proyeccionLentes3d-logo-300x201.png" width="250" />

To:
![logo](proyeccionLentes3d-logo-300x201.png)

Stoped the messages from showing.

My build changed from:

  • 32min 55sec with avif activated
  • 8min 20sec without avif but with the long query messages
  • 7min 3sec without the img tags in the mdx

I think using avif rirght now is not a good idea for me!

PD: i only have 63 pages..

@github-actions
Copy link

github-actions bot commented May 9, 2021

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

@github-actions github-actions bot added the stale? Issue that may be closed soon due to the original author not responding any more. label May 9, 2021
@jodiedoubleday
Copy link

I'm getting this on all my recent builds on Netlify, its take my build times from ~5 minutes to 30+ minutes.
I'm using gatsby-image as I can't upgrade yet.

@github-actions github-actions bot removed the stale? Issue that may be closed soon due to the original author not responding any more. label May 19, 2021
@LekoArts LekoArts added topic: DX Developer Experience (e.g. Fast Refresh, i18n, SSR, page creation, starters) topic: performance Related to runtime & build performance and removed topic: page creation labels May 28, 2021
@github-actions
Copy link

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

@github-actions github-actions bot added the stale? Issue that may be closed soon due to the original author not responding any more. label Jun 18, 2021
@github-actions
Copy link

Hey again!

It’s been 60 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to comment on this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale? Issue that may be closed soon due to the original author not responding any more. topic: DX Developer Experience (e.g. Fast Refresh, i18n, SSR, page creation, starters) topic: performance Related to runtime & build performance type: bug An issue or pull request relating to a bug in Gatsby
Projects
None yet
Development

No branches or pull requests