"update schema" step really slow after upgrading from 2.17 to 2.18

[asilgag]

Description

Site with 20k pages, currently using version 2.17.6. We pass JSON data via context (hence not using GraphQL), so we have switched off type inference for SitePage.context to speed up compilation, as stated here.

These are the timings we get using 2.17.6:

2.17.6 timings
=======================================
success source and transform nodes - 4.247s                                                                                                                    
success building schema - 1.292s                                                                                                                                                                                                                                               
success createPages - 23.121s                                                                                                                                  
success createPagesStatefully - 0.079s                                                                                                                         
success onPreExtractQueries - 0.001s                                                                                                                           
success update schema - 0.051s 

After upgrading from 2.17.6 to 2.18.12, "update schema" step is taking ~40 seconds instead of 0.051s:

2.18.12 timings
=======================================
success source and transform nodes - 5.167s                                                                                                                    
success building schema - 1.206s                                                                                                                                                                                                                                               
success createPages - 27.748s                                                                                                                                  
success createPagesStatefully - 0.108s                                                                                                                         
success onPreExtractQueries - 0.000s                                                                                                                           
success update schema - 41.675s 

It's this problem related to "update schema" step not taking into account that type inference is disabled for SitePage.context?

Steps to reproduce

Our repo is private, but we are willing to provide access to any Gatsby employee (maybe @pvdz)

Expected result

"update schema" step should not take any time, as it did with v2.17.6.

Actual result

It takes ~40 seconds instead of ~0s

Environment

  System:
    OS: Linux 4.15 Ubuntu 18.04.3 LTS (Bionic Beaver)
    CPU: (4) x64 Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    Shell: 4.4.20 - /bin/bash
  Binaries:
    Node: 12.12.0 - ~/.nvm/versions/node/v12.12.0/bin/node
    Yarn: 1.21.1 - /usr/bin/yarn
    npm: 6.13.4 - ~/.nvm/versions/node/v12.12.0/bin/npm
  Languages:
    Python: 2.7.17 - /usr/bin/python
  Browsers:
    Chrome: 79.0.3945.88
    Firefox: 71.0
  npmPackages:
    gatsby: ^2.18.12 => 2.18.17
    gatsby-plugin-emotion: ^4.1.18 => 4.1.18
    gatsby-plugin-eslint: 2.0.8 => 2.0.8
    gatsby-plugin-manifest: ^2.2.34 => 2.2.34
    gatsby-plugin-react-helmet: ^3.1.18 => 3.1.18
    gatsby-plugin-webpack-bundle-analyzer: ^1.0.5 => 1.0.5
    gatsby-source-filesystem: ^2.1.43 => 2.1.43
    gatsby-transformer-json: ^2.2.22 => 2.2.22
  npmGlobalPackages:
    gatsby-cli: 2.8.22
    gatsby: 2.1.19

[pvdz]

There have been some changes to graphql. Could you post the whole output before and after? I'm half expecting (or hoping? :) ) that the time is moved to a different step and that the total runtime is about the same.

If that's not the case I'm definitely going to look into it as this is not a regression I'd be keen on keeping.

[pvdz]

@vladar anything on your radar that might relate to this?

[asilgag]

This is the whole output from 2.17.6:

success open and validate gatsby-configs - 0.855s
success load plugins - 0.875s
success onPreInit - 0.091s
success delete html and css files from previous builds - 0.011s
success initialize cache - 0.015s
success copy gatsby files - 0.022s
success onPreBootstrap - 0.005s
success source and transform nodes - 3.775s
success building schema - 1.190s
success createPages - 22.467s
success createPagesStatefully - 0.059s
success onPreExtractQueries - 0.002s
success update schema - 0.046s
success extract queries from components - 0.372s
success write out requires - 0.100s
success write out redirect data - 0.002s
success Build manifest and related icons - 0.025s
success onPostBootstrap - 0.080s
⠀
info bootstrap finished - 33.478 s
⠀
success Building production JavaScript and CSS bundles - 8.861s
success Rewriting compilation hashes - 0.004s
success run queries - 60.079s - 19896/19896 331.17/s
success Building static HTML for pages - 67.070s - 19888/19888 296.53/s
info Done building in 166.188489269 sec

And this the one from 2.18.12:

success open and validate gatsby-configs - 0.927s
success load plugins - 0.855s
success onPreInit - 0.092s
success delete html and css files from previous builds - 0.010s
success initialize cache - 0.009s
success copy gatsby files - 0.022s
success onPreBootstrap - 0.004s
success createSchemaCustomization - 0.005s
success source and transform nodes - 3.814s
success building schema - 1.223s
success createPages - 26.515s
success createPagesStatefully - 0.077s
success onPreExtractQueries - 0.002s
success update schema - 37.295s
success extract queries from components - 0.386s
success write out requires - 0.109s
success write out redirect data - 0.003s
success Build manifest and related icons - 0.023s
success onPostBootstrap - 0.083s
⠀
info bootstrap finished - 75.192 s
⠀
success Building production JavaScript and CSS bundles - 9.096s
success Rewriting compilation hashes - 0.005s
success run queries - 27.439s - 19896/19896 725.09/s
success Building static HTML for pages - 67.882s - 19888/19888 292.98/s
info Done building in 177.231419989 sec

As you can see:

• "update schema" is slower (0s vs 37s)
• "createPages" is slower (22s vs 26s)
• "run queries" is significantly faster (60s vs 27s)

All in all, the difference between two versions is about ~11 seconds.

Since we have switched off type inference for SitePage.context, shouldn't "update schema" step be faster, or even bypassed? We effectively know that we haven't changed any field after createPages, so could be opt-out of that schema update just like we did with type inference for SitePage.context?

[asilgag]

More info about this issue.

We have removed the "update schema" step (following directions from #14797 (comment)) and our build time has gone down from 177 sec to 127 sec, 50 seconds less:

success open and validate gatsby-configs - 0.851s                                                                                                                                              
success load plugins - 1.102s                                                                                                                                                                                                                                                                                                                    
success onPreInit - 0.188s                                                                                                                                                                     
success delete html and css files from previous builds - 0.009s                                                                                                                                
success initialize cache - 0.007s                                                                                                                                                              
success copy gatsby files - 0.027s                                                                                                                                                             
success onPreBootstrap - 0.002s                                                                                                                                                                
success createSchemaCustomization - 0.004s                                                                                                                                                     
success source and transform nodes - 0.074s                                                                                                                                                    
success building schema - 0.271s                                                                                                                                                                                                                                                                                                                
success createPages - 24.446s                                                                                                                                                                  
success createPagesStatefully - 0.073s                                                                                                                                                         
success onPreExtractQueries - 0.000s                                                                                                                                                           
success extract queries from components - 0.180s                                                                                                                                               
success write out requires - 0.096s                                                                                                                                                            
success write out redirect data - 0.001s                                                                                                                                                       
success Build manifest and related icons - 0.023s                                                                                                                                              
success onPostBootstrap - 0.065s                                                                                                                                                               
⠀                                                                                                                                                                                              
info bootstrap finished - 29.771 s                                                                                                                                                             
⠀                                                                                                                                                                                                                                                         
success Building production JavaScript and CSS bundles - 8.173s                                                                                                                                
success Rewriting compilation hashes - 0.001s                                                                                                                                                  
success run queries - 25.558s - 19888/19888 778.15/s                                                                                                                                           
success Building static HTML for pages - 64.330s - 19888/19888 309.16/s                                                                                                                                                                                                                                                   
info Done building in 126.490407588 sec                                                                                                                                                        
Done in 126.91s.

We haven't detected any issue with the resulting public folder, so we think that the problem comes from Gatsby not properly honoring that we have switched off type inference for SitePage.context. It's that assumption ok?

[vladar]

I need to check this when I have a moment and then will get back to you. Minimum reproduction could have helped a lot (disabling inference for SitePage and then showing that inference still occurs, i.e. SitePageContext type gets additional fields from data).

[vladar]

Confirmed this. A PR with a fix: #20662