From 7c8ce647cd676c008810794a41631a2cbc2536b2 Mon Sep 17 00:00:00 2001 From: Peter van der Zee Date: Tue, 2 Jun 2020 12:17:43 +0200 Subject: [PATCH 1/2] perf(gatsby): drop severe scaling regression caused by analytics Regression introduced in https://github.com/gatsbyjs/gatsby/pull/22851 The problem seems to be that these calls to `v8.serialize` trigger the gc to start a full hold-the-world mark-and-sweep step sooner. In a benchmark of 150k files, the step would trigger almost always after between 100k and 110k queries had run, and it would pause the process for 60+ seconds. Example benchmark results from before and after that PR: ``` info bootstrap finished - 86.758 s success Building production JavaScript and CSS bundles - 9.404s success run queries - 205.676s - 150002/150002 729.31/s success Building static HTML for pages - 142.800s - 150002/150002 1050.44/s info Done building in 451.33 sec ``` ``` info bootstrap finished - 85.933 s success Building production JavaScript and CSS bundles - 8.335s success run queries - 84.795s - 150002/150002 1769.00/s success Building static HTML for pages - 141.000s - 150002/150002 1063.84/s info Done building in 320.158 sec ``` This is very consistent behavior. We looked at the change and agreed that the best was to just drop this measurement since it was for the sake of analytics and a non-vital metric to record. We'd rather have the perf than the metric. Numbers for the fix, same benchmark, first on current master and then on this PR: ``` info bootstrap finished - 79.788s success Building production JavaScript and CSS bundles - 9.635s success run queries - 201.542s - 150002/150002 744.27/s success Building static HTML for pages - 141.535s - 150002/150002 1059.82/s info Done building in 440.766 sec ``` ``` info bootstrap finished - 80.751s success Building production JavaScript and CSS bundles - 9.570s success run queries - 87.162s - 150002/150002 1720.95/s success Building static HTML for pages - 142.609s - 150002/150002 1051.84/s info Done building in 319.151 sec ``` nice! --- packages/gatsby/src/query/graphql-runner.ts | 7 ------- 1 file changed, 7 deletions(-) diff --git a/packages/gatsby/src/query/graphql-runner.ts b/packages/gatsby/src/query/graphql-runner.ts index 24d1dc9594331..b78c201e902cb 100644 --- a/packages/gatsby/src/query/graphql-runner.ts +++ b/packages/gatsby/src/query/graphql-runner.ts @@ -156,13 +156,6 @@ export class GraphQLRunner { if (typeof statsQuery !== `string`) { statsQuery = statsQuery.body } - this.stats.uniqueOperations.add( - crypto - .createHash(`sha1`) - .update(statsQuery) - .update(v8.serialize(context)) - .digest(`hex`) - ) this.stats.uniqueQueries.add( crypto.createHash(`sha1`).update(statsQuery).digest(`hex`) From 9eaa7a82e6db4e810a1debeb4aed3a03072198ef Mon Sep 17 00:00:00 2001 From: Peter van der Zee Date: Tue, 2 Jun 2020 14:00:25 +0200 Subject: [PATCH 2/2] And drop the import --- packages/gatsby/src/query/graphql-runner.ts | 1 - 1 file changed, 1 deletion(-) diff --git a/packages/gatsby/src/query/graphql-runner.ts b/packages/gatsby/src/query/graphql-runner.ts index b78c201e902cb..e4c6490a6ac12 100644 --- a/packages/gatsby/src/query/graphql-runner.ts +++ b/packages/gatsby/src/query/graphql-runner.ts @@ -1,5 +1,4 @@ import crypto from "crypto" -import v8 from "v8" import { Span } from "opentracing" import { parse,