perf: use must-revalidate cache-control header as common and only create header routes for routes with different cache-control

[pieh]

Description

before on left, with the change in the pr on right for sample reproduction with 10000 dummy pages (this is zoomed in on execution of just this line

gatsby/packages/gatsby-adapter-netlify/src/route-handler.ts

Lines 262 to 263 in 94b2482

	const { redirects, headers, lambdasThatUseCaching, fileMovingPromise } =
	processRoutesManifest(routesManifest, headerRoutes)

):

Note that in above cpu profile biggest drain is actually way content of _headers file is produced and not necessarily amount of them (tho they both combined cause the problem). String concatenations for each route (

gatsby/packages/gatsby-adapter-netlify/src/route-handler.ts

Line 242 in 94b2482

acc += buildHeaderString(curr.path, curr.headers)

) and then headers concatenation in

gatsby/packages/gatsby-adapter-netlify/src/route-handler.ts

Line 136 in 94b2482

acc += ` ${curr.key}: ${curr.value}\n`

is causing allocating a lot of new strings and subsequently create a lot of work for garbage collector (and garbage collection is where majority of time was actually spent). This change just lowers amount of times this need to happen, but possibly that part could be improved (i.e. instead of doing acc += 'string-to-append' we could push entries to array and finally to .join('') on that array to produce final string, but this change alone seems to just mostly avoid hitting issues in the first place

In practice (first reported time is just the build, and second is build + time spent in adapter handling):
before:

info Done building in 25.2874965 sec
✨  Done in 140.97s.

and after

info Done building in 25.716898292 sec
✨  Done in 33.33s.

this also result in much smaller public/_headers file as we no longer create entries for each html and page-data files - see https://www.diffchecker.com/VWVHsM5R/

Documentation

Tests

Updated unit tests + https://github.com/gatsbyjs/gatsby/blob/master/e2e-tests/adapters/cypress/e2e/headers.cy.ts still passing

Related Issues

[mraerino]

:gotta-parrot-fast: 🚀