fix: add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine #38822

pieh · 2024-01-24T15:52:48Z

Description

This PR fixes issues when fs.createWriteStream (and few others) are being used in functions/lambdas that are mounted in read-only locations

Example error could look like this:

Jan 24, 11:05:05 AM: error [gatsby-source-contentful] Could not getDominantColor from image TypeError: Cannot read properties of undefined (reading 'apply')
Jan 24, 11:05:05 AM: at new WriteStream (/var/task/.cache/query-engine/index.js:6355:29)
Jan 24, 11:05:05 AM: at Object.createWriteStream (/var/task/.cache/query-engine/index.js:6378:12)
Jan 24, 11:05:05 AM: at /var/task/.cache/query-engine/index.js:16689:75
Jan 24, 11:05:05 AM: at new Promise (<anonymous>)
Jan 24, 11:05:05 AM: at requestRemoteNode (/var/task/.cache/query-engine/index.js:16687:10)
Jan 24, 11:05:05 AM: at fetchFile (/var/task/.cache/query-engine/index.js:297107:61)
Jan 24, 11:05:05 AM: at fetchWorker (/var/task/.cache/query-engine/index.js:297015:26)

But any issue (there might be multiple context in which this issue occurs) that looks more or less like this:

TypeError: Cannot read properties of undefined (reading 'apply')
 at new WriteStream (/var/task/.cache/query-engine/index.js:X:Y)
 at Object.createWriteStream (/var/task/.cache/query-engine/index.js:X:Y)

is caused by same root issue which is fact that linked FS (that rewrites path from read-only location to writeable location) didn't cover required FS methods.

This PR adds rewrite handling for:

fs.createWriteStream
fs.createReadStream
fs.rm
new fs.WriteStream
new fs.ReadStream

And add rewrite support for second (to) argument for ( currently only first (from) argument is being rewritten)

fs.rename
fs.renameSync

This PR also handles few issues that were discovered during work on missing FS rewrites:

E2E test for Remote File (Image/File CDN feature that utilize above FS methods) was not actually being SSR which did hide some problems
File CDN url generator was working properly, due to added babel helper that could not be resolved in Netlify Lambda
PathPrefix was not set in SSR/DSG engine and URLs generated for Image/File CDN didn't honor it

Tests

Fixed adapter e2e test fixture to actually use SSR for SSR Remote File tests. Enabled placeholder tests for SSR Remote File page (placeholder generation did not work due to missing FS rewrites)

Related Issues

https://linear.app/netlify/issue/FRA-236/fatal-createwritestream-problem-with-linkfs

pieh · 2024-01-24T16:29:44Z

e2e-tests/adapters/cypress/e2e/remote-file.cy.ts

@@ -29,7 +29,7 @@ const configs = [
  {
    title: `remote-file (SSR, Page Query)`,
    pagePath: `/routes/ssr/remote-file/`,
-    placeholders: false,
+    placeholders: true,


This test was disabled because of problems that this PR is fixing. In context of this test those problems weren't fatal, however it meant that image placeholders were just not working.

The fix in this PR also solves other problems than just placeholders, but this was existing test that already we had

pieh · 2024-01-25T08:55:20Z

packages/gatsby-adapter-netlify/src/file-cdn-url-generator.ts

@@ -1,4 +1,4 @@
-import crypto from "crypto"
+import { createHash } from "crypto"


This caused some problems with

Cannot find module '@babel/runtime/helpers/interopRequireDefault'

in lambdas, so easiest way was just to avoid having babel adding that

pieh · 2024-01-25T08:55:52Z

packages/gatsby/src/schema/graphql-engine/entry.ts

+  private setupPathPrefix(pathPrefix: string): void {
+    if (pathPrefix) {
+      store.dispatch({
+        type: `SET_PROGRAM`,
+        payload: {
+          prefixPaths: true,
+        },
+      })
+
+      store.dispatch({
+        type: `SET_SITE_CONFIG`,
+        payload: {
+          ...store.getState().config,
+          pathPrefix,
+        },
+      })
+    }
+  }
+


needed for pathPrefix to be actually honored when generating FILE_CDN urls

pieh · 2024-01-25T08:56:43Z

e2e-tests/adapters/src/pages/routes/ssr/remote-file.jsx

+export function getServerData() {
+  return {
+    props: {
+      ssr: true,
+    },
+  }
+}


This page wasn't actually SSR... Making it SSR uncovered some other problems that this PR is also fixing

… in dsg/ssr engine

kathmbeck · 2024-01-25T14:30:46Z

packages/gatsby/src/utils/page-ssr-module/lambda.ts

+    // @ts-ignore TS doesn't like extending prototype "classes"
+    lfs.ReadStream = LinkedReadStream
+
+    const dbPath = path.join(TEMP_CACHE_DIR, `data`, `datastore`)



Seems like there's some repetition here with extending fs.WriteStream and fs.ReadStream, you could consider extracting/reusing.

Yes, it's basically the same thing twice (just once for WriteStream and once for ReadStream - I think it's confusing as it is already and trying to create reusable utility would probably make debugging / understanding it worse than it already is.

Doing class ... extends would be much more readable, but you can't actually do that is "class" you extend is prototype one like fs.WriteStream and fs.ReadStream are ( https://github.com/nodejs/node/blob/6b6bcee747c2007117262efb2ff6d61ea888f499/lib/internal/fs/streams.js#L321-L398 being one of them - the other is also in that module)

Got it, I'm ok leaving as is.

…ssr engine (#38822) * test(adapters-e2e): enable placeholder tests in ssr remote-file * fix(gatsby-adapter-netlify): bundling file-cdn * fix(gatsby): set pathPrefix in engines * fix(gatsby): add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine (cherry picked from commit bbdddd7)

…ssr engine (#38822) (#38823) * test(adapters-e2e): enable placeholder tests in ssr remote-file * fix(gatsby-adapter-netlify): bundling file-cdn * fix(gatsby): set pathPrefix in engines * fix(gatsby): add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine (cherry picked from commit bbdddd7) Co-authored-by: Michal Piechowiak <misiek.piechowiak@gmail.com>

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jan 24, 2024

pieh added topic: render-mode Related to Gatsby's different rendering modes topic: media Related to gatsby-plugin-image, or general image/media processing topics and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Jan 24, 2024

pieh commented Jan 24, 2024

View reviewed changes

pieh force-pushed the fix/lambda-writestream-linkfs branch from 1d97c5a to b6d9442 Compare January 24, 2024 16:34

pieh commented Jan 25, 2024

View reviewed changes

pieh added 4 commits January 25, 2024 10:21

test(adapters-e2e): enable placeholder tests in ssr remote-file

8454f94

fix(gatsby-adapter-netlify): bundling file-cdn

e9d20a9

fix(gatsby): set pathPrefix in engines

c08ed55

fix(gatsby): add missing fs method rewrites to handle fetchRemoteFile…

9d8b979

… in dsg/ssr engine

pieh force-pushed the fix/lambda-writestream-linkfs branch from 903659a to c08ed55 Compare January 25, 2024 09:23

pieh marked this pull request as ready for review January 25, 2024 12:29

kathmbeck reviewed Jan 25, 2024

View reviewed changes

kathmbeck approved these changes Jan 25, 2024

View reviewed changes

pieh merged commit bbdddd7 into master Jan 25, 2024
34 checks passed

pieh deleted the fix/lambda-writestream-linkfs branch January 25, 2024 14:48

gatsbybot mentioned this pull request Jan 25, 2024

fix: add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine (#38822) #38823

Merged

X-oss-byte mentioned this pull request May 8, 2024

[Snyk] Upgrade gatsby from 5.12.4 to 5.13.4 X-oss-byte/gatsby-1#710

Open

monizb mentioned this pull request May 8, 2024

[Snyk] Security upgrade gatsby-source-contentful from 2.3.56 to 8.13.2 monizb/gatsby#1452

Open

This was referenced Oct 17, 2024

[Snyk] Upgrade gatsby from 5.0.0 to 5.13.7 X-oss-byte/gatsby-1#921

Open

[Snyk] Upgrade gatsby-source-wordpress from 7.0.0 to 7.13.5 X-oss-byte/gatsby-1#922

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine #38822

fix: add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine #38822

pieh commented Jan 24, 2024 •

edited

Loading

pieh Jan 24, 2024 •

edited

Loading

pieh Jan 25, 2024

pieh Jan 25, 2024

pieh Jan 25, 2024 •

edited

Loading

kathmbeck Jan 25, 2024

pieh Jan 25, 2024

pieh Jan 25, 2024

kathmbeck Jan 25, 2024

		@@ -1,4 +1,4 @@
		import crypto from "crypto"
		import { createHash } from "crypto"

fix: add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine #38822

fix: add missing fs method rewrites to handle fetchRemoteFile in dsg/ssr engine #38822

Conversation

pieh commented Jan 24, 2024 • edited Loading

Description

Tests

Related Issues

pieh Jan 24, 2024 • edited Loading

Choose a reason for hiding this comment

pieh Jan 25, 2024

Choose a reason for hiding this comment

pieh Jan 25, 2024

Choose a reason for hiding this comment

pieh Jan 25, 2024 • edited Loading

Choose a reason for hiding this comment

kathmbeck Jan 25, 2024

Choose a reason for hiding this comment

pieh Jan 25, 2024

Choose a reason for hiding this comment

pieh Jan 25, 2024

Choose a reason for hiding this comment

kathmbeck Jan 25, 2024

Choose a reason for hiding this comment

pieh commented Jan 24, 2024 •

edited

Loading

pieh Jan 24, 2024 •

edited

Loading

pieh Jan 25, 2024 •

edited

Loading