enhancement/Issue-1118: Single File Bundles for SSR and API routes

[DevLab2425]

Related Issue

Resolves #1118

Summary of Changes

• Refactor the API Rollup config generation to create a single entry point for each API route, rather than a single Rollup process for all API routes.
• Refactor the SSR Rollup config generation to mimic the API generation, in that, the SSR array is now separate input config files.
• Refactor SSR pages entry point "facade" implementation in bundle lifecycle to use URL so we can bundle instead of the hacky string replace solution
• Documented breaking changes in adapter plugin docs

Demo implementation in - ProjectEvergreen/greenwood-demo-adapter-vercel#22

TODO

1. Documentation
2. automatic SSR chunks / bundle mapping (nice to have)
   • actually, this is just the same SSR pages issue with us now using URL, otherwise it would just be inlined, fwiw
   • just did this for API routes for now, will circle back on SSR pages as part of Isomorphic Asset Bundling #1163
3. fix windows test cases
4. WCC Upstreams and delete patches/ + postinstall
   • Some sort of bug with clobbered custom element definitions - registration of a custom element is being assumed when initializing a custom element wcc#144
   • Seems like I may have introduced an infinite loop bug? 😮‍💨 - duplicate loads of the DOM shim leads to an infinite rendering issue wcc#147
5. Should create a discussion to see if there's a better alternative to the URL mechanism, since its good for bundling, but forces a chunk since it really can't be inlined... which isn't the worst thing in the world I suppose 🤷‍♂️
   • should capture this a comment in adapter plugins that have to find / filter for route chunk files to copy over to adapter output directories
   • URL bundled assets and Single File Executable (SFE) #1204

[DevLab2425]

@thescientist13

I'm not totally sure if this PR is ready. At the moment, there are 50+ 80+ test failures, all seemingly around the fetch call to the Artists API. One example is here:

greenwood/packages/cli/test/cases/build.default.ssr-static-export/src/pages/artists.js

Line 34 in 8fad6f7

const artists = await fetch('http://www.analogstudios.net/api/artists').then(resp => resp.json());

If that call does not return an array, the suite fails. I attempted to add some defensive code as a test, which resulted in a single failure of a spec due to a mismatch of values. The code below allows for the process to continue, but it doesn't produce the desired outcome...obviously.

const artistsListItems = Array.isArray(artists) ? artists : []
    .filter(artist => artist.isActive === '1')
    .map((artist) => {

When hitting the API directly in the browser, I get the same error I'm seeing from the fetch
https://www.analogstudios.net/api/artists

[thescientist13]

My bad @DevLab2425 🤦

Please see #1188 for the fix 😇

[DevLab2425]

@thescientist13
I'm struggling to make sense of what I'm seeing at the moment and starting to feel like I'm chasing my tail.

If I isolate a suite of tests, that suite will typically pass. However, anytime I attempt to run the full set of tests, I get errors from various suites outside of the bundling logic I've changed.

Here's an example of the failures I'm seeing outside of the build.config.plugins-adapter.spec.js.

As I work through the errors I'm seeing things like in-use ports, missing file imports, etc. I'm just not sure whether what I'm seeing is environmental or code-based due to my changes. Could we setup some time later this week to help me see where my blast zone might be, and what might be a red herring?

Also, why type of tests are running with test:exp? They seem to be E2E since the CLI is multi-layered, but, are the test cases independent/standalone, or do they rely on suites running in sequence? Is using .only a recommended approach for this? (I thought that's what you were doing during the walkthrough).

The latest suite I'm chasing is the serve.default.api.spec.js suite. The api/fragment.js is failing to be created, but the other API files do get created for those tests. That sounds like it could be bundling related for sure, but since other files are being generated, maybe not?

Serve command with API specific behaviors for an HTML ("fragment") API
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/Users/paul/Projects/ProjectEvergreen/greenwood/packages/cli/test/cases/serve.default.api/public/api/fragment.js' imported from /Users/paul/Projects/ProjectEvergreen/greenwood/packages/cli/src/lifecycles/serve.js
    at new NodeError (node:internal/errors:393:5)
    at finalizeResolution (node:internal/modules/esm/resolve:323:11)
    at moduleResolve (node:internal/modules/esm/resolve:916:10)
    at defaultResolve (node:internal/modules/esm/resolve:1124:11)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:841:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ESMLoader.import (node:internal/modules/esm/loader:525:22)
    at importModuleDynamically (node:internal/modules/esm/translators:110:35)
    at importModuleDynamicallyCallback (node:internal/process/esm_loader:35:14) {
  code: 'ERR_MODULE_NOT_FOUND'
}

 0 passing (10s)
 4 failing

🤷

[thescientist13]

If we want to do something async in a loop, we should use a for ... of or for ... in instead. forEech is synchronous and combining these two can lead to unexpected behaviors (something I would like to more thoroughly clean up in #823 )

for (const rollupConfig of apiConfigs) {
  const bundle = await rollup(rollupConfig);
  await bundle.write(rollupConfig.output);
}

[thescientist13]

As I work through the errors I'm seeing things like in-use ports, missing file imports, etc. I'm just not sure whether what I'm seeing is environmental or code-based due to my changes.

Could you be running other Greenwood project's at the same time by chance, occupying ports 1984 or 8080 / 8181 (I definitely do this from time to time)? Also, depending on the process exits when running tests, it might not teardown the server running as part of a test case, thus causing a zombie process.

One way to confirm if a process is running is run yarn develop or yarn serve in this project and see if either of them crash. You can also do a ps aux | grep node to find running Node instances and see if develop or serve commands are running. You can then kill the process using kill -9 <PID>

Another tip is to pass the debug flag to new Runner to get all console output from Greenwood

runner = new Runner(true);

Fwiw, I ran all the tests locally from this branch and only got 8 tests failing, exclusively related to API behaviors, which is also what CI is reporting, so seems like your blast radius is small 💣 😃

    3070 passing (5m)
  30 pending
  8 failing


Serve Greenwood With:

API Routes

Serve command with API specific behaviors for an HTML ("fragment") API

should return a 200 status:
AssertionError: expected 500 to equal 200

expected - actual

-500

+200
at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:109:36)

at process.processImmediate (node:internal/timers:471:21)


Serve Greenwood With:

API Routes

Serve command with API specific behaviors for an HTML ("fragment") API

should return a custom status message:
AssertionError: expected 'Internal Server Error' to equal 'SUCCESS!!!'

expected - actual

-Internal Server Error

+SUCCESS!!!
at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:114:40)

at process.processImmediate (node:internal/timers:471:21)


Serve Greenwood With:

API Routes

Serve command with API specific behaviors for an HTML ("fragment") API

should return the correct content type:
AssertionError: expected 'text/plain; charset=utf-8' to equal 'text/html'

expected - actual

-text/plain; charset=utf-8

+text/html
at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:119:57)

at process.processImmediate (node:internal/timers:471:21)


Serve Greenwood With:

API Routes

Serve command with API specific behaviors for an HTML ("fragment") API

should return the correct response body:

AssertionError: expected 'Internal Server Error' to include '
Hello Greenwood!!!
'

at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:124:25)

at process.processImmediate (node:internal/timers:471:21)



Serve Greenwood With:

A Server Rendered Application (SSR) with API Routes importing TypeScript

Serve command with API specific behaviors for an HTML ("fragment") API

should return a 200 status:
AssertionError: expected 500 to equal 200

expected - actual

-500

+200
at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:75:36)

at process.processImmediate (node:internal/timers:471:21)


Serve Greenwood With:

A Server Rendered Application (SSR) with API Routes importing TypeScript

Serve command with API specific behaviors for an HTML ("fragment") API

should return a custom status message:
AssertionError: expected 'Internal Server Error' to equal 'OK'

expected - actual

-Internal Server Error

+OK
at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:79:40)

at process.processImmediate (node:internal/timers:471:21)


Serve Greenwood With:

A Server Rendered Application (SSR) with API Routes importing TypeScript

Serve command with API specific behaviors for an HTML ("fragment") API

should return the correct content type:
AssertionError: expected 'text/plain; charset=utf-8' to equal 'text/html'

expected - actual

-text/plain; charset=utf-8

+text/html
at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:83:57)

at process.processImmediate (node:internal/timers:471:21)


Serve Greenwood With:

A Server Rendered Application (SSR) with API Routes importing TypeScript

Serve command with API specific behaviors for an HTML ("fragment") API

should make sure to have the expected CSS inlined into the page for each :
AssertionError: expected 0 to equal 2

expected - actual

-0

+2
at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:89:42)

at process.processImmediate (node:internal/timers:471:21)


error Command failed with exit code 8.

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

---

Also, why type of tests are running with test:exp? They seem to be E2E since the CLI is multi-layered, but, are the test cases independent/standalone, or do they rely on suites running in sequence?

All test cases are pretty much the same, they run the CLI and validate some output, so in the context of a CLI app I suppose they would be E2E tests? (technically though for something like the static router feature, I would love to run it in a browser). But no, the answer is more boring than that.

The test:exp is just a way to distinguish tests that need to run with the --experimental-loader flag for NodeJS set, that enables Greenwood to support Custom Imports. You will see these test cases prefixed with exp- in their folder / file name.

Is using .only a recommended approach for this? (I thought that's what you were doing during the walkthrough).

Not related to test:exp, it's a just a convenient way to "solo" / isolate instead of running all the specs. Often I'm just iterating TDD style with one test case until I get it right, then I run against the rest and hope for the best. 😅

But then if others fail, I will .only my way towards solving them all.

---

The latest suite I'm chasing is the serve.default.api.spec.js suite. The api/fragment.js is failing to be created, but the other API files do get created for those tests. That sounds like it could be bundling related for sure, but since other files are being generated, maybe not?

I wonder if it could be related to #1186 (comment)? I confirmed that rollup config had an entry when logging input but yeah, no output file. I would see if refactoring those for loops helps and I can take a look into it as well, since the code seems pretty on the 💰 from what I can tell.

[DevLab2425]

I have passing tests locally, but the CI is still failing many times. I'll use the CI reports to try to track things down, but having difficulty identifying things without failures locally.

[thescientist13]

Hmm, not sure why GitHub Actions is failing so hard here, even after re-running the jobs, but I ran yarn test:exp locally and all the specs passed. 😕

While i look into that, I think last steps would be to step into each of the adapter plugins and see if we can now remove all the extraneous file copying we had to do.

• Vercel - https://github.com/ProjectEvergreen/greenwood/blob/master/packages/plugin-adapter-vercel/src/index.js#L107
• Netlify - https://github.com/ProjectEvergreen/greenwood/blob/master/packages/plugin-adapter-netlify/src/index.js#L118

Thanks for all this, please keep the faith! 😇

---

I'm going to play around with this locally a bit to examine the new output and will aim to patch these changes into one of the respective demo repos to do some final validation testing, which I think I can do this weekend. (and also look into the test cases <> GitHub Actions.)

[thescientist13]

minor tip: but for each of these loops, if we use a for ... of, we can get access to the index directly.

async function bundleApiRoutes(compilation) {
  const apiConfigs = await getRollupConfigForApis(compilation);

  if (apiConfigs.length > 0 && apiConfigs[0].input.length !== 0) {
    for (const config of apiConfigs) {
      const bundle = await rollup(config);
      await bundle.write(config.output);
    }
  }
}

[thescientist13]

@DevLab2425
OK, so I think I found the culprit. Looks like Greenwood fails on Node 18.19.0, which is the version I see running in GitHub Actions. If I nvm use 18.19.0 on my machine, I can reproduce the error in CI and with some additional logging can get some helpful information. I basically see a lot of errors like this

TypeError: Failed to parse URL from file://file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/node_modules/lit-html/lit-html.js.map
    at new Request (node:internal/deps/undici/undici:5272:19)
    at NodeModulesResource.resolve (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/plugins/resource/plugin-node-modules.js:41:12)
    ... 5 lines matching cause stack trace ...
    at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
    at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
    at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37 {
  [cause]: TypeError [ERR_INVALID_URL]: Invalid URL
      at new NodeError (node:internal/errors:405:5)
      at new URL (node:internal/url:676:13)
      at new Request (node:internal/deps/undici/undici:5270:25)
      at NodeModulesResource.resolve (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/plugins/resource/plugin-node-modules.js:41:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:44:29
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37 {
    input: 'file://file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/node_modules/lit-html/lit-html.js.map',
    code: 'ERR_INVALID_URL'
  }
}

Taking a look, maybe it's something was doing wrong, or perhaps a regression in Node? What version are you using?
Perhaps something related to package exports changed or went unflagged or something?

Worst case we can pin CI to 18.15.0 and i can open issue to tracking 18.19.0 compat. Will report back! 🫡

---

edit 2: OOOOOOH, I get it now. import.meta.resolve is now supported in NodeJS and so this code is running successfully for the first time ever lol 🤦

It's obviously breaking in some way.

So this weekend let me make an issue for tracking this bug with 18.19.0 and if it's a quick fix then i will submit a PR, otherwise I can submit a PR to just pin this project's CI to 18.15.0 in the meantime.

edit 1:

By changing this line to omit the file:// I got it down to one error for the test case i was working against

  1) Develop Greenwood With:
       Default Greenwood Configuration and Workspace
         Develop command specific HTML behaviors
           should return an import map shim <script> in the <head> of the document:
     AssertionError: expected undefined to equal '/node_modules/regenerator-runtime/runtime.js'
      at file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/develop.default/develop.default.spec.js:488:37
      at Array.forEach (<anonymous>)
      at Context.<anonymous> (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/develop.default/develop.default.spec.js:487:40)
      at process.processImmediate (node:internal/timers:476:21)

That could hint to a package exports / node module resolution change on NodeJS side, so might require a bit more investigation, so let me tinker with this some more and we'll get some way forward on this ASAP. (perhaps time to escalate this issue - #684 . 🤔 )

[thescientist13]

Hey @DevLab2425 , just have some updates to share:

Regarding the build failures, I've opened #1191 to remove the offending code path completely until I can properly fix / implement it later in #684. For local dev, I would just use Node < 18..19.0

I'll plan to get it merged Wednesday evening after the weekly meeting.

---

I also got around to doing some testing over the weekend of these changes into our Vercel demo repo and it worked great! I even patched in some cleanup of the Vercel adapter to remove a few of the cp calls we were making and was able to see a definite reduction in file generated. 💪

# before CLI
➜  greenwood-demo-adapter-vercel git:(main) tree public
public
├── 1733327959.ed8fa148.js.map
├── 396762157.af333475.js.map
├── 404.html
├── 758806547.5433709a.js.map
├── 949355521.f4273cc9.js.map
├── __products.js
├── _products.js
├── api
│   ├── assets
│   ├── card.51aef4b7.js
│   ├── fragment.js
│   ├── greeting.js
│   ├── products.c34c47ae.js
│   └── search.js
├── card.51aef4b7.js
├── card.51aef4b7.js.map
├── favicon.ico
├── graph.json
├── index.html
├── manifest.json
├── resources.json
├── search
│   └── index.html
└── styles
    └── main.19738545.css

5 directories, 21 files

# after CLI
➜  greenwood-demo-adapter-vercel git:(chore/upgrade-greenwood-0.30.0) ✗ tree public
public
├── 1733327959.ed8fa148.js.map
├── 396762157.af333475.js.map
├── 404.html
├── 758806547.5433709a.js.map
├── 949355521.f4273cc9.js.map
├── __products.js
├── _products.js
├── api
│   ├── assets
│   ├── card.51aef4b7.js
│   ├── fragment.js
│   ├── greeting.js
│   └── search.js
├── card.51aef4b7.js
├── card.51aef4b7.js.map
├── favicon.ico
├── graph.json
├── index.html
├── manifest.json
├── resources.json
├── search
│   └── index.html
└── styles
    └── main.19738545.css

5 directories, 20 files

# before adapter
➜  greenwood-demo-adapter-vercel git:(89730d7) ✗ tree .vercel
.vercel
└── output
    ├── config.json
    ├── functions
    │   ├── api
    │   │   ├── fragment.func
    │   │   │   ├── assets
    │   │   │   ├── card.51aef4b7.js
    │   │   │   ├── fragment.js
    │   │   │   ├── index.js
    │   │   │   ├── package.json
    │   │   │   └── products.c34c47ae.js
    │   │   ├── greeting.func
    │   │   │   ├── assets
    │   │   │   ├── card.51aef4b7.js
    │   │   │   ├── greeting.js
    │   │   │   ├── index.js
    │   │   │   ├── package.json
    │   │   │   └── products.c34c47ae.js
    │   │   └── search.func
    │   │       ├── assets
    │   │       ├── card.51aef4b7.js
    │   │       ├── index.js
    │   │       ├── package.json
    │   │       ├── products.c34c47ae.js
    │   │       └── search.js
    │   └── products.func
    │       ├── __products.js
    │       ├── _products.js
    │       ├── card.51aef4b7.js
    │       ├── index.js
    │       └── package.json
    └── static
        ├── 1733327959.ed8fa148.js.map
        ├── 396762157.af333475.js.map
        ├── 404.html
        ├── 758806547.5433709a.js.map
        ├── 949355521.f4273cc9.js.map
        ├── __products.js
        ├── _products.js
        ├── api
        │   ├── assets
        │   ├── card.51aef4b7.js
        │   ├── fragment.js
        │   ├── greeting.js
        │   ├── products.c34c47ae.js
        │   └── search.js
        ├── card.51aef4b7.js
        ├── card.51aef4b7.js.map
        ├── favicon.ico
        ├── graph.json
        ├── index.html
        ├── manifest.json
        ├── resources.json
        ├── search
        │   └── index.html
        └── styles
            └── main.19738545.css

16 directories, 42 files

# after adapter
➜  greenwood-demo-adapter-vercel git:(chore/upgrade-greenwood-0.30.0) ✗ tree .vercel
.vercel
└── output
    ├── config.json
    ├── functions
    │   ├── api
    │   │   ├── fragment.func
    │   │   │   ├── card.51aef4b7.js
    │   │   │   ├── fragment.js
    │   │   │   ├── index.js
    │   │   │   └── package.json
    │   │   ├── greeting.func
    │   │   │   ├── card.51aef4b7.js
    │   │   │   ├── greeting.js
    │   │   │   ├── index.js
    │   │   │   └── package.json
    │   │   └── search.func
    │   │       ├── card.51aef4b7.js
    │   │       ├── index.js
    │   │       ├── package.json
    │   │       └── search.js
    │   └── products.func
    │       ├── __products.js
    │       ├── _products.js
    │       ├── index.js
    │       └── package.json
    └── static
        ├── 1733327959.ed8fa148.js.map
        ├── 396762157.af333475.js.map
        ├── 404.html
        ├── 758806547.5433709a.js.map
        ├── 949355521.f4273cc9.js.map
        ├── __products.js
        ├── _products.js
        ├── api
        │   ├── assets
        │   ├── card.51aef4b7.js
        │   ├── fragment.js
        │   ├── greeting.js
        │   └── search.js
        ├── card.51aef4b7.js
        ├── card.51aef4b7.js.map
        ├── favicon.ico
        ├── graph.json
        ├── index.html
        ├── manifest.json
        ├── resources.json
        ├── search
        │   └── index.html
        └── styles
            └── main.19738545.css

13 directories, 37 files

One call out is that when using new URL in a file, a chunk will always be generated as opposed to being "inlined" when it is an import since that's just how that works. As you can see in the output above, I still had to copy card.js into every API function folder, even though it is technically only needed for two of the APIs.

It would be interesting to see if there's anyway to link these to the entry point that produced it, so we could have a stronger coupling between assets of an API or SSR page. Probably something I can make another ticket for.

---

After Adapter (Final)

➜  greenwood-demo-adapter-vercel git:(chore/upgrade-greenwood-0.30.0) ✗ tree .vercel 
.vercel
└── output
    ├── config.json
    ├── functions
    │   ├── api
    │   │   ├── fragment.func
    │   │   │   ├── card.51aef4b7.js
    │   │   │   ├── fragment.js
    │   │   │   ├── index.js
    │   │   │   └── package.json
    │   │   ├── greeting.func
    │   │   │   ├── greeting.js
    │   │   │   ├── index.js
    │   │   │   └── package.json
    │   │   └── search.func
    │   │       ├── card.51aef4b7.js
    │   │       ├── index.js
    │   │       ├── package.json
    │   │       └── search.js
    │   └── products.func
    │       ├── index.js
    │       ├── package.json
    │       ├── products.route.chunk.d46126ed.js
    │       └── products.route.js
    └── static
        ├── 1733327959.ed8fa148.js.map
        ├── 396762157.af333475.js.map
        ├── 404.html
        ├── 758806547.5433709a.js.map
        ├── 949355521.f4273cc9.js.map
        ├── api
        │   ├── card.51aef4b7.js
        │   ├── fragment.js
        │   ├── greeting.js
        │   └── search.js
        ├── card.51aef4b7.js
        ├── card.51aef4b7.js.map
        ├── favicon.ico
        ├── graph.json
        ├── index.html
        ├── manifest.json
        ├── products.route.chunk.d46126ed.js
        ├── products.route.js
        ├── resources.json
        ├── search
        │   └── index.html
        └── styles
            └── main.19738545.css

12 directories, 36 files

[thescientist13]

OK, #1190 has been merged 👍

[thescientist13]

Sweet, felt to clean up some things here!

Just need to publish some upstream changes for WCC then this will be good to go!