URL bundled assets and Single File Executable (SFE)

[thescientist13]

Overview

Coming out of the PR for #1186 , a couples techniques / refactorings were applied around the usage of URL for consuming assets with project files and for combining user page routes and the underlying executeRouteModule implementation.

For example, a user could write this in an API route

import { renderFromHTML } from 'wc-compiler';
import { getArtists } from '../services/artists.js';

export async function handler(request) {
  const params = new URLSearchParams(request.url.slice(request.url.indexOf('?')));
  const offset = params.has('offset') ? parseInt(params.get('offset'), 10) : null;
  const headers = new Headers({ 'Content-Type': 'text/html' });
  const artists = getArtists(offset);
  const { html } = await renderFromHTML(`
    ${
      artists.map((item, idx) => {
        const { name, imageUrl } = item;

        return `
          <app-card
            title="${offset + idx + 1}) ${name}"
            thumbnail="${imageUrl}"
          ></app-card>
        `;
      }).join('')
    }
  `, [
    new URL('../components/card.js', import.meta.url)
  ]);

  return new Response(html, { headers });
}

And when bundling SSR pages for production, Greenwood does something like this

async function bundleSsrPages(compilation) {

  // ...
  const hasSSRPages = compilation.graph.filter(page => page.isSSR).length > 0;
  const input = [];

  if (!compilation.config.prerender && hasSSRPages) {
    const { executeModuleUrl } = compilation.config.plugins.find(plugin => plugin.type === 'renderer').provider();
    const { pagesDir, scratchDir } = compilation.context;

    for (const page of compilation.graph) {
      if (page.isSSR && !page.prerender) {
        const pagesPathDiff = compilation.context.pagesDir.pathname.replace(compilation.context.projectDirectory.pathname, '');

        // ...

        await fs.writeFile(entryFileUrl, `
          import { executeRouteModule } from '${normalizePathnameForWindows(executeModuleUrl)}';

          const moduleUrl = new URL('../${pagesPathDiff}${filename}', import.meta.url);

          export async function handler(request) {
            const compilation = JSON.parse('${JSON.stringify(compilation)}');
            const page = JSON.parse('${JSON.stringify(page)}');
            const data = await executeRouteModule({ moduleUrl, compilation, page, request });
            // ...
        `);

        input.push(normalizePathnameForWindows(entryFileUrl));
      }
    }

    const ssrConfigs = await getRollupConfigForSsr(compilation, input);

    if (ssrConfigs.length > 0 && ssrConfigs[0].input !== '') {
      for (const configIndex in ssrConfigs) {
        const rollupConfig = ssrConfigs[configIndex];
        const bundle = await rollup(rollupConfig);
        await bundle.write(rollupConfig.output);
      }
    }
  }
}

Problem Statement

Not so much a "problem" per se, since this technique works really and is standards compliant, but will never allow us to totally self contain a bunch of files down to one. For example, when bundling an SSR page, you will always have a single output file (good) but also need a "chunk" file that is the by product of the above mentioned bundling approach.

public/
  artists.route.js
  artists.route.chunk.41392711.js

Or for an API route, an adjacent chunk

public/
  api/
    fragment.js
    card.6d3096d9.js

It does mean for adapters they do need to do something like, and manually copy over any assets of say an API route

  for (const [key, value] of apiRoutes.entries()) {
    const outputType = 'api';
    const id = key.replace(`${basePath}/api/`, '');
    const outputRoot = new URL(`./${basePath}/api/${id}.func/`, adapterOutputUrl);
    const { assets = [] } = value;

    await setupFunctionBuildFolder(id, outputType, outputRoot);

    await fs.cp(
      new URL(`./api/${id}.js`, outputDir),
      new URL(`./${id}.js`, outputRoot),
      { recursive: true }
    );

    for (const asset of assets) {
      await fs.cp(
        new URL(asset),
        new URL(`./${asset.split(path.sep).pop()}`, outputRoot),
        { recursive: true }
      );
    }
  }

h3. Thoughts

So not so much a good or bad thing, or if anyone would really notice and or want it improved from a user perspective, but there is something really nice a about a final single file artifact that can be deployed all one. Maybe there is a way to convert a URL into something more like an import statement that could then be inlined?

Anyway, more thinking out loud, hence the discussion.

[thescientist13]

Just a heads up on this bug that came up during alpha testing the original feature - #1210

[thescientist13]

So ran into an issue in #1465 where-in bundling an API route (using the AWS SDK) emitted a bunch of extra bundle chunks

In a couple attempts to solve this problem, I found some seemingly promising results to truly achieve SFEs, but none came "quite" close, with a lot of the ideas coming out from this Rollup issue

output.file

So one suggestion was to set output.file to truly ensure a single file is emitted

// before
output: {
  dir: `${normalizePathnameForWindows(outputDir)}/api`,
  entryFileNames: `${id}.js`,
  chunkFileNames: `${id}.[hash].js`,
},

// after
output: {
  file: `${normalizePathnameForWindows(outputDir)}/api/${id}.js`,
},

However, as we emit applicable usages of new URL as chunks, to ensure they get included in Rollup's module graph, this conflicts with this setting

Error [RollupError]: Invalid value for option "output.file" - when building multiple chunks, the "output.dir" option must be used, not "output.file". To inline dynamic imports, set the "inlineDynamicImports" option.

The other option for emitFile is as an asset, but that automatically puts things into an assets/ directory but is only emitted as an arbitrary file with its own required tracking, which I don't think is what we want in the case of a JS file?

There are a lot more options here, so perhaps this could work with new URL, because if it did, this would be ideal outcome since testing this out in the source project resulted in exactly what we wanted, a single bundle! 💯

inlineDynamicImports

Another idea from that Rollup issue was to use inlineDynamicImports which with just standard JS imports, also resulted in a similar single file bundle

// before
output: {
  dir: `${normalizePathnameForWindows(outputDir)}/api`,
  entryFileNames: `${id}.js`,
  chunkFileNames: `${id}.[hash].js`,
},

// after
output: {
  dir: `${normalizePathnameForWindows(outputDir)}/api`,
  entryFileNames: `${id}.js`,
  chunkFileNames: `${id}.[hash].js`,
  inlineDynamicImports: true,
},

However, this seems to confuse Rollup as we seemingly get circular references, which is also breaking SSR

WARNING: customElement <app-card> is not defined.  You may not have imported it.
WARNING: customElement <app-card> is not defined.  You may not have imported it.

For example, a Fragment API will output these files, starting with public/api/fragment.js

// ok looks good...
export { h as handler } from './fragment.BGdBeKyj.js';
import 'fs';

Which then imports public/api/fragment.BGdBeKyj.js

// so far so good
// ... bunch of code

async function handler(request) {
  const params = new URLSearchParams(request.url.slice(request.url.indexOf("?")));
  const offset = params.has("offset") ? parseInt(params.get("offset"), 10) : null;
  const headers = new Headers({ "Content-Type": "text/html" });
  const artists = getArtists();
  const { html } = await renderFromHTML(
    `
    ${artists
      .map((item, idx) => {
        const { name, imageUrl } = item;

        return `
          <app-card
            title="${offset + idx + 1}) ${name}"
            thumbnail="${imageUrl}"
          ></app-card>
        `;
      })
      .join("")}
  `,
    [new URL(new URL('fragment.7h871Zjw.js', import.meta.url).href, import.meta.url)],
  );

  return new Response(html, { headers });
}

And so that file imports fragment.7h871Zjw.js...

// what the heck!?
export { C as default } from './fragment.BGdBeKyj.js';
import 'fs';

And that last file just imports the previous file? Maybe related to this open issue - rollup/rollup#4166? 😕

Should maybe try and make a minimal repro to see if new URL counts as a dynamic import?

[thescientist13]

Not sure if this one is relevant too - rollup/rollup#5601 ?

[thescientist13]

Also, I think there's a slight dis-connect in that for API routes we specifically have to check for additional chunk assets
https://github.com/ProjectEvergreen/greenwood/blob/v0.32.0-alpha.7/packages/plugin-adapter-aws/src/index.js#L97

for (const [key, value] of apiRoutes.entries()) {
  const outputType = "api";
  const { id, outputHref } = apiRoutes.get(key);
  const outputRoot = new URL(`.${basePath}/api/${id}/`, adapterOutputUrl);
  const { assets = [] } = value;

  // ...

  for (const asset of assets) {
    const name = path.basename(asset);

    await fs.cp(new URL(asset), new URL(`./${name}`, outputRoot), { recursive: true });
  }
}

But with SSR pages seems like we can just pattern match

for (const page of ssrPages) {
  const outputType = "page";
  const { id, outputHref } = page;
  const outputRoot = new URL(`./routes/${basePath}/${id}/`, adapterOutputUrl);
  const chunks = (await fs.readdir(outputDir)).filter(
    (file) => file.startsWith(`${id}.route.chunk`) && file.endsWith(".js"),
  );

  // ...

  // and any (URL) chunks for the page
  for (const chunk of chunks) {
    await fs.cp(new URL(`./${chunk}`, outputDir), new URL(`./${chunk}`, outputRoot), {
      recursive: true,
    });
  }
}

So maybe that could be cleaned up a bit too?