Refactor to use svelte.compile and svelte.preprocess

[matthew-ia]

Todo:

• preprocess and compile as a part of the prerender pages
• preprocess and compile Cayo Components and save them to a file for runtime
• pass preprocess settings from config to any instance of svelte.preprocess
• Optimize watching pages and components, and re-prerender pages when their components get updated
• Catch svelte.compile errors and warnings and report them without exiting

Related Issues:

• Changing svelte preprocess options don't do anything #42
• Can't import JS files with normal ESM syntax inside components #46
• cli exits if a svelte compiler error occurs #28
• Restructure files #49

[matthew-ia]

Something big to consider is that the .cayo/pages.js file was also intended to help be able to force a cache refresh on all the page modules, so they weren't stale between building the prerendered pages.

EDIT: actually, I don't think this will be an issue, because the modules won't actually be imported, but re-compiled each time using their file contents as the source. As long as the source is updated at least as often as the respective file changes, then should be fine.

[matthew-ia]

In order to get svelte-process goodness for all components and pages, every component, whether it's a dependency of a page or just another component, will need to be compiled the new way.

That means I'll need to maintain some sort of dependency tree, just for knowing what does/doesn't need to be compiled when the watcher detects a change in a particular Svelte component. Something like:

// I'm thinking the key is the dependee, and its value is an array of its dependers
const dependees = {
  '/path/to/component1.svelte': ['/path/to/component2.svelte', '/path/to/page1.svelte'],
  '/path/to/component2.svelte': ['/path/to/component3.svelte', '/path/to/page2.svelte'],
}

Just regarding the inner component dependencies: In this case, 2 has 1 in it. 3 has 2 in it. So technically, 3 has two dependees: 2 and 1. Need to figure out how to solve for that too, "nested" dependencies n levels deep.

The upside down tree, I was thinking would be useful because you could see: ok this component was modified, dependees[watchedPath] would get you the list of things you now have to recompile because of that change. But, I feel like the nested deps thing is where that gets tricky, because it's easier to derive the nested ones from a top-down tree, I think (key being the depender, dependee as array elements).

---

Also need a flat list of all things that were compiled per refresh, so that unchanged dependencies of multiple Svelte components don't have to be recompiled if they were already compiled due to another dependency relationship.

[matthew-ia]

Would it make sense to put the logic of resolveImports into a custom preprocessor, and pass that as a function to svelte.preprocess? I think it'd be cleaner, and aligns with the "svelte" way (it's doing exactly what a custom preprocessor would, but after compilation, even though it doesn't need to happen afterward).

Edit: Yes. Make a custom preprocessor with a similar signature to svelte-preprocess, and have it provide a markup function for now:

// preprocess.js

export default function cayo() {
  const markup = (...) => {
    // ... resolveImports logic
  }
  return {
    markup,
  }
}

markup should look like:

markup?: (input: { content: string, filename: string }) => Promise<{
  code: string,
  dependencies?: Array<string>
}>

[matthew-ia]

New pages.js and components.js will effectively replace the functionality of most of prerender.

CLI should import compilePages and compileComponents from the respective new files, and replace the existing prerender calls.

Still need to figure out how to flow the compiled code through the other things prerender did. Right now the two new compile functions also write the resulting code to a file. Basically, the "post-process" stuff from prerender that happened right after the old render. (The new compile functions can be considered to have replaced the old calls to render within prerender.) Can probably optimize the "post-process" stuff that needs to happen for pages happen for them but not components, and/or possibly create preprocessors for the functionality instead of doing it after. That way all that happens within the compile process, and then the output is good to immediately use/render client side.

Still to do:

• "post-process" stuff
• building the new process flow into the CLI
• consider breaking this process up until some separate files so CLI is really just the
• rename "template" to "layout" in all instances
• getting to watch to work sensibly with dependencies
• add cayo components to the dependency tree?

---

Flow Chart for the new process

[matthew-ia]

The stale module issue is still around even after a good majority of the rewrite.

page modules recompile/render their local contents fine, but their external dependencies, even though the deps themselves are updating, the instance is getting stale versions of the modules.

E.g., Page uses component A. Module A gets updated when user changes the source file (A.svelte). But Page does not see the changes even tho it also re-renders after it is recompiled and reloaded via dynamic import.

I think the fix may be to move the external dependencies into the page module, so there are no external deps. This wouldn't be too hard I don't think, since I have each file and could just copy it directly into the file but wrapped in a function to call instead of the imported object. Pretty confident that'd work.

Think of it like each page is a bundle.

[matthew-ia]

why can't we just use rollup at these steps instead of svelte.preprocess & svelte.compile??? I think I shied away from it just bc the rollup setup I initially had didn't work well.

Preprocess > compile was the main concern, but that' what rollup can do with the right plugins, it just also bundles.

[matthew-ia]

Biggest problem with using just rollup is figuring out how to reorgnize the processing that was done on the source code before/after compilation in the original flow.

I think svelte.preprocess with custom preprocessors can help with these:

• replacing relative paths with absolute paths (we need this since the bundle output is in a diff directory than the source, src/ vs .cayo/)
• changing cayo ids in the markup from the path strings to their UUID names. E.g., <div data-cayo-id="./some/path/a.cayo.svelte"> to <div data-cayo-id="some__path__a-{hash}">

[matthew-ia]

Not sure when/where this issue is from, but right now the only pages that get compiled are the ones you save after running the command. It should compile all of them when starting.

• running cayo dev should compile all pages as a part of startup

[matthew-ia]

Since we don't need to compile normal components, change the references for generic components to cayos or cayoComponents (e.g. compile.components -> compile.cayos

• Make component-bundling specific to Cayo Components aka cayos

[matthew-ia]

7339477 adds component dependency watching, which triggers a rerender of dependent pages. This as the last big piece of the puzzle.