Ractive template build watcher

[evs-chris]

Here's the process that I use to build my views incrementally in my larger projects:

1. I have a script that initializes the output structure of my client code.
2. The script then starts a custom node script (see below) that watches my views directory and spits out ts modules in a gitignored build directory. The script backgrounds this process.
3. The script starts tsc in watch mode on the client source and the view output directory, which is separate. This is also backgrounded.
4. The script starts rollup in watch mode on the output from tsc, which will now include the compiled templates.

Here's the nodemon watcher:

const nodemon = require('nodemon');
const path = require('path');
const { build } = require('ractive/lib/component');
const fs = require('fs-extra');

const dir = 'views';
const outdir = 'src/node_modules';
const prefix = `${path.resolve('.')}/`;
let first = true;

const deps = {};

function mkReadFile(file) {
  const base = path.dirname(file);
  return function readFile(names) {
    if (Array.isArray(names)) names = path.join.apply(path, names);
    const name = path.isAbsolute(names) ? names : path.join(base, names);
    if (!~deps[file].indexOf(name)) deps[file].push(name);
    return fs.readFile(name, { encoding: 'utf8' });
  };
}

async function rebuild(f, nodep, rebuilt) {
  const file = f.replace(prefix, '');
  if (rebuilt && ~rebuilt.indexOf(file)) return;
  const target = path.join(outdir, file.replace('.ractive.html', '.ts'));
  console.log(`views: building ${file}`);
  const content = await fs.readFile(file, { encoding: 'utf8' });
  deps[file] = [];
  const output = await build(content, { autoExport: true, escapeUnicode: true }, mkReadFile(file));
  await fs.mkdirp(path.dirname(target));
  fs.writeFile(target, output, { encoding: 'utf8' });
  if (!nodep) {
    rebuilt = rebuilt || [];
    rebuilt.push(file);
    for (const k in deps) if (~deps[k].indexOf(file)) await rebuild(k, false, rebuilt);
  }
}

async function lsr(p) {
  const dir = (await fs.readdir(p)).map(f => path.join(p, f));
  let res = dir.slice();
  for (const f of dir) if ((await fs.stat(f)).isDirectory()) res = res.concat(await lsr(f));
  return res;
}

let cache = [];
let tm;

function flush() {
  const files = cache;
  cache = [];
  tm = null;
  const rebuilt = [];
  files.forEach(f => rebuild(f, false, rebuilt));
}

// change the watch array if your views aren't in the views dir
nodemon({ watch: ['views'], verbose: true, ext: 'html', exec: 'false' }).
  on('start', async () => {
    if (first) {
      first = false;
      const files = (await lsr(dir)).filter(f => /\.ractive\.html$/.test(f));
      for (let i = 0; i < files.length; i++) await rebuild(files[i], true);
      console.log('views initial build complete');
    }
  }).on('restart', files => {
    let i = files.length;
    while (i--) if (!~cache.indexOf(files[i])) cache.push(files[i]);
    if (tm) clearTimeout(tm);
    tm = setTimeout(flush, 500);
  }).on('stop', () => {
    console.log('done watching views');
    process.exit(0);
  });

process.on('SIGINT', () => {
  nodemon.emit('stop');
});

[DillonSadofsky]

That's interesting. In our project, we've been able to use rollup plugins to chain everything, and that has seemed really nice. I even wrote a couple custom rollup plugins to do various generated code translations 'before' typescript (which we use the rollup typescript plugin to do).

I suppose we could un-rollup the ractive template build step, but its so close to being able to watch our source code and generate the bundled output, cradle to grave. The only thing that seems to not be working is that the ractive template rollup plugin writes output files successfully on the input files changing, but even though it goes 'before' typescript in the plugin order, the updated files are not present by the time typescript plugin runs.

My guess is that you generate and write the output view files to disk asynchronously and therefore they are not available to the next plugin by the time its invoked in the chain. Would it be possible to change that? Its nice that there is this plugin to rollup that can piggy back on the file watching and generate output, but if its not guaranteed as input to the next step in the rollup chain, it doesn't work quite right.

I think when I was reviewing the typescript rollup plugin some plugins will generate 'virtual' output that can be passed on to the next plugin (essentially an in-memory file), so at least if that could be returned to the rollup handler when it goes to 'look' for the generated file that would fix it, even if the files won't be flushed to disk until later.

[DillonSadofsky]

Yuck. I think I am right. I added a sleep to rollup-plugin-ractive-bin\index.js right after the file write and it worked, though it slowed down the build (obviously):

I'm not sure of a good way to work around this race condition between files being output and read in by typescript. I think I've been able to avoid it because my rollup plugins are stupid and just do everything in buildStart which must be separated enough timing-wise. I guess I don't know if there is a good way around this. Even the 'rollup twice' solution might not work if it happens fast enough because of the cache delay. I know this is now only tertiarily your project's problem, but curious if you have any suggestions.

[evs-chris]

Ahhh, yes, now I remember. I couldn't get the typescript or typescript2 plugins to compile stuff and check types still and not have to be goaded to compile two or three times per save. So I switched my setup around to use a separate tsc process, which removes a bit of the faffery with rollup and caching. It's not ideal, but it's a lot smoother.

We could add a plugin option to chill for xms after running. That's effectively what you've done, but I imagine that the ideal number may change from project to project. Since the plugin ostensibly supports ts by allowing you to change the extension, it would make sense to try to work a little better with the typescript plugins.

[DillonSadofsky]

Well, I wrote this up for them, since I guess its really their bug: rollup/plugins#1122
IDK if they'll just close it since I didn't submit a minimal repro (due to race conditions being inherently difficult to repro). I think the issue with the delays is that your rollup plugin will build .ts files 'one at a time' as part of load being called, and each one will hit the typescript's watch code, potentially having a 250ms delay, then the typescript rollup plugin's load is called, interleaved. So since we don't do 'all ractive' then 'all typescript' we can't do just a single 250ms delay, to be safe, we have to wait 250ms after every view file is generated by the view generation code, which could really stack up on a large project.

I'm starting to think the safest solution would be to have your module generate .js code instead of .ts code so that typescript will not look for it, and it will be on disk for rollup to find and bundle later. Though then I can't 'import' the output of the ractive transpile. IDK, I'd have to experiment with that, and it feels like just working around the issue instead of solving it.

For now, I think the 'build twice' thing will work since rollup is slow and therefore always going to be slower than the 250ms delay, but if they ever sped it up, build twice might not work right.

[evs-chris]

Do you think trying to dump all of the output to disk at once would be a viable workaround? It would probably be possibly to have a global debounce that gates everything that happens within a certain window such that they return as a sort of batch.

[DillonSadofsky]

I do think that would technically resolve the issue, but it means making a 'dumber' rollup plugin. The rollup plugins I made are super dumb and just do their work as part of the buildStart callback in rollup, but the downside is that you have to explore the source tree with globs to decide what files to 'do' vs handling them just in time as part of resolveId or load. The downside of this 'dumb' way is you have to process all /**/*.ractive.html or whatever. The nice thing about the way you're doing it is that rollup explores the source graph and only asks you to process files as they are 'hit'.

So I think that would work, but it feels kinda sucky to have to make that transition.