Code generation extensibility point

[weswigham]

@alexeagle from @angular has asked for a code generation extensibility point as part of #9038.

A code generation extension should be able to:

• Access the program after (or during) parsing, but before typechecking
• Inject new (already parsed) source files (not necessarily backed by files in the host) into the compilation

Proposed API shape:

interface CodegenProviderStatic {
  readonly ["extension-kind"]: "codegen";
  new (context: {
    ts: typeof ts,
    args: any,
    getCommonSourceDirectory: () => string,
    getCurrentDirectory: () => string,
    getCompilerOptions: () => CompilerOptions, // NOTE: These are unverified at this stage
    /**
     * addSourceFile causes processRootFile to be called on the provided SourceFile
     * and the file to be added to the filesByName map inside the Program
     */
    addSourceFile: (file: SourceFile) => void
  }): CodegenProvider
}

interface CodegenProvider {
  /**
   * Called each time a file is added to the filesByName set in a program
   *  - should trigger when a generated file is added, this way your
   *    generated code can cause code to generate
   */
  sourceFileFound?(file: SourceFile): void;
  /**
   * Called when all processing is complete and the program is about to be returned,
   * giving you the opportunity to finalize your emitted set of generated files
   */
  processingComplete?(program: Program): void; 
}

My immediate thought after taking a cursory glance as implementing this:

• Do injected files need to affect module resolution? (Are they actually discoverable by other files' imports, or are they just compiled as part of the project? Discoverability could be difficult as module resolution hosts aren't expected to layer on pre-resolved paths or virtual filesystems - maybe this is best dealt with by coordination with a provided host?)
• A program cannot be provided in the constructor, as it has to be constructed before we start traversing files - so before a program is available. I've tried to make the things from the program you'd be likely to want available, so hopefully that's good enough.

[alexeagle]

Thanks for getting this started, Wes.

Which API call actually causes the code generator to run? Would sourceFileFound be called at the beginning, for the original set of sources loaded into the program?

Do injected files need to affect module resolution?

Yes, this is one of our requirements - the user's entry point TypeScript source should import from a generated file, so that the resulting module dependency graph can be completely understood by a tree-shaking tool like Rollup/WebPack2. We currently do this by creating a new program after codegen and before type-check/emit. Here: https://github.com/angular/angular/blob/db54a84d1418950a9c1ad07f2553179cbf231135/tools/%40angular/tsc-wrapped/src/main.ts#L36
That seems easiest and doesn't involve the host. We write files to the underlying filesystem so they don't require any sort of shared VFS layer.

A program cannot be provided in the constructor

We had this issue as well. The only thing we need from the program is getSourceFiles() (and getSourceFile(path: string) is also useful). Usage is here: https://github.com/angular/angular/blob/db54a84d1418950a9c1ad07f2553179cbf231135/modules/%40angular/compiler-cli/src/codegen.ts#L86
Of course we need the program to be parsed so that sources not listed in program.getRootFiles() but transitively imported by one of those will be available to us.

In your design, I see that the codegen gets another chance to run after new files get generated. We don't need this, and also this introduces the need to continue until there are no newly generated files, which seems prone to causing cycles in naive implementations.

This may go without saying (and not require any provision in the design): it will be useful to generate non-TS files, for example the .metadata.json files we produce now to propagate decorator metadata, or .xmb files for i18n translations (cc @vicb), etc

[weswigham]

@alexeagle After looking over your code, I believe I can see that you shouldn't need either. You use getSourceFiles just to enumerate what-you-need-to-compile (which sourceFileFound should handle that enumeration for you). You use getSourceFile just to look up the source file associated with the metadata you read, which you then just pass into writeFile (which then, after doing nothing and getting threaded all the way through the TS compiler for no real reason, finds its way back to your MetadataWriterHost, where it finally gets its metadata collected and written to disk. Why is this not just inline in that compile function? Why do you need a MetadataWriterHost?).

You should just be able to write:

class CodegenProvider {
  // Most of CodeGenerator's utility functions here
  sourceFileFound(file: SourceFile) {
    if (GENERATED_FILES.test(file.fileName)) return;
    const metadata = this.readFileMetadata(file.fileName);
    const generatedModules = this.compiler.compileSync(metadata.fileUrl, this.compiler.analyzeModule(metadata.appModules), metadata.components, metadata.appModules);
    generatedModules.forEach(module => {
      const fullText = PREAMBLE + module.source;
      const newFile = ts.createSourceFile(module.moduleUrl, fullText, ScriptTarget.ES6, /*setParentNodes*/true, ScriptKind.ES6);
      this.host.writeFile(this.calculateEmitPath(module.moduleUrl), fullText, /*writeBOM*/false, () => {}, [newFile]);
      this.addSourceFile(newFile);
    });
  }
}

Also, quite blatantly, there is no way I can make this API work for for you if you cannot do template compilation synchronously (note how I wrote this.compiler.compileSync), as the creation of a program is a synchronous operation. (From a quick glance, however, not much in your compile method is actually asynchronous other than normalizeDirective though - which has a possibly synchronous result (and you could totally use the synchronous XHR to allow it to work synchronously in all cases if need be... wait a second, that entire async-only branch depends on something that's not even implemented)... so there is nothing asynchronous going on yet...?)

[alexeagle]

That API looks sufficient to me. We can do some experimentation to verify when the extension point is prototyped.

We could walk through those APIs sometime - the short version is that they're structured to allow the compiler to be run in a few different environments, to separate codegen from the tsc-wrapped (avoiding the cyclical dependency of angular build depending on angular) and also to borrow TS logic for module->filepath, how compilerOptions affect which files to write where and when, etc.

re. async - I believe these are all consequences of having an online compilation mode, where eg. template files may need to be loaded from the server with an XHR. We have a synchronous XHR in https://github.com/angular/angular/blob/master/modules/@angular/compiler-cli/src/codegen.ts#L124 - @tbosch is the expert on most of the APIs inside the compiler, but is on vacation. I suspect the tsc-based compiler can be entirely synchronous. Worst case we could just count down as files are written and block the caller until the count reaches zero.

[mhegazy]

This is should be covered by #13940