BUILD: revamp frontend building

[stephanwlee]

Current state

Most of our build is done in our Vulcanization process today.
Before bringing up what a Vulcanization entails, it is important to note that dependencies of a module is defined in HTML via script and import tags. Vulcanization talks the dependency graph and:

• first builds TypeScript files (along with necessary d.ts files) that outputs .js and a .d.ts file
• compile all JSs (generated included) using JSCompiler into one bundle
• depending on options or "environment", it inlines the bundle into the script tag
  In all of these processes, it makes sure a dependency is properly provided via Bazel deps.

One interesting artifact of Vulcanization is the concept of webPath. Instead of using relative or absolute path to a module, modules, in its tf_web_library, define a webPath which indicates a path of the module in the bundle artifact. For instance, say there is a module in a/b/c/d/foo.js which defines webPath of /tf-foo/foo. When other modules use the foo, it will define dependency as <import href="/tf-foo/foo" /> (or a relative path from its webPath). This loosely is related to concept of webfiles of Bazel/rules_closure, though, stephanwlee lacks profound knowledge in it yet.

The design choice of webPath and Vulcanization impact how we author JS/TS. The natural way of defining dependency in JavaScript is via CommonJS or ES module (modern browsers have native ES module support) but we have to use namespace and window object (e.g., window.tf.graph.common) because we define dependency via HTML. This is not to say concept of Vulcanization is incompatible with es module completely but it requires non-trivial changes.

Future state

In the future, we want our solution to:

• use Bazel: we don't want part of our app to be on Bazel and others on another build system
• have ability to easily test FE (karma integration at the very least is required)
• have less customization of build system if possible (i.e., deprecate Vulcanization.java)
• other framework (e.g., React) friendly

With above requirements in mind, this section only is attempting capture one point in the vast solution space.

1. Use Bazel/rules_typescript to build TypeScript
   This is the canonical way of building TypeScript with Bazel.

2. Utilize Bazel/rules_nodejs as much as possible
   rules_nodejs allows us to pull dependencies from NPM and use node ecosystem to build and test.

3. Move off of JSCompiler and use Webpack, Rollup, or None
   JSCompiler is quite sophisticated but it is a bit overkill for our application today. Shaving extra bytes off of JS payload is not worth the complexity and additional maintenance burden. Good alternatives to JSC are Webpack and Rollup.

We can also consider not bundling and loading compiled JavaScript using ES module. With HTTP/2 more mainstream and because our code is not written for Node.js, there is no reason to (1) build JS to shim node module system & Node APIs and (2) create one binary for performance optimization. This at least generally is very good approach for development. One drawback of this is that there are interesting third-party hosted TensorBoard and it may negatively impact them performance wise.

[wchargin]

Thanks for writing this up!

I agree with all your analysis in the “Current state” section, and have
a few clarifying questions about the “Future state” section:

• Why is Karma specifically a requirement? I have no strong opinions
  as to what test runners is best for us, but it’s not clear to me why
  Karma is a must-have.

• I’ve never worked with native ES modules in development before,
  having always used sourcemapped Webpack or something philosophically
  similar. What makes you say that it’s a “very good approach for
  development”? In particular: we’d still expect to have compilation
  steps as opposed to running with literal raw sources, right?
  (TypeScript compilation at the least, and maybe JSX if we go the
  React route.)

• In the Bazel/rules_nodejs line item, you link to dropbox/rules_node.
  Is there some integration between the two that you think will be
  helpful? rules_node doesn’t seem to depend on rules_nodejs. (Also,
  just in case you didn’t see, Bazel’s rules_typescript appears to
  have been absorbed into rules_nodejs as well.)

[stephanwlee]

Sorry for leaving you hanging for awhile. Now that we are getting closer to actually changing the build system, I think I owe you the answers.

1. Karma is not required and what I really meant is following. In my past experiences, I had better time with Jest or other test runners running in node than on browser. In my ideal world, I would want some macro that wraps Jest, but if, all fails, I would at least want a Karma to be available. I would not want to use WebDriver/Selenium to run all (even unit) tests. Realistically, because we will be using rules_nodejs, we have an option to use jasmine_node_test or karma_web_test (ts_web_test wraps karma_web_test).

2. I wrote this near the time when I dabbled with JavaScript without build system and I drank a bit too much koolade. I think this item mainly comes from my ideal where incremental build does not scale with number of dependencies from the entry point. For our case, since we would like to use TypeScript, if we use the TSC to transpile individual file (sure, the type-checking will actually walk the dependency graph but I assume TSC watch mode to be smart to cache the type information and incremental type-checking to be fast) and use ES module, the ideal can be mostly achieved.

3. updated the link to rules_nodejs. It seems quite solid.

[stephanwlee]

How

Managing a dependency

# Add -d if it is for development only. Note that the distinction between development vs. production is arbitrary for TensorBoard.
yarn add <name> [-d]   

# to remove
yarn remove <name>

Depending on the dependency

# Can be any macro/rule.
ng_module(
    name = "foo",
    srcs = [ … ],
    deps = [
         "@npm//<name>",
    ],
)

Using Dependency

import * as foo from "<name>";

Bazel Gotcha

rollup_bundle does not seem understand transitive dependency and cannot resolve the dependency

If the dependency is new and is part of the transitive dependency of the binary, you must add a line to the rollup_bundle.

rollup_bundle(
    name = "binary",
    entry_point = ":meow",
    deps = [
         ...
         "@npm//<name>",
         ...
    ],
)

What was it before and Why do we need this change?

As of July 2019, TensorBoard uses own build system that share some similarity with rules_closure (the macro was largely changed to accommodate building TypeScript alongside traditional JavaScript and HTML). In this world, all dependencies were maintained by Bazel through declarations in WORKSPACE and *.bzl files (e.g., js.bzl). However, maintaining the custom build system requires engineer-hour and, as indicated here in the public issue, we are thinking of using better Bazel team supported rule like rules_nodejs.

In the canonical macros provided by rules_nodejs, the dependencies are still maintained by Bazel but is defined in the package.json, a natural place for projects using node-based infrastructures.

TODO

• figure out why we need to define transitive deps in rollup_bundle.