RFC: Migrate to TypeScript

[dstaley]

Summary

Migrate kinto.js and kinto-http.js to TypeScript, replacing browserify with either Webpack or Rollup. Publish three versions of each library: an ES-module version written in a recent Ecmascript version, a CommonJS version written in ES5, and a bundled version written in ES5 for browsers.

Motivation

Over the past few years, many JavaScript developers have switched to using TypeScript in their projects. They're drawn to the many benefits that TypeScript's type system brings to their codebase, primarily stronger type checking. TypeScript is now the seventh most popular language on GitHub, the third fastest growing language, and four of the top ten projects on GitHub are written in TypeScript.

As more developers migrate to TypeScript, even more libraries are working on their TypeScript integrations. Many libraries not written in TypeScript provide definition files that describe the external API so that developers using TypeScript (and tools that understand TypeScript definition files) can have type checking for their use of the library. This only improves the experience of third-party developers integrating the library into their own code. By migrating a library to TypeScript, both library developers and library consumers will benefit from the additional safety that TypeScript provides.

TypeScript isn't just types over JavaScript; it's also a transpilation system similar to Babel. Developers can write TypeScript using the latest JavaScript features, and the TypeScript compiler will handle converting those features down to the developer's chosen target. Kinto can simplify its build system by replacing Browserify with TypeScript and a bundler like Webpack or Rollup. This will allow Kinto to publish versions of the package using ES modules, which provides better performance due to tree-shaking and other optimizations that bundlers can provide.

Guide-level explanation

Transition

At a high level, migrating to TypeScript should be a relatively smooth, stress-free experience because of the fact that TypeScript allows for an incremental transition. The transition can happen in three distinct phases. We should start with kinto-http.js, complete the transition there, and then move to kinto.js.

1. Switch from Browserify to TypeScript + Webpack/Rollup

The first step in the transition to TypeScript would be to replace the current Browserify-based build system with the TypeScript compiler along with a bundler like Webpack or Rollup. In this step, no additional type checking is added to the codebase.

2. Incrementally transition to TypeScript

After implementing the TypeScript compiler, files can be converted from JavaScript to TypeScript on a file-by-file basis, in smaller pull requests to make things easier. It would be best to work from the outer edges of the module graph inward, so that when we get to the larger files we have enough typings on external modules. (More details on this in the reference-level explanation section.)

3. Publish typings

Once the transition is complete, we can then emit type declaration files when publishing the package to NPM. This will allow library consumers to type check their own code when using Kinto.

Build Process Changes

TypeScript can replace Browserify for the NPM package, but for the browser bundle we'd need to use a tool like Webpack or Rollup to produce the final bundle. With TypeScript we can create the following versions of the libraries:

1. ES module version, built by TypeScript, written in a recent version of Ecmascript (ES2018). Used in tools that understand ES modules like Webpack, Rollup, Parcel, etc.
2. CommonJS module version, built by TypeScript, written in ES5. Used in Node.
3. UMD module version, built by TypeScript, bundled by Webpack/Rollup. Used in browsers by manually including a <script> tag.

This selection of outputs covers all the major use cases, and ensures that developers using a modern build system can benefit from the tree-shaking that ES modules provides. It also allows developers to ship modern JavaScript should they choose to do so.

Reference-level explanation

Transition

To start, we'd begin the transition with the kinto-http.js library since it's a dependency of kinto.js (and because it's a smaller library, perfect for nailing down all the specifics).

1. Switch from Browserify to TypeScript + Webpack/Rollup

We'd begin by replacing Browserify with the TypeScript compiler, configured to allow JavaScript files. We'd also add a bundler like Webpack or Rollup to support the creation of a UMD module suitable for use in a browser. TypeScript should be configured to output to ES2017 with ES modules and ES5 with CommonJS modules. The bundler would then consume the CommonJS version and output an ES5 UMD module.

Switching would consist of creating a TypeScript config file along with a bundler config file, modifying the scripts in package.json to call the TypeScript compiler and bundler, and editing the main property of package.json to point to the CommonJS output. We'd also add a new module property pointing to the ES modules output.

The only major code changes necessary in this step is removing the Babel polyfill along with the Browserify "hack" at the bottom of index.js. The Babel polyfill is no longer needed in the output as TypeScript will correctly polyfill when converting to ES5. The ES modules version should not have any polyfills.

After this step is completed, a new version of the package can be published. At this step, the primary benefits would be a more modern build system, he ability for library consumers to employ tree-shaking, and a reduced bundle size for library consumers due to the removal of the Babel polyfill.

2. Incrementally transition to TypeScript

Once the TypeScript compiler and bundler are implemented, work can begin on adding types to the codebase. Ideally, we'd work from the outer edges of the module inward. Thankfully both kinto.js and kinto-http.js only rely on the uuid module as an external dependency, which already has typings published.

For kinto-http.js, this would be the suggested order of transition:

1. batch.js
2. errors.js
3. endpoint.js
4. utils.js
5. http.js
6. requests.js
7. collection.js
8. bucket.js
9. base.js
10. index.js

For kinto.js, this would be the suggested order of transition:

1. utils.js
2. adapters/base.js
3. adapters/IDB.js
4. collection.js
5. KintoBase.js
6. index.js

Each file can be transitioned individually in its own pull request so that it makes reviewing easier. As we progress upward through the library, it should require less and less typing as TypeScript will be able to automatically infer more and more types.

3. Publish typings

Once all files are written in TypeScript, we can enable declaration file emission in the TypeScript configuration. Once published, any user using TypeScript or a TypeScript-enabled tool will then automatically get type-checking in their codebase.

Build Process Changes

As part of the first transition step, we'd replace Browserify with TypeScript and a bundler like Webpack or Rollup (which one is largely inconsequential as we'd only be using them for bundling ES modules into a UMD bundle; in this sense they're quite similar). We'd also remove Babel from the build process and let TypeScript handle the step down to ES5. (It's likely that we'll still need to retain Babel for testing, but there are replacements if we'd like to completely remove it.)

To support the most popular use cases, we'd publish three versions of the library (contained within one NPM package): an ES modules version written in a modern version of Ecmascript and referenced by the module property in package.json, a CommonJS version written in ES5 and referenced by the main properly in package.json, and a UMD module contained within the dist folder for use in the browser without a bundler. The primary benefits of this structure are allowing users to ship modern code if they want to (as opposed to forcing ES5), along with supporting tree-shaking.

Drawbacks

One of the biggest drawbacks to migrating to TypeScript is increasing the barrier to entry for developers who aren't familiar with TypeScript. However, this is offset by adding stronger type checking to library internals, which actually makes it easier for developers to contribute high-quality code.

Rationale and alternatives

TypeScript isn't the only typed JavaScript alternative. Flow offers a similar approach to TypeScript. However, community support for Flow isn't as large as it is for TypeScript. Flow also lacks large, public, open-source projects that implement it. Recently, Jest switched from Flow to TypeScript.

Prior art

The following are projects that transitioned to TypeScript:

• Jest
• Expo
• Vue
• Angular
• Reddit
• Unsplash
• Hootsuite
• Affinity

Unresolved questions

• What are the requirements for the Firefox-specific build of Kinto? Can we reasonably bring this version in line with the rest of the library?

Future possibilities

Switching to TypeScript and reworking the build process could be a good opportunity to rethink the testing process. Currently, kinto.js and kinto-http.js have over 30 dependencies that are related to testing. By switching to a new testing tool, particularly one that already understands TypeScript, we might be able to reduce the number of dependencies (and thus the number of pull requests that need review).

Currently functions are documented using JSDoc. With the transition to TypeScript, we might be able to also transition to TSDoc.

Frequently Asked Questions

If Kinto is written in TypeScript, would it be possible to use it without TypeScript?

Yes! Even if the project is written in TypeScript, all consumed versions of the library will be written in JavaScript. If you use a TypeScript-aware tool like VS Code, you'll even get type checking for your JavaScript code when interacting with Kinto.

Can we just add typings instead of moving the entire project to TypeScript?

Adding type declarations is a great way to provide some additional safety for library consumers. However, this wouldn't benefit contributors to Kinto, and it would require someone to manually keep the typings up-to-date with Kinto. By writing the project in TypeScript, we can be sure that the typings will always be up-to-date.

What if we decide to switch back to JavaScript?

Reverting to JavaScript is incredibly easy. We can simply replace the TypeScript source files with the TypeScript compiled versions. This will remove the typings, leaving a plain JavaScript file. Also, since we'd start with kinto-http.js, we'd be able to detect issues early on before migrating the primary library.

[leplatrem]

Hi Dylan,

Thank you for this thorough analysis! We don't really have the bandwidth to carry out the transformation ourselves, but we would not object to PRs that did. We are enthusiastic about typed languages, but we haven't yet tried TypeScript. We acknowledge the risk of increasing the barrier to contribution for JavaScript developers, but we think it's a worthwhile tradeoff.

The requirements we can think of are:

• Gecko output -- we still have to produce a blob of JS which is vendored into mozilla-central. The build process is mostly contained to this repository. Here are some additional notes:
  • The output is ES6 without common-js imports (ie. transpiled to require()): https://searchfox.org/mozilla-central/rev/928742d3ea30e0eb4a8622d260041564d81a8468/services/common/kinto-offline-client.js
  • We browserify this entrypoint https://github.com/Kinto/kinto.js/blob/master/fx-src/index.js with --ignore uuid --ignore kinto-http --ignore isomorphic-fetch.
  • We add a custom header using cat :)

    kinto.js/package.json

    Line 17 in ad789ca

    "dist-fx": "BABEL_ENV=firefox browserify -s Kinto --bare --ignore uuid --ignore kinto-http --ignore isomorphic-fetch -e fx-src/index.js -o temp.jsm -t [ babelify . ] && mkdir -p dist && cp fx-src/jsm_prefix.js dist/moz-kinto-offline-client.js && echo \"\n/*\n * Version $npm_package_version - $(git rev-parse --short HEAD)\n */\n\" >> dist/moz-kinto-offline-client.js && cat temp.jsm >> dist/moz-kinto-offline-client.js && rm temp.jsm && echo >> dist/moz-kinto-offline-client.js",

• Tests should be run as easily as they are now (using something like npm test)
• Source maps are usually useful

Are you sure it'd be easier to migrate kinto-http.js first?

Would you be willing to start this longhaul task yourself?

[dstaley]

• Tests should be run as easily as they are now (using something like npm test)

Definitely agree! We can technically leave the entire testing infrastructure alone and just change the main import from src/index.js to dist/es/index.js and it should just work. However, this would require building before each test, which isn't really ideal. I'll explore various solutions, but in the end it should be just as easy to run the tests.

• Source maps are usually useful

I'll make sure that we have sourcemaps for all the various outputs!

Are you sure it'd be easier to migrate kinto-http.js first?

My thinking here is that since kinto.js relies on kinto-http.js, it'd be wise to tackle that one first.

Would you be willing to start this longhaul task yourself?

Definitely! The main place I'll need assistance in will be reviewing the pull requests once I start adding types to the source files. Thankfully everything is really well documented with JSDoc, so those will be incredibly helpful. Since those PRs will be small I imagine it won't take too much time to review them.

I'm going to go ahead and start step one with kinto-http.js. The first PR will be the one that switches to TypeScript for building the library, but doesn't begin to add types.

[dstaley]

This landed in #1061 🎉