Build system

[kristojorg]

Hi all,

As discussed in the weekly meeting yesterday, here is my proposal for a build system for Readium web.

Our needs

1. Transpile Typescript into JS with properly exported types
2. Bundle into various module formats for consumers to import: CommonJS, ESM, and UMD
3. Watch Mode for development
4. Monorepo support
5. Testing setup for unit and possibly integration tests
6. Treeshaking for size optimization
7. Code quality choices and enforcement
8. Examples should live in this repo and be easy to start working with
9. CI/CD - code should be linted, built and tested on every PR. Example would ideally be deployed somewhere so newcomers can easily demo.

The recommendation

As you probably know, the world of web build systems can be quite frustrating. There is seemingly endless configuration, difficult debugging, and a rapidly changing ecosystem of tools and best practices. It's prettymuch the embodiment of why web dev is annoying and difficult. Over the past 5 years, I've been reaching more and more towards so-called "zero-config" tools, which basically set up and maintain a ready-to-use build and development environment for you with best practices. Usually they can be configured, but don't need to be. For TS libraries I recommend TSDX. It's extremely simple, has all the features we need, and is officially recommended by Typescript themselves.

TSDX

You can read more about it on the website, but below is an overview of what it uses internally. The nice part is you don't really need to know most of this because it is already configured and for our use case there would be little need to change anything:

• Rollup for transpilation and bundling. This has become the defacto bundler for libraries, while Webpack remains the main bundler for full apps.
• Jest for testing. Another defacto test runner and assertion library. Generally has replaced the Mocha/Chai/Sinon combination with a single package.
• Prettier for formatting. I have found it to be extremely useful. I never think about formatting while writing code anymore (a surprising amount of effort used to go in to making lines the right length and brackets line up et). It also importantly guarantees that code is formatted the same when it reaches Github, which means you won't ever see formatting changes in a git diff, only real code changes.
• Eslint for linting with a very good set of defaults that can be modified if desired. You don't get formatting based lint errors because that's all handled automatically by Prettier, only actual code-quality stuff.
• Parcel for the example playground. Parcel is a simple bundler that is great for a little development playground.
• Github Workflows for CI/CD.

I don't think we would need any configuration changes, besides maybe massaging some formatting or lint settings if desired. I have used TSDX many times and have been very pleased with the results, and it is also in use by many other projects and companies large and small. Actively maintained, official Typescript recommendation, etc.

Other options

I haven't tried many other zero-config setups (not sure there are many others for this use case?), but we could always set up our own custom build system if that is desired. Some tooling options:

• Transpiling: Typescript's compiler or Babel
• Bundling: Webpack or Rollup, we would want to configure this with tree shaking, watch mode, and bundling to multiple module formats.
• Testing: Jest, Mocha, Jasmine, and if it doesn't have an assertion library built in then probably Sinon or another
• Formatting: Prettier, basic eslint formatting rules, or no formatter
• Code Quality: Eslint (Tslint is deprecated in favor of eslint) or no linting
• Examples: Webpack or parcel or just plain HTML if we wanted to.
• CI/CD: Github workflows, CircleCI, Travis, or something else.

Let me know what you all think. I'm happy to make a PR adding TSDX if you'd like. Would love to get this set up soon so that I can start using the shared modules!

[mickael-menu]

Prettier for formatting. I have found it to be extremely useful. I never think about formatting while writing code anymore (a surprising amount of effort used to go in to making lines the right length and brackets line up et). It also importantly guarantees that code is formatted the same when it reaches Github, which means you won't ever see formatting changes in a git diff, only real code changes.

👍 The anti-bikeshedding auto formatting of Go really grew on me, and that's something I'd like to add on mobile repos as well.

For the rest I'll let people more versed in Web development chime in, but I'll suggest two guidelines:

• Staying with "in-house" GitHub features when they fit the bill, for simplicity (e.g. GitHub workflows and actions).
• Favoring well-established tools instead of cutting-edge, to reduce friction for new/fleeting contributors.

[kristojorg]

Yep I agree on all accounts

[JayPanoz]

We’ll take a look and discuss it ASAP @ Jellybooks so that we can provide more thorough/robust feedback – not that there is anything surprising, but we have to do our homework about TSDX.

Thanks for the proposal.

[kristojorg]

Yes good point. From what I understand, the only piece we need to test at this point is the dev setup. These are the options from the discussion above:

• Rollup bundling, babel (within rollup) compiling our packages, custom server serving HTML example files
• Rollup bundling, sucrase compiling our packages, custom server serving HTML example files
• Esbuild bundling and compiling packages, custom server serving HTML example files.
• Parcel bundling, compiling and serving the HTML example files and source code (we import source code from sibling directories directly: import Publication from "../shared-models").

For production we will use Rollup + Babel. So we can already make a PR with that setup.

One other question is if we use Yarn 1 or Yarn 2. Yarn 2 makes some pretty significant changes. It supposedly comes with a lot of benefits of course, but I haven't used it. I'm guessing Yarn 1 is what we want unless @julienetie has an opinion here?

[julienetie]

Great,
I think the first two should be pretty similar.
I'm new to Yarn 2. I've just glanced at a few articles, it seems like sticking to Yarn 1 makes sense, good call.

[julienetie]

Hey @kristojorg

So what's the plan in terms of implementing. Would it be possible to get something out by Friday?
I'm happy to help with the Rollup builds if needed.

[kristojorg]

I was planning to discuss in the readium meeting today. I'm a bit backed up with some other work so probably won't be able to get this done until early next week. If you wanted to make a PR for the rollup builds (or any piece of this) sooner, I'm happy to review it though

[julienetie]

Ditto, (this was a question asked to me). I'll make an attempt at Rollup production/ dev and I'll reach to you to make sure I'm not missing out anything.

[kristojorg]

Another option I discovered recently: tsup which is powered by esbuild. It doesn't support as many features out of the box, we would have to build our own testing, examples, code quality etc. But we could pull the defaults of those from tsdx and use the bundler from tsup if desired.

[JayPanoz]

I’ll ask Julien to take part in the discussion should he have anything to clarify, discuss or add.

@mickael-menu BTW is the test-case for Kotlin/Swift available publicly somewhere? I can recall you mentioning you would gladly share it but fails to remember if you ever had to do that “manually.”

[mickael-menu]

@mickael-menu BTW is the test-case for Kotlin/Swift available publicly somewhere? I can recall you mentioning you would gladly share it but fails to remember if you ever had to do that “manually.”

You can find the automated tests for the shared models here:

• https://github.com/readium/r2-shared-kotlin/tree/develop/r2-shared/src/test/java/org/readium/r2/shared/publication
• https://github.com/readium/r2-shared-swift/tree/develop/r2-shared-swiftTests

For the parsers here:

• https://github.com/readium/r2-streamer-kotlin/tree/develop/r2-streamer/src/test/java/org/readium/r2/streamer/parser
• https://github.com/readium/r2-streamer-swift/tree/develop/r2-streamer-swiftTests/Parser

And these are my manual regression test sheets:

• Kotlin
• Swift