Join forces with Esperanto?

[joliss]

The Esperanto module transpiler seems so far to be the module transpiler with the best spec compliance for tricky edge cases like live binding (#1117) and circular imports. We're using it in Ember CLI, and it works great in practice.

Perhaps there are, conversely, some nice things in the Babel module code that Esperanto doesn't handle correctly yet.

May I suggest that you all get on a hangout some time to explore the possibility of joining forces?

/cc @Rich-Harris, @eventualbuddha

[sebmck]

We've discussed this before and it was actually bought up in the initial esnext + 6to5 call. The only reason it hasn't really been pursued is that the codebases are far too different for any type of easy integration. Babel works with an actual AST and generates code from that while esperanto will manipulate the input source directly.

/cc @stefanpenner too

[jamiebuilds]

We've discussed this previous with @Rich-Harris @eventualbuddha @stefanpenner @sebmck and I. It was agreed at the time that Babel shouldn't get into module bundling the way esperanto does.

Edit: Damn it @sebmck

[sebmck]

There's probably some way to do it however.

[sebmck]

Actually, just remembered one reason why we weren't up for it. Esperanto doesn't support the system module format (only module format that you can accurately transpile ES6 modules to) and systemjs has a Babel option so I don't really want to have any regressions in that regard.

[Rich-Harris]

Certainly open to the possibility, though as @sebmck says the two tools have wildly different approaches. System support will happen eventually! Though I'm currently more concerned with the bundling aspect (inside a bundle, modules are accurately transpiled - it's only at the boundaries that things get tricky)

[stefanpenner]

Another thing worth considering is that the bundling phase and transpilation phase happens at different times in ember-cli. In all honestly, I would likely prefer a separate bundling tool, to do this efficiently we likely have to agree on the intermediate state or some mechanism of sharing the required meta-data.

[Rich-Harris]

@stefanpenner since both tools use acorn, it's not too hard to imagine esperanto having a mode where you provide individual modules in AST rather than string form. Though I haven't really looked into whether parsing the source code is where the bulk of esperanto's time is spent, in bundling mode

[stefanpenner]

parsing is generally fast, although no work is faster then some work.

To further clarify bundling, is something I view similarly to uglification. My app may take different approaches depending on if it is for development or production.

e.g. heavy code folding for small and near zero cost abstraction, is great for production builds, but tricky to get fast enough for development (when you consider also the source map work required to make dev reasonable).

[stefanpenner]

also for reference, ember-cli today does:

1. 1:1 transforms using babel (with modules and use-strict disabled)
2. 1:1 esperanto (absolute path: true)
3. M:1 sourceMap concat for app.js
4. M:1 sourceMap concat for vendor.js

currently between 2 and 3, we lose meta-data (import/exports) and blindly concat, but @chadhietela (mostly him now) and myself have begun work on another approach.

1. 1:1 transforms using babel (with modules and use-strict disabled)
2. 1:1 esperanto (absolute path: true) extracts import/exports graph
3. graph is produced for each tree
4. resolve used dependencies given an entry tree's module graph, and once again leave the resulting graph for subsequent tooling to use
5. defer to a user exposed packager to re-combine
   • i believe in production this may prefer more extreme code folding.

the reason for this is we have found that, selective builds are ideal but the final output might vary dramatically on a app-by-app basis. So by breaking up the phases, and exposing the packager step as public api, we believe we can further satisfy requirements while still abstraction the most complex bits.

[sebmck]

Ok so I've thought about this some more. I don't really see the ember-cli pipeline setup being an issue. You can continue to blacklist es6.modules and directly use esperanto.

One possible solution (probably the only one) is to call esperanto after the actual codegen has been done. I can easily mutate the AST to have the correct range information which means that all esperanto would need to do is expose an API that takes:

• An already parsed AST with updated range information
• A generated string
• A source map that it will "use" (this should be fairly trivial, here is the method that Babel uses to "inherit" a sourcemap)

The only missing feature that I'm concerned about is system, but I can just leave the system module formatter in so that's really not an issue.

@Rich-Harris How practical would this be from an esperanto perspective? I dislike duplicated effort and you've done and awesome job on esperanto so I'm more than happy to use it for modules. It'll also allow me to focus more time on other core features.

[Rich-Harris]

@sebmck That's definitely plausible from esperanto's perspective. It uses the start and end properties of each node in the AST to modify the code (rather than the range, which as I understand is just a [start, end] array, though it's probably of no consequence which it uses).

I've had some problems with SourceMapGenerator.fromSourceMap (i.e. #827 (comment) - though I haven't dug too deeply to find out what's going wrong) - my feeling is that transpilers shouldn't attempt to account for input sourcemaps, since it's surface area for bugs and it's useless unless everybody does it. I've found it to be much simpler and more reliable to flatten a chain of sourcemaps once, at the end - I wrote sorcery for this, I frequently use it with 4 or 5 chained sourcemap-generating transformations.

I'll open a corresponding issue on esperanto so we can figure out exactly what work would need to happen. It sounds like prioritising System support would be worthwhile in any case.

One thing to be aware of though, and this may be a showstopper, is that esperanto has a different approach to interoperability with CommonJS and AMD. Rather than working around the problems that default imports/exports introduce, esperanto has two modes. Ordinarily, your module/bundle can only use default imports/exports - if you want named imports/exports, you have to opt in to 'strict' mode. The philosophy here is that those of us who want to see ES module adoption should be the ones suffering any minor inconvenience (as opposed to a CommonJS user requiring our modules and getting something slightly different to what they expected), and that an explicit choice a) helps educate devs about what's happening, and b) avoids certain edge cases. (Also, at the time, discussions like #95 and esnext/es6-module-transpiler#85 showed no sign of resolution!) That may be untenable from your perspective.

[sebmck]

@Rich-Harris

I've found it to be much simpler and more reliable to flatten a chain of sourcemaps once, at the end - I wrote sorcery for this, I frequently use it with 4 or 5 chained sourcemap-generating transformations.

Ah right, understood. That's completely fine.

One thing to be aware of though, and this may be a showstopper, is that esperanto has a different approach to interoperability with CommonJS and AMD. Rather than working around the problems that default imports/exports introduce, esperanto has two modes. Ordinarily, your module/bundle can only use default imports/exports - if you want named imports/exports, you have to opt in to 'strict' mode.

Hm, that's a good point. Hadn't really considered that. I'm fine with not having the CommonJS export interop (I consider it an anti-pattern at this stage anyway). The CommonJS import interop is pretty critical, a possible "hack" would be for Babel to turn:

import from "bar";

into:

var _interopRequire = function (obj) { return obj && obj.__esModule ? obj : { default: obj }; };

import foo as * from "bar";
foo = _interopRequire(foo).default;

Only problem with that is getting __esModule on the exports. Babel will do:

Object.defineProperty(exports, "__esModule", {
  value: true
});

so it's not enumerable so adding export var __esModule = true; probably isn't practical but if this ends up being the only roadblocker to using esperanto then I'm fine with it just being enumerable.

[monsanto]

Is Esperanto the only tool that allows "concatenating" ES6 modules (by sorting dependencies and mangling names, as opposed to going through an indirection object like browserify does)? I haven't followed this space closely. If it was, I would expect it to be more front and center on their documentation.

Up to date information about this sort of thing would be great for the Babel documentation.

EDIT: One more thing, if Babel is to support optimizing code, it seems essential to add module bundling features so it can do things like global dead code elimination.

[Rich-Harris]

@monsanto as far as I'm aware esperanto is the only actively developed ES6 'native' bundler, yes. Originally it was for one-to-one transformations only, but then it learned how to bundle and the README/demo page haven't quite kept up.

The ability to eliminate dead code is a nice side-effect of using ES6 modules, though doing bundling through esperanto probably isn't going to help much there since you don't get a transformed AST out the other side. Possibly a task for a separate tool?

[sebmck]

Thanks everyone for the discussion! I'm making this as revisit as something to look into post-5.0.0 since I'm planning on releasing in a few days and this would be a pretty intrusive change.

[joliss]

The ability to eliminate dead code is a nice side-effect of using ES6 modules,

Yes, this so much - it's a pretty big deal especially for browser apps. For example it means we don't have to resort to per-method packages to save space.

though doing bundling through esperanto probably isn't going to help much there since you don't get a transformed AST out the other side. Possibly a task for a separate tool?

Yes, I think so. I've always thought dead-code-elimination to be part of minification. UglifyJS2 and the Closure Compiler both seem to do a pretty good job at this, and the output from Esperanto's bundle mode looks well-suited for them.

[sebmck]

@joliss

I think UglifyJS2 and the Closure Compiler both do a pretty good job at this, and the output from Esperanto's bundle mode looks well-suited for them.

Closure Compiler should handle it like a champ (Babel totally could too in the future 😜). Not too sure about UglifyJS2 since the minification it does is pretty rudimentary.

[Rich-Harris]

Yes, I think so. I've always thought dead-code-elimination to be part of minification.

Agree, though I've often wished for a separate tool that just did tree-shaking, and gave me a nice report about what code I could do away with (using sourcemaps to point to the original lines, natch). Would be a big help for refactoring. Anyone know if such a thing already exists?

[joliss]

@sebmck UglifyJS2 has a compressor mode (--compress) that isn't on by default for some reason. With the compressor it removes dead code paths and functions; it even understands const statements, like const DEBUG = 0; if (DEBUG) ....

[sebmck]

@joliss Babel can already do that with the utility.deadCodeElimination transformer. Was more referring to stuff like eliminating function declarations if they're never used etc.

[sebmck]

Although it's probably likely that Uglify can do that with the --compress option.

[benlesh]

Wanted to jump in and point out that the usability around this is really hard/bad. I found this while googling esperanto babel because I'm trying to figure out the best way to set up my build for a project.

The only reason I even knew to google Esperanto with Babel is because I hassled @stefanpenner about "how to bundle with Babel" a few weeks ago. Up to that point I'd never even heard of Esperanto (as a JS tool).

Another thing I want to point out: Everyone with input in this thread is a stone-cold expert at doing this. There is a real, unmet use case for a lot of developers that we need ES6 transpilation and module bundling in a simple easy to understand broccoli step, if possible.

... if that already exists, it's not well advertised.

TL;DR: I guess I'm hoping to urge you all to solve this problem and fill this gap in a meaningful way, whether that's teaming up with Esperanto, or just reimplementing it's best features in Babel.

I'm happy to help where I can, although I'm not sure where I could even being to help.

[jamesplease]

For posterity, if any future developer is trying to figure out Babel + Esperanto, I spent a long time figuring it out as part of a Gulpfile. My work can be seen in the official Babel boilerplate. That same code works well for me in both my little libraries and my big client side applications.

[jamesplease]

Well Esperanto is 💀

[eventualbuddha]

@jmeas that doesn't really mean much. It just got superceded by rollup which is by the same author as the original. Very similar approach, but it only does bundling and includes a feature that only puts code into the bundle that you actually use.