Conventions/Standards for handling JS files in jars

[niloc132]

JS included in Jars falls into roughly three categories:

• JS with code that is expected to be consumed by j2cl and/or closure (this includes .native.js files, which are consumed explicitly by j2cl and hidden from closure).
• JS with externs that is expected to be consumed by closure to describe other types not present in the other sources (note that this will break into two topics later).
• Js with code that is not expected to be consumed by closure (will be referenced at runtime by URL from the server, or will be passed through some codegen mechanism before j2cl runs to emit other kinds of output), and must be excluded from closure.

Externs fall into another category of problems: briefly, duplication of externs by independent sources will cause compile errors, with no option to "union" them in the compiler. As they cannot be simply detected while setting up the build (all JS would need to be parsed to achieve this), a convention of "extern JS" vs "code JS" would help to let the plugin make decisions about externs, such as a hypothetical "merge externs" step.

Aside from the special case of the .native.js standard enforced by j2cl, there is no convention that distinguishes these files, except from checking the actual contents of an externs file and finding a @externs annotation. This discussion aims to propose an approach where we can at least make it clear how these files can be handled by j2cl-maven-plugin when found in a jar or in a java project, and perhaps suggest a consistent standard to be followed by other j2cl build tools.

---

Exclude */public/* from Closure, copy */public/* to output

In GWT 2, the "public" package is used to indicate that some sources will be copied directly to the output, though nothing is said about whether or not those sources are present in the rest of the build. One proposal is to exclude all public/ directories from having their contents be passed to closure, and in turn, expect that any contents found within any public/ directory end up in the final output.

Exclude /META-INF/resources/* from Closure, copy */META-INF/resources/* to output

Another option, borrowed from webjars.org, is to include these resources in the jar's META-INF/ directory at its root. From https://www.webjars.org/documentation#servlet3

With any Servlet 3 compatible container, the WebJars that are in the WEB-INF/lib directory are automatically made available as static resources. This works because anything in a META-INF/resources directory in a JAR in WEB-INF/lib is automatically exposed as a static resource.

I am not proposing that we actually use the same path, since that would generally imply either that client-only jars are deployed to the server, or that client libraries split their resources into a jar just for server content and another just for j2cl/gwt. There is also the problem of the J2cl/Java code within a project understanding the appropriate absolute/relative path to use the find the resources (a *.war file might be hosted at /, or might have a sub-path). With that said, the idea does offer a canonical source directory to indicate what should be copied to the output directory.

It might make sense to use some path within META-INF/ rather than resources/ to be clear that this is specific to Java being compiled to JS before being built into a server, rather than just following the same conventions as webjars. On the other hand, allowing arbitrary webjars to be added to a j2cl project as a runtime dependency would be a handy way to depend on external JS resources without otherwise copying them into source control, or adding npm build steps.

Mark all contents that match /META-INF/closure-externs or */*.externs.js as externs and pass them to Closure

This would serve as a convention, any jar following this would enable the application author to opt-in to a later "merge externs" step, or "exclude externs, I'll provide my own" for manual merging or preferring one set of externs over another.

Handling externs in this way also suggests that j2cl libraries should include their own externs, rather than rely on closure-compiler's built in ones (which vary from release to release). Note that this is already possible through the <env>CUSTOM</env> configuration option, though the default value is BROWSER, as it is in closure-compiler itself. We would need to coordinate with libraries such as https://github.com/google/elemental2/ and https://github.com/akasha/akasha/ to achieve this consistency.

Proposed summary for discussion, sample patch to follow

• Exclude /META-INF/* entirely from closure-compiler, to allow these use cases and more unanticipated ones
• Exclude */public/* entirely from closure-compiler
• Assume any file matching *.externs.js OR /META-INF/externs/ is an externs file, emit an optional warning otherwise (to future-proof libraries for a merge/exclude option)
• Copy any contents of META-INF/resources/ and */public/ to the same output directory as the generated JS. This would not be exclusive to .js files, but could include other resources like images, fonts, css, etc.

[treblereel]

@niloc132 thank you for such a detailed explanation of the externs issue. First of all, i think we need the @realityforge opinion, simply because he is a maintainer of the most complete set of externs that exists. The Closure team is not very active at merging fixes and updates, so we might rethink which externs we are going to use as base level for our projects. 

I like the idea of a public folder because it’s close to gwt2 experience and very familiar to all gwt devs and it’s nice to be able to hold all app related resources in one place.

Also I like the idea of /META-INF/externs/ as a mandatory path for externs just because it helps us keep our libs clean.

An ability to serve web jars out of the box is a very cool feature, so it’s great to have.

Resolving externs conflicts is a huge and complex topic, so maybe we can resolve it later.

[mP1]

I believe conventions are useful but explicit declarations are better where possible.

Something like a file with paths or patterns of files that should be copied to the output. That gives maximum flexibility and control for minimal effort.

I believe explicit files are also better for new people as they stand out, and if you call the name something relatively unique they can google for that name and find the doco. With conventions like copying certain magic directories or directories with patterns, how exactly does the new reader know ? Even if they know its the j2cl-maven-plugin what do they search for to discover the paths/patterns. Its not obvious, sure they can find out, but again the unique filename is considerably quicker to find the online doco.

[niloc132]

I agree that maximum flexibility and control would be to offer a manifest file of some kind per artifact, but at this time I have to reject that as being necessary or reasonable in the j2cl ecosystem. There are several factors for this:

• J2CL itself exists and was explicitly designed as a rejection of "the compiler is the build tool", in favor of "the compiler will fit into many build tools and will adhere to their conventions". However, it was mostly designed for bazel (note how RTA only works for bazel, etc), where it is expected and enforced that for any code not already presented in a bazel project, your own project must reinvent the build, configuration, dependencies for that dependency and all of the transitive dependencies.
• As such, jars/etc designed to be usable in J2CL will not use manifests of any kind if they originate from a bazel project, but will instead follow established conventions. This includes j2cl's internals such as the jre and junit emulation, as well as jsinterop-base and elemental2. The conventions established so far are fairly minimal, but set a precedent, such as .native.js files being excluded by closure-compiler, while other .js files must be handled explicitly (though j2cl itself will not copy them automatically).
• The .externs.js convention (optional, currently one of two options required to use to opt-in to planned externs processing, but failure to use this may eventually/optionally warn or error) derives explicitly from this, and is clearer just by looking at it (unique filename) than the current system of any .js file being accepted with an @externs annotation inside of it. Enabling a warning/error would make this clearer to end users and make it harder to make a mistake in this area, so this seems to fit to the highest degree possible to both j2cl's own principles and your own comment.
• The META-INF/resources convention from webjars is documented at least on their website and in the Servlet 3+ specs, and will be respected if we follow this approach. The public/ directory is a convention that can be configured in GWT2, but it is enabled by default, and to my knowledge I have never seen anyone modify it from that default - it has historically been clear.
• The one addition then is supporting META-INF/externs as an optional location for plain .js files to be read as externs only. This is one that I am most skeptical of. My reasoning is that jsinterop-generator itself is deficient in the output it generates, as there is no possible way for a non-bazel build to build on top of elemental2-core, for example, so any downstream invocation of jsinterop-generator will necessarily emit output that is incompatible with elemental2. In keeping with the j2cl/jsinterop authors other design ideas, I don't expect them to provide a manifest file that supports configuration, but instead follow a standard convention to expose the name mappings and offer the already-handled externs, etc). It has been a few years since I talked about this with them though, and I need to follow up to solidify this.
• Back on more solid footing: A bazel project consuming a maven/gradle project following a convention already needs to reimplement configuration/convention to some degree. We have effectively two choices, pick a single convention so that only one bazel rule needs to be implemented to be compatible (maven_j2cl_library if you will), or provide a manifest file that bazel must first parse in starlark, and reading from that emit the appropriate rule output.

[mP1]

J2CL itself exists and was explicitly designed as a rejection of "the compiler is the build tool", in favor of "the compiler will fit into many build tools and will adhere to their conventions". However, it was mostly designed for bazel (note how RTA only works for bazel, etc), where it is expected and enforced that for any code not already presented in a bazel project, your own project must reinvent the build, configuration, dependencies for that dependency and all of the transitive dependencies

Bazel conventions or whatever are irrelevant here, because this is a maven plugin. We dont need to worry about bazel at all.

I agree that maximum flexibility and control would be to offer a manifest file of some kind per artifact, but at this time I have to reject that as being necessary or reasonable in the j2cl ecosystem.

Why would an end user not want this convenience for 30 seconds worth of effort in authoring and editing a manifest file ?

Conventions dont always work, this always happens. There is on provision in this thoughts for catering for those projects and people we dont know.

What exactly is being saved but not supporting a manifest ?

[niloc132]

Apologies, for having too much unrelated content in my reply, I'll be more explicit:

The bazel ecosystem will not support manifests, this is not an option. Maven jars that require a manifest to be read will not be understood in bazel, and jars published to maven via bazel will not contain them even if maven requires them. This will make the ecosystems incompatible. The bazel build rules become a manifest of a sort, though with some conventions that are configurable and many that are not. Google is the maintainer of the j2cl bazel rules, and will not support a manifest except the build rules themselves, which makes sense since you must declare explicit build rules for each and every dependency that doesn't live in a bazel project (and often even for those that do).

In contrast, offering consistent conventions will give us a chance to demonstrate good ways of handling complexity that make sense to support in multiple build systems. These conventions could be offered to bazel to ensure compatibility, and they might accept them (they will not accept a manifest file).

As such, there is no build tool agnostic manifest that can exist, unless maven itself invokes bazel, at which time it probably makes more sense to stop using maven entirely.

I am not interested in designing a maven plugin that cannot interoperate with bazel, that is a major loss from my perspective. I know you maintain your own plugin to support other features that I consider anti-goals - I have no objection to that, but this repo has specific aims of performance and interoperability that must be met.

[mP1]

The bazel ecosystem will not support manifests, this is not an option.

What problem are you trying to solve here. Im guessing you want Bazel to be able to produce the same jar from roughly the same sources as j2cl-maven-plugin.

In contrast, offering consistent conventions will give us a chance to demonstrate good ways of handling complexity that make sense to support in multiple build systems. These conventions could be offered to bazel to ensure compatibility, and they might accept them (they will not accept a manifest file).

Nobody cares about this.

I am not interested in designing a maven plugin that cannot interoperate with bazel, that is a major loss from my perspective.

Why ?

Theres no reason why you wouldnt simply make a special case of letting Google build their jar files they way they do, and coming back and adding a "manifest file" which matches their end products dir layout. Yes its a pain but its not that bad.

Unless im missing something problem solved, and everybody is happy with maximum flexibility.

[realityforge]

I think the most important question to answer before any decision can be made is roughly. "Do you want to use the same jar to support both GWT 2.x and J2CL?" In the beginning of our journey we thought the answer was yes. We would only switch isinterop-annotations and jsinterop-base when we moved between GWT and J2CL builds. Thus most of our (internal) packaging system was predicated on this assumption but now I no longer think this is a good idea.

Essentially the code that is optimal for J2CL is rarely the optimal shape for GWT when it comes to optimisations and vice versa. i.e. In Akasha (the browser adapter layer for those not familiar) we very deliberately use typedefs (which closure compiler can optimise) in preference to Object or ? types in GWT (which has no concept of typedefs or ability to use advanced type base optimisations in js compilation phase). This is particularly useful for unions. There is also lots of other small differences between the two (i.e. globals are <window> in GWT and goog.global in j2cl (so you can use logging during start phase in GWT or early crashes do not try to use un initialised goog.global etc). We also have constructs that are not supportable in j2cl without unchecked casts due to the closure type system whereas can just use native methods in GWT. Which means we often have additional @JsOverlay methods in j2cl that are not required in GWT. Yadda yadda yadda. So I don't think the js code will be the same between the two environments let alone any of the other supporting infrastructure. So while we have identical java APIs, the actual java code can be quite different under the API. Similar things should occur in other of our libraries but don't for various legacy reasons

Unlike @niloc132, in almost all the places where we have javascript included via the public directives in GWT modules we have multiple public paths in every GWT jar. In the javascript ecosystem it is pretty common to have a foo.js and a foo.min.js (and possibly foo.profile.js etc). These are just different versions of the javascript compiled with different minimisation techniques applied. So we will often include a FooDev and FooProd (or just Foo for libraries) GWT modules in a jar depending on whether you are building javascript that has debug-ability enabled or not. Each of these GWT modules will include different js via different public paths. An example of this is at DomDev.gwt.xml where our dev GWT module includes react-dom.js rather than react-dom.min.js (or whatever the patterns is for react - I forget atm).

Conceptually, in a "normal library" we would typically include non-closure js, closure js, .native.js js, .externs.js js, assets that that should not be modified but copied to output artefact (maybe with an encoded version ala brotli files to speed up serving), assets that can be modified (i.e. final css and images that can be optimised such as images and final css), assets that need to be processed (i.e. foo.less, foo,scss). While you may be able to come up with conventions that cover some of these scenarios that may work some amount of the time, I can't think of anything that would work in a broad range of scenarios. (Well unless you build in a build system into a manifest which is an explicit non-goal). I can easily see a case for excluding paths from processing (i.e. META-INF/* should be ignored by j2cl/closure tools) but it is hard to see anything more sophisticated that would be adopted across build tools.

It should be noted that this thinking is very different (the opposite?) from my past thinking where I thought that it would be possible to use a single jar between GWT and J2CL and we should include a manifest that multiple tools could read and process. While such a tool may be possible I would be more inclined to have separate tools for each task. i.e. the j2clmavenplugin includes all javascript in jar ignoring META-INF/**. Another separate plugin that copies META-INF/assets/* (or whatever) to the war. Each plugin can be optimised for own workflow and an application can be assembled from these plugins as appropriate

[realityforge]

Please note that goog.global is available in gwt2 as well as of 2.9, and it is set to whatever $wnd is.

Correct ... but it is possible that GWT generated code can run before that variable is initialised. There was a few "normal" flows that could lead this to occur and a few "abnormal" flows (a.k.a. errors) that could result in this scenario. We used to find it very annoying when our console logging failed during error scenarios because goog.global was not initialised ;-)

If you do not care about libraries being compatible between GWT2.x and J2CL then a lot of problems go away. (I think it is a nice to have but a bit of a fools errand that just leads to suboptimal outcomes for both ecosystems). The best approach from our experience is just to have separate maven artefacts for each variant (with different dependencies etc).

As to what we would have in our j2cl jar in the ideal world.

• The java source code and all stuff generated by annotation processors related to the original source code. (i.e If using the libraries you referenced above it would be MyModel.java, Arez_MyModel.java, MyView.java, React4j_MyView.java, MyInjector.java, Sting_MyInjector.java etc).
• The .class files for the above.
• Any *.native.js associated with the above classes.
• Other javascript *.js ... may be externs or may be other javascript. For the "other javascript", it is usually limited in scope but our libraries include goog.define(...) that is loaded via a *.native.js as a simple mechanisms for managing compile time constants. (i.e. arez.js is required by ArezConfig.native.js
• nothing else...

It should be noted that this is basically the same way Bazel sees the world and I am not sure if I was influenced by Bazel or evolved to the same mindset. This works as the UI toolkit we use mostly has a "views as code" and "css as code" mindset. It also means you can basically just pass the entire jar to the j2cl/closure toolset and it mostly works. I don't know if it would work when using a framework that used external css, external js and/or external html as part of a library. However I suspect that if all of this data could be shoved off into a META-INF/<framework specific> hierarchy a separate maven plugin is responsible for doing whatever is required to get it into the war (whether that is transform, merge or just copy) then you are more likely not to create a legacy approach from the start.

It seems like you already have a design in mind and are just trying to validate, if so just go with it. However the GWT 2.x use case you are trying to cover the 80% for was designed for the web as it existed in 2011 or so and the ecosystem has moved on. I would recommend just focusing on the java->js part with your plugin and leave the rest for other tooling but 🤷‍♂️

[niloc132]

In my defense, I'm not trying to optimize for all possible layouts X use cases (in fact I'm explicitly not doing that, but seeking to define conventions that handle the use cases consistently - I apologize for not being clearer about this.

I am seeking to validate the design, but as "could this be a reasonable middle ground that expected existing/future use cases could meet in" rather than "will this satisfy all possible configurations that do or could exist". Leaving integration points for other tooling to kick in ("generate code before j2cl or closure run", "do some webpack-ish thing after closure has acted on code it can handle", etc) is definitely part of the goal. Handling the ~80% use case of GWT 2 libraries is ideal, but I have yet to see an existing GWT 2 app that didn't at least have some JSNI or custom linker/generator - the last mile of porting those is always going to be manual.

We could certainly focus only on java->js, but there isn't a huge market for "we have closure-compatible es6+ JS, what do you want to do next" and a good workflow that proceeds from there, so integration of closure-compiler is necessary too.

---

Correct ... but it is possible that GWT generated code can run before that variable is initialised.

Thanks - I hadn't run into that yet, I might get back to you separately about that topic. That said, this is pretty safely in the ~20% "this won't cleanly work across versions", and I think we expect some libraries to be j2cl-only, or to support them separately.

---

However I suspect that if all of this data could be shoved off into a META-INF/ hierarchy a separate maven plugin is responsible for doing whatever is required to get it into the war (whether that is transform, merge or just copy) then you are more likely not to create a legacy approach from the start.

That seems like a good way to put it. Phrased roughly as "META-INF/externs is one way we support externally externs", "META-INF/resources is a well-supported way to handle web resources, put it on your server classpath to handle it there, put it on the client classpath to copy it here", and "anything else in META-INF is up to your own build wiring".

---

I agree on your ideal jar, the only edge cases are shipping JS that isn't safe for use in closure, or needing to cheat and use externs to handle shortcomings of closure's export mechanism (arez.js is a lot like jre.js and jsinterop.js, meant to be passed to closure-compiler). Up until now, this ideal jar is all that this plugin has supported, though we've long had people asking us to extend it slightly, and the months of off-and-on discussion have lead me to this proposal.

Thanks for taking a look!

[realityforge]

That seems like a good way to put it. Phrased roughly as "META-INF/externs is one way we support externally externs", "META-INF/resources is a well-supported way to handle web resources, put it on your server classpath to handle it there, put it on the client classpath to copy it here", and "anything else in META-INF is up to your own build wiring".

Any reason why externs can not just be part of main source tree and just have an @externs tag in @fileoverview section ala https://github.com/react4j/react4j/blob/master/core/src/main/java/react4j/react.externs.js#L4 or https://github.com/akasha/akasha/blob/master/akasha/java/src/main/java/akasha/core/akasha_core_patches.extern.js#L3

(While we name the externs with the pattern *.externs.js, this convention is for humans rather than to support tools.)

I agree on your ideal jar, the only edge cases are shipping JS that isn't safe for use in closure, or needing to cheat and use externs to handle shortcomings of closure's export mechanism (arez.js is a lot like jre.js and jsinterop.js, meant to be passed to closure-compiler). Up until now, this ideal jar is all that this plugin has supported, though we've long had people asking us to extend it slightly, and the months of off-and-on discussion have lead me to this proposal.

I was one of those people that keep asking for it ;-) Mostly so we could use a single jar and share it across GWT and J2CL ... GWT already has this solved easily enough. We only have a few libraries where it is actually relevant but unfortunately the libraries are at the bottom of a dependency tree.

i.e. Pretending this is the tree (But in real apps there are usually a lot more MyLib* libs.

MyApp
  |----> MyLib3
           |----> MyLib2
                    |----> MyLib1
                             |----> React4j
                                    |----> BrainCheck
                                           |----> Akasha
                                    |----> Akasha
                                    |----> Arez
                                           |----> BrainCheck
                                                  |----> Akasha
           |----> MyLib4
                    |----> React4j
                           |----> BrainCheck
                                  |----> Akasha
                           |----> Akasha
                           |----> Arez
                                  |----> BrainCheck
                                         |----> Akasha
           |----> BrainCheck
                  |----> Akasha

Akasha, BrainCheck and React4j have different artefacts depending on whether they are used by J2CL or GWT libraries while the MyLib* and Arez libraries are all "pure" or at least there is small enough differences between GWT and J2CL that it is irrelevant. Currently we are forced to pick a variant (i.e. GWT) and then use artefact replacement strategies to target artefacts in the dependency tree. This is most likely similar to how you handle jsinterop-base, jsinterop-annotations and jre. A total PITA and where most of our issues arise.

[niloc132]

Externs totally work without a naming convention, and could continue to do so, but different jars might model the same browser apis in different ways, and closure doesn't like seeing more than one externs definition for the same thing. It seems simple enough to me, having not thought about it, closure could just union the types and let both definitions work, but there must be some edge cases that prevent that from actually making sense.

Simplest example is akasha vs elemental2 - even in a perfect world where we put elemental2's externs in elemental2 and use CUSTOM, and let akasha contribute its own externs based on IDLs, those two sets are going to disagree about something. If we require all externs be identifiable without parsing them (as closure will inevitably do), the absolute worst case becomes "pick your own set of final externs and name them directly, and configure your project to reject any externs in jars". Less awful is "naive and stupid externs generator that replaces all types with ?s" and emits only one giant externs file, progressing up to pretty-damn-good of "some project which actually cleanly merges externs, maybe we can land this upstream?".

The alternative is that we parse each incoming .js file (or just grep it and hope for the best like BUNDLE does ;) ) for @externs and use that as a way to identify them, but I'd rather not parse things more than necessary.

With that said, the typedast approach, if successful, is going to be pre-closure anyway, so we could parse as little as once and be done with it, and then relax this constraint again.

And note that this constraint isn't actually a constraint - we won't even so much as warn if an externs file isn't following this set of conventions (yet), but this is really just an attempt to see if we can standardize not-really-JS slightly and let it be processed differently.

---

I see you keep mentioning jsinterop-annotations as one of the artifacts that must be different, but I don't think this is true? j2cl uses 2.0.0, and gwt2.9 also uses 2.0.0?

[realityforge]

... different jars might model the same browser apis in different ways...

Good luck with that! Externs that are disjoint are easy to merge, externs that are overlapping will not be easy to change without changing massive amounts of code, externs and or giving up on type based optimisations. I spent a long time trying to get enough changes to closure externs ... and then a similar amount of time trying to get akasha to align with the closure externs that I can't imagine that will be done anytime in near future.

And note that this constraint isn't actually a constraint - we won't even so much as warn if an externs file isn't following this set of conventions (yet), but this is really just an attempt to see if we can standardize not-really-JS slightly and let it be processed differently.

Maybe when there is a tool that will take advantage of such a convention in the future but wouldn't it be better to defer that decision until there is an actual use?

I see you keep mentioning jsinterop-annotations as one of the artifacts that must be different, but I don't think this is true? j2cl uses 2.0.0, and gwt2.9 also uses 2.0.0?

True ... GWT will just ignore a bunch of them ... JsNullable, JsNonNull, JsOptional, JsAsync, JsEnum .. we have switched back and forth whether we allow them in our libraries. In some of our builds we switch out the jars so that annotations that have do not work as advertised in GWT are not part of the GWT jar. But I believe in akasha we still use JsNullable in both GWT and J2CL variants ... for some reason ...