Discussion: entrypoints and dependency mode

[niloc132]

This issue is filed as a place to have a discussion, we'll make a decision after seeing what opinions pop up here.

Closure Compiler has a flag to specify what "dependency mode" that it should use to handle your inputs. In "PRUNE" mode (the default of this plugin until now), it requires that you specify entrypoints, and will include them and their dependencies in the order you specify. One caveat: if one entrypoint A (directly or indirectly) depends on another entrypoint B, this may change the order, since the B is then required to be included before A. But entrypoint classes with no reference to them will be dropped entirely, and never compiled at all.

Quick aside to discuss what an "entrypoint" is, relative to GWT2. While in GWT2, it meant that the onModuleLoad() method would be called. At this time in "gwt3", the pattern is to make a .native.js file for each entrypoint, which defers creating an instance of the class and calling onModuleLoad. This means that if the class is included at all in the resulting JS, its onModuleLoad will run. Encouraging that all entrypoints be private classes could help here, so you can't accidentally reference them. Another distinction: GWT2 runs them all in order in the same init method, while this "gwt3" pattern will run each in a different event loop entry, so one exception won't affect others, and the running time will be split up and potentially have a smaller UI freeze on startup.

The other "dependency mode" flag that can be provided is "SORT_ONLY", where all possible JS files are included in the output (at least to start with), and their contents are sorted so that everything can consistently depend on the things it needs. This mode requires no entrypoints to be listed, since they will all always be included, and run in whatever order the compiler picks.

With no other context, SORT_ONLY seems like it is usually the wrong answer, with the possible exception of libraries and frameworks that want to try to run their entrypoints before the application itself, without risking the application authors forgetting to list the library entrypoint in the plugin. We could solve this with some kind of classpath scanning approach or something like that, but that would require a convention that gradle, bazel, buck, maven, etc all agree on (like .gwt.xml of GWT2). There is however another case where this could matter: making development faster by caching the generated JS of every artifact on the classpath.

This approach says that we run Closure Compiler in BUNDLE (or potentially WHITESPACE_ONLY?) mode on each artifact individually, with the dependency mode set to SORT_ONLY. Since we don't know what downstream projects of a given jar might use, we have to include every single class. This results in much larger JS in the browser, but it also dramatically reduces the compile time of the application on incremental changes. The other downside is as discussed: any "entrypoint" in a dependency will be run before the depending library even has the chance to be loaded into the page.

Aside: I'm currently referring to this as not a new dev mode, but a new compilation level called "CHUNK", where you get a single .js file output, and that file includes each of the libs it needs at startup. It is like BUNDLE, but far faster and with far larger output. The closure compiled not only does less work this way, but even when you change a module with many downstream projects, all the re-bundling can happen in parallel. Besides the SORT_ONLY issue, there is one other downside: this mode is very particular about duplicated classes, so you absolutely cannot do any tricks with classpath hiding or replacing.

This approach has even better implications for tests, where instead of needing to recompile the entire project per test class, we can just run the same already-compiled test "library" code with a different entrypoint - much faster to produce, and will occupy much less disk space to do so.

Problem: if you develop and test in this new dev mode with "SORT_ONLY", but deploy to prod in "PRUNE", you might be removing entrypoints that you actually relied on in your development and your tests. So, we need to encourage projects to pick a dependency mode and stick with it.

So, some choices:

• Only support PRUNE: entrypoints are always required (or make up a serviceloader-like solution), order is manually specified, no more CHUNK dev mode with pre-compiled dependencies. Advantages: more "obviously correct", consistent in all modes.
• Support both PRUNE and SORT_ONLY: developers should pick the most appropriate for them and only use one, or need to be careful with tests vs dev vs prod. Advantages: most flexible solution, gives developers the most power. CHUNK+PRUNE will throw an exception rather than produce broken code
• Only support SORT_ONLY: entrypoint configuration is either optional or placebo (or removed), "watch" and "test" goals can now default to CHUNK (though BUNDLE is still an option) Advantages: consistent in all modes, CHUNK is available, etc

Supporting both could come with "default to SORT_ONLY" and maybe a warning on builds where it looks like you might switch between modes, but detecting that will be tricky.

One more wrinkle: J2CL's RTA tooling could make for a more effective way to prune unneeded classes, but it isn't yet exposed in a way that is easy to use anywhere except inside Google. Once we find a way to use this, it might be possible that we can simply re-bundle the CHUNKs and cache them (still more parallel than BUNDLE even in worst case), or it might be necessary to fall all the way back to BUNDLE, and render this discussion moot.

[FrankHossfeld]

"PRUNE" mode sounds like you have more control over the things that happen.

Having a choice between two modes might make it harder to use j2cl. Not sure, if this will scare people and so they avoid using j2cl ...

[niloc132]

"PRUNE" mode sounds like you have more control over the things that happen.

And that would be incorrect - you are just saying "i will tell you which entrypoints to pick, and from there, you should find what you need" instead of saying "just run it all"

Having a choice between two modes might make it harder to use j2cl. Not sure, if this will scare people and so they avoid using j2cl ...

This is my dilemma - there are power users who probably want access to this, but at the same time, providing a config option means someone will use it wrong :p. I think you can make a strong case for "only SORT_ONLY" or "support both", but I don't know what to do.

[FrankHossfeld]

I think many GWT developers love the easy way. They get paid to write business software. Making j2cl harder to use might let them move away.

So, I think, we should choose the way, that makes it easier, or more or less works similar to GWT 2 ...

But, atm, I have no idea, what would be the best ...

[niloc132]

Worst case, usability-wise, is that we support both, but then if we make the default to be SORT_ONLY, we can make it fast by default for tests, dev mode, and consistent in prod mode, but this still lets power users control it when they want to. We could go so far as to leave it "undocumented" or something, so that you have to know what you are looking for to find it, but that just seems evil to me.

Best case for usability is that we just pick one - I'm presently voting "sort-only" and we get rid of . Any class anywhere in your dependencies that tries to start stuff via a .native.js file would be enough then to start things up. This is going to prevent power users though from having the ability to kill certain entrypoints without forking the library they are using which provides it.

So: is it enough to just not advertise a config option and pick a sane default? Or do we need to actively take the choice away, and only allow what we decided to be right (in 99% of cases, and that last 1% is just in trouble)?

[hpehl]

I'm all about supporting only one option and completely remove the other one. From what I've read this should be SORT_ONLY. It comes with the most advantages especially in dev mode.

To me developer productivity is a very important point. Having very fast reload cycles even for big applications will attract developers and could be a USP for J2CL.

[niloc132]

Okay - Hearing from both of you (as library+framework authors, not just app authors) makes me feel like we could get away with that, but both of you still come from the fairly narrow range of "people who write 99-100% gwt tooling" rather than someone who mixes and matches js into the app, so I'm still hoping for some more opinions. I'll try to make the internals flexible enough to change our minds, but will generally only enable SORT_ONLY from the perspective of the end user (and likewise, will remove the entrypoint config option).

One caveat for library authors who want to include entrypoints in their own code: don't. It will be impossible for the application author to turn them off this way, without forking your library. Unless you are 100% sure that it is always the right answer, and there never will possibly be an exception, move such initialization logic into something that you force the user to call from their application.

[treblereel]

I think SORT_ONLY is the best option, fast hot-reload is one of the most handy features for dev. I never liked Entry Points so i agree, we must avoid EntryPoint inside our libraries.

[niloc132]

A few months of simmering on this, and some more discussion. The conclusion is that we're going to use SORT_ONLY, and if it is configurable, it will be somewhat "hidden" away. There will need to be some checks inside the plugin to confirm that this isn't in an incompatible state, but changing this won't be directly supported.

Advantages we will get:

• libraries can specify entrypoints, they will be unconditionally enabled (libraries should mitigate this by being cautious about introducing entrypoints, providing a way to disable them, offering more than one build, or telling users to invoke some code directly to enable the library). generally we want to discourage this anyway, as @treblereel pointed out.
• tests and dev mode can be super fast, each jar can be made into an single unpruned BUNDLE js file, rather than a whole-world compile at the end
• applications which function as a JS library can avoid introducing any entrypoints, and simply export the symbols that consumers will use
• entrypoint order will be based only on dependencies, rather than seeming to be based on the order of the entrypoint tags in the configuration
• entrypoints are usually implicit anyway, one less thing to configure.

Conclusions:

• entrypoint config tag will be deprecated, removed
• dependencyMode will be not be added - instead, may add a closureOptions tag so that this can be manually set, and entrypoints manually added this way, with caveats in the documentation that using this is generally unsupported.
• closure compiler will be invoked by default with dependencyMode=SORT_ONLY and no entrypoints
• bundle-per-jar mode can be added as a psuedo-compilationLevel (or something like that), and will be the default for test, watch goals.

Will leave this open for a little longer, then close and start an update. This will be a new version update to not break existing apps.

[niloc132]

Downside discovered prepping this for a push/release: if you use SORT_ONLY and there is a jsinterop type anywhere, even unused, that doesn't have externs provided, the compile will fail.

This is becoming an issue since there are 5 or so types in elemental2 1.0.0 that are missing from upstream closure-compiler - things removed "recently", such as a few days before 1.0.0 was cut.

I'm going to continue to proceed down this path, but keep patching our fork of closure-compiler to include these various missing externs.

Edit: I'm adding an integration test that builds elemental2 jars as part of the build. Also discovered along the way: svg externs are missing in closure, and elemental2-webassembly incorrectly is missing its dependency on elemental2-dom.