Task runner capabilities

[klntsky]

Often in our build scenarios we have multiple spago commands running against the same set of files.

E.g. something like:

spago test
spago bundle-app -m M1 --to file1.js
spago bundle-app -m M2 --to file2.js
spago bundle-module -m M3 --to file3.js
...

I think that by allowing spago to inspect the scenario beforehand, it is possible to achieve a speedup. Config re-reading, re-validation of packages and some calls to purs could be skipped, if only spago were smart enough to determine which steps are no-op.

The interface for this could be as simple as specifying commands multiple times:

spago bundle-app -m M1 --to file1.js bundle-app -m M2 --to file2.js bundle-module -m M3 --to file3.js

This does not even require a breaking change, AFAIK.

[f-f]

I'm not too keen on this, as I see that kind of interface quite complex (like, see #353 for an idea of which corner cases these kind of interfaces entail)

As far as I know config reading and purs calls take very little time, so if you're encountering any performance issues I'd say we should try to address these underlying problems instead of trying to optimize some specific case

[klntsky]

OK, the proposed interface is only one of the possibilities.

purs calls take very little time

With a decent hardware, maybe. I'm currently having this for a no-op build (after a warm-up):

$ time spago build
Installation complete.
Build succeeded.
spago build  13.28s user 2.31s system 147% cpu 10.564 total

[f-f]

@klntsky oh wow, a no-op shouldn't take this long 🤔

Could you try using tdiff on spago -v build and posting the results here?
Also, on which project are you running this? It would be interesting to have a replication so I can compare with the times I get on my machine

[klntsky]

(2.86 s) Running `spago build`
(102.95 µs) Ensuring that the package set is frozen
(21.09 µs) Getting transitive deps
(11.94 µs) Running `fetchPackages`
(10.18 µs) Checking if `purs` is up to date
(14.96 µs) Running `getGlobalCacheDir`
(14.25 µs) Installation complete.
(8.26 s) Running command: `purs compile  ".spago/aff/v5.1.1/src/**/*.purs" ".spago/aff-promise/v2.1.0/src/**/*.purs" ".spago/argonaut-codecs/v6.0.2/src/**/*.purs" ".spago/argonaut-core/v5.0.0/src/**/*.purs" ".spago/argonaut-generic/v5.0.0/src/**/*.purs" ".spago/arraybuffer-types/v2.0.0/src/**/*.purs" ".spago/arrays/v5.3.0/src/**/*.purs" ".spago/avar/v3.0.0/src/**/*.purs" ".spago/bifunctors/v4.0.0/src/**/*.purs" ".spago/catenable-lists/v5.0.1/src/**/*.purs" ".spago/colors/v5.0.0/src/**/*.purs" ".spago/console/v4.2.0/src/**/*.purs" ".spago/const/v4.1.0/src/**/*.purs" ".spago/contravariant/v4.0.1/src/**/*.purs" ".spago/control/v4.1.0/src/**/*.purs" ".spago/coroutines/v5.0.0/src/**/*.purs" ".spago/css/v4.0.0/src/**/*.purs" ".spago/datetime/v4.1.1/src/**/*.purs" ".spago/distributive/v4.0.0/src/**/*.purs" ".spago/dom-indexed/v7.0.0/src/**/*.purs" ".spago/effect/v2.0.1/src/**/*.purs" ".spago/either/v4.1.1/src/**/*.purs" ".spago/enums/v4.0.1/src/**/*.purs" ".spago/exceptions/v4.0.0/src/**/*.purs" ".spago/exists/v4.0.0/src/**/*.purs" ".spago/exitcodes/v4.0.0/src/**/*.purs" ".spago/foldable-traversable/v4.1.1/src/**/*.purs" ".spago/foreign/v5.0.0/src/**/*.purs" ".spago/foreign-object/v2.0.3/src/**/*.purs" ".spago/fork/v4.0.0/src/**/*.purs" ".spago/free/v5.2.0/src/**/*.purs" ".spago/freeap/v5.0.1/src/**/*.purs" ".spago/freet/v4.0.0/src/**/*.purs" ".spago/functions/v4.0.0/src/**/*.purs" ".spago/functors/v3.1.1/src/**/*.purs" ".spago/gen/v2.1.1/src/**/*.purs" ".spago/generics-rep/v6.1.1/src/**/*.purs" ".spago/globals/v4.0.0/src/**/*.purs" ".spago/halogen/v5.0.0-rc.5/src/**/*.purs" ".spago/halogen-css/v8.0.0/src/**/*.purs" ".spago/halogen-vdom/v6.1.0/src/**/*.purs" ".spago/html-parser-halogen/890da763cdd2a1049ab8837e477c5ba1fcf6d4ce/src/**/*.purs" ".spago/identity/v4.1.0/src/**/*.purs" ".spago/integers/v4.0.0/src/**/*.purs" ".spago/invariant/v4.1.0/src/**/*.purs" ".spago/js-date/v6.0.0/src/**/*.purs" ".spago/js-timers/v4.0.1/src/**/*.purs" ".spago/lazy/v4.0.0/src/**/*.purs" ".spago/lcg/v2.0.0/src/**/*.purs" ".spago/lists/v5.4.1/src/**/*.purs" ".spago/markdown-it/v0.4.0/src/**/*.purs" ".spago/markdown-it-halogen/08c9625015bf04214be14e45230e8ce12f3fa2bf/src/**/*.purs" ".spago/math/v2.1.1/src/**/*.purs" ".spago/maybe/v4.0.1/src/**/*.purs" ".spago/media-types/v4.0.1/src/**/*.purs" ".spago/memoize/v5.0.0/src/**/*.purs" ".spago/newtype/v3.0.0/src/**/*.purs" ".spago/node-buffer/v5.0.0/src/**/*.purs" ".spago/node-fs/v5.0.1/src/**/*.purs" ".spago/node-fs-aff/v6.0.0/src/**/*.purs" ".spago/node-path/v3.0.0/src/**/*.purs" ".spago/node-process/v7.0.0/src/**/*.purs" ".spago/node-readline/v4.0.1/src/**/*.purs" ".spago/node-streams/v4.0.1/src/**/*.purs" ".spago/nonempty/v5.0.0/src/**/*.purs" ".spago/now/v4.0.0/src/**/*.purs" ".spago/nullable/v4.1.1/src/**/*.purs" ".spago/options/v5.0.0/src/**/*.purs" ".spago/optparse/v3.0.1/src/**/*.purs" ".spago/ordered-collections/v1.6.1/src/**/*.purs" ".spago/orders/v4.0.0/src/**/*.purs" ".spago/parallel/v4.0.0/src/**/*.purs" ".spago/partial/v2.0.1/src/**/*.purs" ".spago/posix-types/v4.0.0/src/**/*.purs" ".spago/prelude/v4.1.1/src/**/*.purs" ".spago/profunctor/v4.0.0/src/**/*.purs" ".spago/proxy/v3.0.0/src/**/*.purs" ".spago/quickcheck/v6.1.0/src/**/*.purs" ".spago/random/v4.0.0/src/**/*.purs" ".spago/record/v2.0.1/src/**/*.purs" ".spago/refs/v4.1.0/src/**/*.purs" ".spago/search-trie/v1.0.0/src/**/*.purs" ".spago/st/v4.0.0/src/**/*.purs" ".spago/string-parsers/v5.0.0/src/**/*.purs" ".spago/strings/v4.0.1/src/**/*.purs" ".spago/tailrec/v4.0.0/src/**/*.purs" ".spago/test-unit/v15.0.0/src/**/*.purs" ".spago/these/v4.0.0/src/**/*.purs" ".spago/transformers/v4.2.0/src/**/*.purs" ".spago/tuples/v5.1.0/src/**/*.purs" ".spago/type-equality/v3.0.0/src/**/*.purs" ".spago/typelevel-prelude/v5.0.0/src/**/*.purs" ".spago/unfoldable/v4.0.2/src/**/*.purs" ".spago/unsafe-coerce/v4.0.0/src/**/*.purs" ".spago/unsafe-reference/v3.0.1/src/**/*.purs" ".spago/web-clipboard/v2.0.0/src/**/*.purs" ".spago/web-dom/v3.0.0/src/**/*.purs" ".spago/web-events/v2.0.1/src/**/*.purs" ".spago/web-file/v2.1.1/src/**/*.purs" ".spago/web-html/v2.2.1/src/**/*.purs" ".spago/web-storage/v3.0.0/src/**/*.purs" ".spago/web-touchevents/v2.0.0/src/**/*.purs" ".spago/web-uievents/v2.0.0/src/**/*.purs" "src/**/*.purs" "test/**/*.purs"`
(181.28 µs) Build succeeded.

when run on docs-search.

[f-f]

Right so the step taking the majority of the time is the call to purs.
We might want to report an issue to the compiler to see if we can improve further the no-op times (but I'd say this is very much a priority, and e.g. the recent rewrite of the parser was also aiming to reduce no-op times)

But in any case we have the --no-install and --no-build flags that you can pass to all the commands that might download packages or call purs to have them skip the expensive steps. Would that work for you?

[klntsky]

But in any case we have the --no-install and --no-build flags

Yep. My point was that they could be used automatically when possible.

Are you against introducing that kind of "cleverness" to spago?

Also, I just realized that it may be possible to detect two or more sequential no-op builds by storing timestamped build history or some info about the last build somewhere. On successful detection of a no-op build, a warning could be spawned, suggesting to use these flags.

[f-f]

Are you against introducing that kind of "cleverness" to spago?
Also, I just realized that it may be possible to detect two or more sequential no-op builds by storing timestamped build history or some info about the last build somewhere. On successful detection of a no-op build, a warning could be spawned, suggesting to use these flags.

How would we detect changes and decide to perform a build instead of skipping it? The most straightforward option would be to hash the source files and check if the hash is the same (if it changed you need to rebuild).
I'd not be against introducing this cleverness here, but it'd be more proper if the compiler itself would have this logic and skip unnecessary build.
In fact, there's an ongoing effort to do exactly this in purs by introducing Shake to use content hashes to determine if a build should happen

[klntsky]

I was thinking that it could be done by checking the time of the end of the last spago call, i.e. "how long ago did the last build end?". So that if there are many sequential runs, only the first would result in calling purs. And the consequent would assume no rebuild needed.

[klntsky]

And to protect this scheme from source file modifications during the build itself, it is possible to watch the files while purs does its work.

[f-f]

I have the feeling this would just be a big workaround for purs not having a smarter build system, and we'd be better off putting effort in fixing that first.
The danger here is that we'd introduce a bunch of heuristics that would work fine most of the time, but then break someone's build because spago is trying to be too smart

[klntsky]

OK, let's close this then.