Allow fractional jobs to automatically scale by number of CPUs (200USD Bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 200USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

so -j 0.5 on a 10 CPU machine would be the same as -j 5, on a 4 CPU machine the same as -j 2, etc. -j 1.0 would be the current default, and -j 2.0 would mean twice as many jobs as CPUs (e.g. if they're mostly IO bound).

I've wanted -j 0.5 myself to use in Mill's own CI, because Mill tests are very heavy and use more than one thread per task, and it would be nice if the same -j config would just do the right thing on different machines with different amounts of available compute

[lefou]

Nice idea.

[wb14123]

Shall we use another flag for this? It's confusing that -j 1.0 is not equal to -j 1.

[lihaoyi]

how about -j 0.5n or -j 0.5*HOST_CPUS (what baze uses)?

[lihaoyi]

Bazel supports basic arithmetic and allows things like HOST_CPUS-1 as well

[wb14123]

Yeah I think we can take what Brazel uses, specifically in this doc:

--local_{ram,cpu}_resources resources or resource expression
These options specify the amount of local resources (RAM in MB and number of CPU logical cores) that Bazel can take into consideration when scheduling build and test activities to run locally. They take an integer, or a keyword (HOST_RAM or HOST_CPUS) optionally followed by [-|float] (for example, --local_cpu_resources=2, --local_ram_resources=HOST_RAM.5, --local_cpu_resources=HOST_CPUS-1). The flags are independent; one or both may be set. By default, Bazel estimates the amount of RAM and number of CPU cores directly from the local system's configuration.

It only allows keyword at the beginning and only supports * and -, which makes parsing not too hard.

[lihaoyi]

Yeah don't even need a real parser, pattern matching on case s"HOST_CPUS*$n" => would be enough

[wb14123]

Oh that's cool! I didn't know you can do that in Scala. I can try to implement this feature.

[lefou]

In Maven it is -T n for using n threads and -T nC for using n threads per CPU core, which I find nice.

On a 16 core cpu, these three command would all mean the same:

mill -j 0
mill -j 16
mill -j 1C

It's basically what @lihaoyi suggested in the first comment, but with a C suffix for "per core".

[lihaoyi]

@wb14123 go for it. I don't mind either the maven way or the bazel way, either works as long as we document it

[wb14123]

I'm wondering if HOST_CPU-1 can be useful for some cases? It's basically using most of the cores but leave 1 core for some other tasks in the OS. If we want to make it shorter we can support things like 0.5C and C-1.

[lefou]

I'm not used to bazel, but I find the term HOST_CPU misleading, since my machine has only one cpu but 16 cores.

[wb14123]

Hmm, then I think we can support the format like maven 0.5C, but also add support for things like C-1.

[lihaoyi]

Yeah subtraction definitely seems useful.

On some further thought i think a bit more verbosity here is good. C is a bit cryptic and hard to google. If we don't want HOST_CPUS, we could call it NUM_CORES or CORES or something similar

[lefou]

If we mention the "C" in the mill --help I think having a compact notation is much better than the long form. Since I need to type it every time on the cli. -j is even harder to google, but we use it.

[LvlAndFarm]

I've pushed a PR which lets you specify fractional thread counts using the C suffix. Happy to change it if you prefer something else!

[wb14123]

Oh I was working on this as well and didn't notice @LvlAndFarm created a PR. Anyway, I pushed my code just in case it helps: #3554

I tried to do some manual test with ./mill -i dist.run example/scalalib/builds/9-realistic -i "foo[3.3.3].run" -j 1 but seems threadCountRaw is always None in my branch. Am I doing the test command right?

[wb14123]

Oh found out -j should be put before the target example/scalalib/builds/9-realistic. Tested some use cases and looks good. Not sure where to add automated tests tho.

[lihaoyi]

Going to close this since we have a PR in #3482.