feature: concurrency action

[Alizter]

• See the discussion: [RFC] concurrent progn action #6937

There are a few cases in which we need an action like (progn ) that runs the child actions concurrently. Take for instance:

• Running inline tests in parallel Run inline tests in parallel #1516
• Diffing multiple files at once without failing early. (See concurrent-multi-diff.t/run.t for an example).
• Or more generally, running multiple actions at once without failing early. (See concurrent.t for an example).

We add a new action called (concurrent ) which acts just like (progn ) with the difference that it's child actions are run concurrently.

• Documentation
• Changelog

[snowleopard]

The current implementation doesn't interact with -j N setting in the right way. Surely, every action under conc should take a slot in the global job throttle.

[snowleopard]

(Less importantly, the pair of names progn and conc is pretty much impossible to guess. I think we should go for something like sequence and parallel, and deprecate progn.)

[rgrinberg]

The current implementation doesn't interact with -j N setting in the right way. Surely, every action under conc should take a slot in the global job throttle.

Are you sure? I think we only throttle spawning processes not actions.

(Less importantly, the pair of names progn and conc is pretty much impossible to guess. I think we should go for something like sequence and parallel, and deprecate progn.)

At least progn is a standard name from lisp. I have no idea what's a "conc".

[snowleopard]

Are you sure? I think we only throttle spawning processes not actions.

Well, and the PR description says this feature is going to be used for running tests in parallel, which are external processes.

I don't see much point in a combinator like conc if it doesn't interact with the -j setting, because that will become a footgun for rendering your machine unusable when running too much stuff in parallel without a throttle.

Basically, as soon as you start caring about action parallelism, you should start caring about the -j throttle.

[snowleopard]

OK, I might have been confused about the purpose of conc all along, as I've written in my comment here.

[rgrinberg]

I don't see much point in a combinator like conc if it doesn't interact with the -j setting, because that will become a footgun for rendering your machine unusable when running too much stuff in parallel without a throttle.

Note that it does interact with the throttle exactly the same as everything else in dune. For example, (conc (run ..) (run ..)) will respect -j as expected according to this implementation because the underlying calls to Process.run all take the throttle into account. It is only other actions that will run "concurrently". But in reality, there will be no concurrent execution of any other actions because dune just executes them all sequentially. But at least we'll be able to avoid the fail-on-first-error behavior. So (conc (diff ..) (diff ..)) will try both diffs even if one fails.

[snowleopard]

@rgrinberg Huh, sorry, I guess I had a different mental model about where throttling happens. That this does interact well with run is great! I'll re-review the PR some time next week.

[Alizter]

There is a little subtlety that I feel deserves some documentation. The actions are not maximally concurrent as one might expect from the name. For example, concurrent A B will deadlock/timeout if A waits for a message from B and Dune is running with -j 1.

Do you have an example? I don't see how you can write an action that waits on the output of another.

[snowleopard]

There is a little subtlety that I feel deserves some documentation. The actions are not maximally concurrent as one might expect from the name. For example, concurrent A B will deadlock/timeout if A waits for a message from B and Dune is running with -j 1.

Do you have an example? I don't see how you can write an action that waits on the output of another.

Sure, imagine writing an ad-hoc test that involves a client app talking to a server app via TCP. One might reach out for the concurrent action and write something like this:

(rule
 (deps (client.exe server.exe))
 (action
  (concurrent
   (with-stdout-to client_stdout (run client.exe))
   (with-stdout-to server_stdout (run server.exe)))))

... and it will work just fine, as long as you're not running with -j 1. Maybe in future, we could extend concurrent to take an argument specifying how many concurrent jobs are necessary for the rule to work, and Dune would fail if -j N is too low to be able to satisfy the requirement (or maybe override the parallelism limit just for this rule?).

[snowleopard]

Just to add: introducing the max_concurrent_jobs field to concurrent may also be useful for cases where you actually want no parallelism, e.g. where all you care about is just the right error-collection behaviour (as I explained in my earlier comment). In those cases, you'll be able to just say (max_concurrent_jobs 1).

But it seems better to do that in a separate PR.

[Alizter]

@snowleopard I'm not sure something about the number of allowed jobs really belongs in the engine. In that case I would say a field should be added to the rule stanza allowing it to override the job limit in the scheduler. In any case, I can add a note in the documentation about this use case, and a warning about the number of jobs.

[snowleopard]

@snowleopard I'm not sure something about the number of allowed jobs really belongs in the engine. In that case I would say a field should be added to the rule stanza allowing it to override the job limit in the scheduler. In any case, I can add a note in the documentation about this use case, and a warning about the number of jobs.

Yes, please a note to the docs that the actual concurrency is limited by -j number.

Thanks for adding the test!

[Alizter]

@snowleopard Thanks! That reads much better.

[snowleopard]

Looks good, thanks! (I made one last little tweak to the doc, hope it's OK.)