[help] Expected behavior with dynamic targets and {crew}

[mglev1n]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm curious about the expected behavior when running a {targets} pipeline that contains dynamic targets using {crew}. When using tar_make_future() or tar_make_clustermq(), if the number of dynamic branches (b) is greater than the number of workers (n), only n branches are reported as "started" when the pipeline is first run. As branches complete, new branches are started, such that n branches are running at any given time.

When using {crew}, the behavior appears to be different. For example, when starting a pipeline with b > n branches, after running tar_make(), {targets} seems to print to the console that all b branches are "started". My sense is that these branches are all submitted to the {crew} controller, which then allocates branches to workers as appropriate? I wasn't sure if the difference in reporting between tar_make_future()/tar_make_clustermq() and tar_make() using {crew} is intentional, or is a bug.

I can work on a reprex if this description is confusing.

[wlandau]

I saw this behavior too, and I recently added a footnote in the manual to explain:

When you see “start target data”, this does not necessarily mean that the R command get_data() is actually running. All it means is the target is ready to run and was pushed to the crew queue. If the crew controller already has a large backlog of unresolved tasks, it may take a long time for the task to actually begin running.

I don't really see a good solution to this. It might be clearer to say "send target x" instead of "start target x", and I will consider it, but it's a heavy lift because of the progress metadata. Alternatively, I thought about withholding new tasks from crew if the workers are already saturated, but this could get inefficient in the general case, and it could seriously bottleneck controller groups.

[wlandau]

Actually, I figured out a way to safely and efficiently withhold tasks from crew controllers until workers to run them can be launched. So as of 2a5874e, "start" should mean actually started (or at least approximates it a lot better).