[help] targets + crew.cluster get stuck towards the end of the pipeline, but get unstuck if I restart

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

[premise] This issue has been recurring for a long time with different workflows and different software versions. I am using the most recent targets+friends ecosystem.

I have a workflow with thousands of targets (targets file below), and after I leave it for few days it gets stuck as below

It stays stuck as such for 20+ hours

The HPC has still running jobs (apparently), so I would assume each target just takes a long time.

However, If I stop the pipeline and relaunch it, I see that this is not the case! Targets finish in minutes and seconds, and the pipeline completes

Perhaps setting tasks_max would avoid permanent workers that get stuck for some reason, or should I set seconds_idle? Although I don't know if those workers are formally idle.

It is not easy to to repex because this behaviour probably depends on the fact that this is a huge workflow. in fact, it only happens when the workflow is toward the end.

[wlandau]

tasks_max = 1 might help if your tasks are somehow accumulating objects in the global environments of the workers. So would worker-level storage management and memory optimization in targets: https://books.ropensci.org/targets/performance.html