Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improper progress indication for cached steps #62

Closed
giuseppe-steduto opened this issue Aug 29, 2023 · 0 comments · Fixed by reanahub/reana-ui#348 or #64
Closed

improper progress indication for cached steps #62

giuseppe-steduto opened this issue Aug 29, 2023 · 0 comments · Fixed by reanahub/reana-ui#348 or #64
Assignees

Comments

@giuseppe-steduto
Copy link
Member

Snakemake has extensive out-of-the-box in-workflow caching capabilities: that is, it saves the results of intermediary steps in a cache directory, so that they can be reused by the same workflow if the inputs, code and parameters did not change.

In REANA, Snakemake's caching capabilities are natively supported if the workflow is re-run in the same workspace (e.g. with the reana-client restart command). If the results of one or more steps can be taken by Snakemake from its cache, REANA won't execute them. This is a nice and desired feature that could save much time to researchers when polishing their workflows.

However, there is a small problem when reporting the progress of a workflow whose results were partially or totally recovered from the cache: REANA does not consider the jobs that were cached as executed, even if the workflow is correctly marked as finished. This leads to some inconsistencies in how the workflow progress is displayed:
image
image

reana-client list --include-progress
NAME               RUN_NUMBER   CREATED               STARTED               ENDED                 STATUS     PROGRESS
roofit-snakemake   2.1          2023-08-29T12:55:39   2023-08-29T12:55:50   2023-08-29T12:56:01   finished   -/-

We should fix this either by marking the cached steps in the same way as the finished ones, or by somehow reporting that some or all the steps were restored from cache.

@giuseppe-steduto giuseppe-steduto changed the title error in workflow step reporting when using cached results improper progress indication for cached steps Aug 29, 2023
giuseppe-steduto added a commit to giuseppe-steduto/reana-ui that referenced this issue Sep 11, 2023
Changes the workflow progress bar to always show 100% for workflows that
are in status "finished", so that even when some steps are restored from
cache, the progress bar stays consistent with the status of the
workflow.

Closes reanahub/reana-workflow-engine-snakemake#62.
giuseppe-steduto added a commit to giuseppe-steduto/reana-workflow-engine-snakemake that referenced this issue Sep 14, 2023
Amend the total number of jobs in a workflow so that when Snakemake
detects that one or more jobs can be restored from the workspace (same
workflow caching) they are not considered in the total count, reflecting
Snakemake's local behaviour.

Closes reanahub#62.
giuseppe-steduto added a commit to giuseppe-steduto/reana-workflow-engine-snakemake that referenced this issue Sep 15, 2023
Amend the total number of jobs in a workflow so that when Snakemake
detects that one or more jobs can be restored from the workspace (same
workflow caching) they are not considered in the total count, reflecting
Snakemake's local behaviour.

Closes reanahub#62.
giuseppe-steduto added a commit to giuseppe-steduto/reana-workflow-engine-snakemake that referenced this issue Sep 19, 2023
Amend the total number of jobs in a workflow so that when Snakemake
detects that one or more jobs can be restored from the workspace (same
workflow caching) they are not considered in the total count, reflecting
Snakemake's local behaviour.

Closes reanahub#62.
giuseppe-steduto added a commit to giuseppe-steduto/reana-workflow-engine-snakemake that referenced this issue Sep 19, 2023
Amend the total number of jobs in a workflow so that when Snakemake
detects that one or more jobs can be restored from the workspace (same
workflow caching) they are not considered in the total count, reflecting
Snakemake's local behaviour.

Closes reanahub#62.
giuseppe-steduto added a commit to giuseppe-steduto/reana-ui that referenced this issue Sep 20, 2023
Changes the workflow progress bar to always show 100% for workflows that
are in status "finished", so that even when some steps are restored from
cache, the progress bar stays consistent with the status of the
workflow.

Closes reanahub/reana-workflow-engine-snakemake#62.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment