Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Resume functionality breaks between builds of nextflow components. #708

Open
2 tasks done
DriesSchaumont opened this issue May 22, 2024 · 2 comments
Open
2 tasks done
Labels
bug Something isn't working

Comments

@DriesSchaumont
Copy link
Contributor

DriesSchaumont commented May 22, 2024

What happened?

Building a new nextflow component always breaks the -resume functionality. This is because the container creation time is added as a LABEL to the image manifest.

val created = List(s"""org.opencontainers.image.created="$opencontainers_image_created"""")

The docker images is one of the sources for the hash that nextflow uses in order to trigger resume events.

Steps to reproduce

/

Expected behavior

The -resume functionality works between builds

Relevant log output

No response

Version

/

Possible solution

I would like to propose to remove this label from the docker images.
Personally I rarely used this functionality, and one could argue that the upload time metadata from the container registry is a close enough proxy for this info.

Confirmation

  • I have searched the existing issues to make sure this is not a duplicate.
  • I have provided clear and concise information about the bug.

Additional context

No response

@DriesSchaumont DriesSchaumont added the bug Something isn't working label May 22, 2024
@DriesSchaumont
Copy link
Contributor Author

Update: turns out that the docker label is not the root cause of this issue. After looking at some hashes with -dump-hashes, it turns out that the whole target folder is added as input for a task because we need the resources. One of items is the strict viash config, which includes metadata that is always updated each build. Note that a component might do introspection on the viash config in order to do its function.

| tuple val(id)$inputPaths, val(args), path(resourcesDir, stageAs: ".viash_meta_resources")

@DriesSchaumont
Copy link
Contributor Author

A workaround for this issue is to add process.cache = "lenient" to the nextflow config. This is because the git_commit and git_tag properties in the build config file are always the same length, and thus the file size of the config does not change. lenient only takes into account the file name and file size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant