Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow outputs (third preview) #5909

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

Conversation

bentsherman
Copy link
Member

@bentsherman bentsherman commented Mar 20, 2025

This PR implements the third preview of the workflow output definition for Nextflow 25.04.

Changes are described in the docs, copied here for convenience:

  • The publish: section can only be specified in the entry workflow.

  • Workflow outputs in the publish: section are assigned instead of using the >> operator. The output name must be a valid identifier.

  • By default, output files are published to the base output directory, rather than a subdirectory corresponding to the output name.

  • The syntax for dynamic publish paths has changed. Instead of defining a closure that returns a closure with the path directive, the outer closure should use the >> operator to publish individual files.

  • The mapper index directive has been removed. Use a map operator in the workflow body instead.

Changes not described in the docs:

  • The onWorkflowPublish trace event has been modified to include the output name. It is emitted once for each output, rather than once for each channel value, and it is emitted regardless of whether the index file is enabled.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman requested review from a team as code owners March 20, 2025 21:44
@bentsherman bentsherman requested a review from pditommaso March 20, 2025 21:44
Copy link

netlify bot commented Mar 20, 2025

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 14c9c11
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/67eab5326c39390008d0d143
😎 Deploy Preview https://deploy-preview-5909--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
*/
void onWorkflowPublish(Object value){}
void onWorkflowPublish(String name, Object value){}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pditommaso regarding this change to onWorkflowPublish

Aside from the fact that it's a preview feature, there is a bigger issue I wanted to raise about the TraceObserver -- these overloads that we add to be backwards compatible don't actually work

Even if we add something like this:

void onWorkflowPublish(Object value) {
    onWorkflowPublish(null, value)
}

If I build a plugin with Nextflow 24.10 and try to use it with 25.04, it will fail with an error like this:

ERROR ~ Receiver class nextflow.validation.ValidationObserver does not define or inherit an implementation of the resolved method 'abstract void onWorkflowPublish(java.lang.String,java.lang.Object)' of interface nextflow.trace.TraceObserver.

This is because the custom trace observer gets compiled against the 24.10 version of TraceObserver and run against the 25.04 version at runtime, but it doesn't receive the new method overload. I think it's a limitation of the Java/Groovy runtime.

So there is no point in adding these extra overloads. Plugins built with an older Nextflow will break no matter what we do

See: nextflow-io/nf-schema#90

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

In a channel that is directed to a publish target, Files listed inside a Map object are not published.
1 participant