feat(nf-tower): Add workflow output dataset upload to Seqera Platform #6515
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for nextflow-docs-staging canceled.
|
edmundmiller
force-pushed
the
dataset-upload-sdk
branch
from
62fbf9d to
7158ef3
Compare
Implement automatic upload of Nextflow workflow output index files to
Seqera Platform datasets when workflows complete, enabling seamless
integration between Nextflow's output syntax and Platform's dataset
management.
Changes:
- Add DatasetConfig class for dataset upload configuration
- Support auto-create or use existing datasets
- Customizable dataset name patterns with variable substitution
- Per-output configuration overrides
- Update TowerConfig to include datasets configuration scope
- Implement dataset upload in TowerClient:
- Collect workflow outputs via onWorkflowOutput() callback
- Upload index files on workflow completion (onFlowComplete)
- Create datasets via Platform API with proper workspace URLs
- Use multipart/form-data for file uploads (matches tower-cli)
- Add URL builders for dataset API endpoints
- Add comprehensive unit tests for DatasetConfig
API Implementation:
- Create dataset: POST /workspaces/{id}/datasets/
- Upload file: POST /workspaces/{id}/datasets/{id}/upload
- Proper multipart/form-data format with file field
- Workspace ID in URL path (not query param)
- Header detection via ?header=true query parameter
Configuration example:
tower {
datasets {
enabled = true
createMode = 'auto'
namePattern = '${workflow.runName}-outputs'
perOutput {
'results' { datasetId = 'existing-id' }
}
}
}
Based on research of tower-cli (v0.15.0) and Seqera Platform API
documentation to ensure correct endpoint structure and payload format.
Signed-off-by: Edmund Miller <edmund.a.miller@gmail.com>
Signed-off-by: Edmund Miller <edmund.miller@seqera.io>
…et upload
Refactor dataset upload implementation to use the official tower-java-sdk
instead of manual HTTP multipart encoding, significantly simplifying the
code and improving maintainability.
Changes:
- Add tower-java-sdk dependency (1.43.1) with GitHub Packages repository
- Replace manual HTTP implementation with DatasetsApi SDK methods:
- createDataset() now uses datasetsApi.createDataset(wspId, request)
- uploadIndexToDataset() now uses datasetsApi.uploadDataset(wspId, id, header, file)
- Remove ~120 lines of manual HTTP code:
- Deleted getUrlDatasets() and getUrlDatasetUpload() URL builders
- Deleted uploadFile() multipart HTTP request construction
- Deleted createMultipartBody() RFC 2388 multipart encoding
- Add comprehensive test coverage:
- 7 unit tests with mocked DatasetsApi (initialization, event collection,
dataset creation, file upload, exception handling)
- 1 integration test with real Platform API (conditional on TOWER_ACCESS_TOKEN)
- Manual test workflow in test-dataset-upload/ directory with documentation
Testing:
- All unit tests passing (BUILD SUCCESSFUL)
- Integration test ready (runs when TOWER_ACCESS_TOKEN available)
- Test workflow provides end-to-end validation guide
Benefits:
- Uses official Seqera SDK (same as tower-cli)
- Easier to test with mocked API
- SDK handles all HTTP/multipart details automatically
- Bug fixes in SDK benefit us automatically
- Code reduced from ~300 lines to ~100 lines
Note: Requires GitHub credentials for tower-java-sdk dependency.
Configure github_username and github_access_token in gradle.properties
or set GITHUB_USERNAME and GITHUB_TOKEN environment variables.
Signed-off-by: Edmund Miller <edmund.a.miller@gmail.com>
Signed-off-by: Edmund Miller <edmund.miller@seqera.io>
The tower-java-sdk dependency from GitHub Packages requires authentication even for public packages, causing CI build failures. This reverts the SDK refactoring and restores the manual HTTP implementation. Changes: - Removed tower-java-sdk dependency from build.gradle - Restored manual HTTP methods in TowerClient.groovy: - getUrlDatasets() and getUrlDatasetUpload() URL helpers - createDataset() with JSON payload and sendHttpMessage() - uploadFile() multipart HTTP implementation - createMultipartBody() RFC 2388 implementation (~120 lines total) - Simplified TowerClientTest.groovy to remove SDK-specific tests - Kept core functionality tests and integration test Functionality remains identical - only the implementation approach changed from SDK calls to direct HTTP requests. This allows the plugin to build successfully in CI without requiring GitHub Package authentication. Signed-off-by: Edmund Miller <edmund.miller@seqera.io>
edmundmiller
force-pushed
the
dataset-upload-sdk
branch
from
7158ef3 to
687b0c3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add automatic upload of workflow outputs to Seqera Platform datasets, enabling seamless integration between Nextflow workflows and Platform's dataset management features.
Features
1. Automatic Dataset Creation & Upload
2. Manual HTTP Implementation
3. Configuration Options
tower { datasets { enabled = true createMode = 'auto' // or 'existing' namePattern = '${workflow.runName}-outputs' perOutput { 'my_output' { enabled = true datasetId = 'existing-dataset-id' // optional } } } }Implementation Details
Dataset Upload Flow:
WorkflowOutputEventevents viaTraceObserverV2HTTP Implementation:
createDataset()- POST JSON payload to datasets APIuploadFile()- Multipart/form-data file upload per RFC 2388createMultipartBody()- Manual multipart encodingHttpClientfrom TowerClient infrastructureConfiguration Classes:
DatasetConfig- Main configuration with validationTesting
Unit Tests (3 tests)
Integration Test
TOWER_ACCESS_TOKEN)Validation Workflow
validation/dataset-upload/Documentation
DatasetConfigoutput {}block supportArchitecture Decision
Why Manual HTTP vs tower-java-sdk?
Initially refactored to use
tower-java-sdk, but reverted because:The manual implementation maintains identical functionality while ensuring CI compatibility.
Breaking Changes
None - feature is opt-in via configuration.
Related Issues
Checklist