Uploading Workflow Artifacts Concurrently in Worker Jobs

[ethany-nv]

What new feature or improvement would you like to see?

For workflows, our worker uploads workflow artifacts (workflow spec, Kubernetes spec, workflow logs) synchronously in jobs. As the workflow scales to hundreds of tasks, and the number of workflow scales, this becomes very inefficient.

For UploadWorkflowFiles, this job should leverage multi-threaded upload to maximize the throughput for the upload.

For CleanupWorkflow, this job should concurrently migrate logs from redis and upload to cloud storage.

Why is this needed?

This would increase the performance of worker operations.

Code of Conduct

• I agree to follow NVIDIA OSMO's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!