AWS ECS Managed Instances executor [POC]

[pditommaso]

Summary

This PR introduces a new awsecs executor for running Nextflow tasks on AWS ECS Managed Instances.

Why ECS Managed Instances?

AWS ECS Managed Instances provides fine-grained control over EC2 instance types while offloading infrastructure management to AWS. This offers significant advantages over Fargate for scientific workflows:

Key Benefits

Capability	ECS Managed Instances	Fargate
Instance Type Control	✅ Full control - select specific instance types, GPUs, network-optimized instances	❌ No control - AWS selects instances
GPU Support	✅ Full GPU access (T4, V100, A10G, etc.)	❌ Not supported
Max vCPUs	Up to instance limits	16 vCPUs
Max Memory	Up to instance limits	120 GB
Ephemeral Storage	30 GiB - 16 TiB	21 - 200 GB
Privileged Containers	✅ SYS_ADMIN, NET_ADMIN, BPF	❌ Limited
Cost Model	EC2 pricing + management fee (multi-task per instance)	Per-task pricing
Spot Support	✅ EC2 Spot instances	✅ Fargate Spot

Ideal Use Cases

• GPU-accelerated workloads (ML training, inference)
• High-memory tasks exceeding Fargate's 120 GB limit
• Large ephemeral storage needs (up to 16 TiB)
• Cost optimization via instance type selection and multi-task packing
• Privileged operations requiring Linux capabilities (e.g., FUSE mounts for Fusion filesystem)

Features

• Compute Resources: CPUs, memory, GPUs, disk size (30 GiB - 16 TiB), optional EC2 instance type
• Storage Strategy: AWS S3 work directory via Seqera Fusion filesystem
• Resilience: Automatic spot instance interruption retry (configurable maxSpotAttempts)
• Observability: CloudWatch Logs integration for task output
• FUSE Support: Linux capabilities (SYS_ADMIN) and /dev/fuse device mapping for Fusion filesystem

Requirements

Pre-configured Infrastructure

• ECS cluster with Managed Instances capacity provider
• IAM execution role for ECS tasks (image pull, CloudWatch logs)
• VPC with subnets and security groups (auto-discovered from default VPC if not specified)
• S3 bucket for work directory

Configuration

process {
    executor = 'awsecs'
    container = 'ubuntu:latest'
    cpus = 4
    memory = '16 GB'
    disk = '100 GB'
    // accelerator 1  // GPU support
    // machineType = 'c6i.2xlarge'  // Optional: pin to specific instance type
}

aws {
    region = 'us-east-1'
    ecs {
        cluster = 'my-cluster'  // Required
        executionRole = 'arn:aws:iam::123456789:role/ecsTaskExecutionRole'  // Required
        // Optional settings with defaults:
        // taskRole = 'arn:aws:iam::...'
        // subnets = ['subnet-...']  // Auto-discovered from default VPC
        // securityGroups = ['sg-...']  // Auto-discovered from default VPC
        // logsGroup = '/aws/ecs/nextflow'
        // assignPublicIp = true
        // maxSpotAttempts = 5
    }
}

fusion.enabled = true
wave.enabled = true

Limitations

• 14-day lifecycle: ECS Managed Instances drain after 14 days (tasks exceeding this limit will fail with clear error)
• Fusion required: S3 work directory via Fusion filesystem is mandatory
• No custom AMIs: Uses AWS-managed Bottlerocket OS

Implementation Status

Phase	Status	Progress
Phase 1: Setup	✅ Complete	4/4
Phase 2: Foundational	✅ Complete	6/6
Phase 3: Basic Task Execution (MVP)	✅ Complete	18/18
Phase 4: GPU Support	⬜ Not Started	0/5
Phase 5: Custom Storage	🟡 In Progress	4/5
Phase 6: Instance Type Selection	⬜ Not Started	0/4
Phase 7: Monitoring & Debugging	🟡 In Progress	4/15
Phase 8: Polish	🟡 In Progress	1/6

Overall: 37/63 tasks (59%)

Documentation

• Feature specification: specs/001-ecs-executor/spec.md
• Implementation plan: specs/001-ecs-executor/plan.md
• Task breakdown: specs/001-ecs-executor/tasks.md

References

• AWS Blog: Announcing Amazon ECS Managed Instances
• AWS Docs: ECS Managed Instances
• ECS Managed Instances FAQs

---

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	b107f3f
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/69547112153e6c0008347061