Issue/5004

[markjschreiber]

Addresses issue 5004 and possibly several others with the AWS Batch backend. The changes allow for re-use of job definitions to prevent eventual consistency collisions. Other changes are largely simplifications or improvements to the backend that simplify the integration with AWS.

1. Added additional documentation
2. Added additional server logging which is useful for diagnostics and has been requested by users
3. Updated cromwell-aws-s3filesystem package to use AWS SDKv2 removing the need for two versions of the SDK
4. Removed the requirement for a proxy container is the AWS worker
5. Removed the need for a custom AMI for the EC2 workers
6. Removed the need for a custom ECS agent
7. Set up /var/lib/docker/docker to auto-expand as inputs are now read directly into the container

[rdavison]

@markjschreiber Thank you for this PR. I had a quick look at it and it looks pretty good. I just had a few questions.

7. Set up /var/lib/docker/docker to auto-expand as inputs are now read directly into the container

Is this the only documentation on this requirement? Also, are you saying that that directory is now being auto-expanded in the underlying ECS image, or that a client needs to create an AMI to auto-expand that directory instead of /cromwell_root? Also, is it /var/lib/docker/docker or /var/lib/docker/containers?

EDIT:

The README.md references a LaunchTemplate which provides a UserData script to an underlying AMI, but it's not linked anywhere. I think I've tracked it down to the document here: https://github.com/aws-samples/aws-genomics-workflows/blob/master/src/templates/aws-genomics-launch-template.template.yaml

[markjschreiber]

The EC2 workers contain a script that automatically expands that mount users don't need to set that up. No custom AMI is required, in theory any AMI that can work with ECS could be used.

…

On Fri, Jul 17, 2020 at 7:35 PM Richard Davison ***@***.***> wrote: @markjschreiber <https://github.com/markjschreiber> 1. Set up /var/lib/docker/docker to auto-expand as inputs are now read directly into the container Is this the only documentation on this requirement? Also, are you saying that that directory is now being auto-expanded in the underlying ECS image, or that a client needs to create an AMI to auto-expand that directory instead of /cromwell_root? Also, is it /var/lib/docker/docker or /var/lib/docker/containers? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5468 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2E6EKVSIMWREHDRE7WZ43R4DN3TANCNFSM4LW5UP3A> .

[rdavison]

@markjschreiber

The EC2 workers contain a script that automatically expands that mount
users don't need to set that up. No custom AMI is required, in theory any
AMI that can work with ECS could be used.

Is this a new feature of all new ECS Optimized Amazon Linux instances?

[markjschreiber]

No. The feature is due to a script that monitors disk storage and mounts new disks into a btrfs filesystem. This isn’t standard for ECS.

…

On Tue, Jul 21, 2020 at 4:08 PM Richard Davison ***@***.***> wrote: @markjschreiber <https://github.com/markjschreiber> The EC2 workers contain a script that automatically expands that mount users don't need to set that up. No custom AMI is required, in theory any AMI that can work with ECS could be used. Is this a new feature of all new ECS Optimized Amazon Linux instances? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5468 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2E6EM4MI3WWXI2AFZAK2LR4XYUHANCNFSM4LW5UP3A> .

[rdavison]

Sorry if I'm still not understanding. Is that script provided when you run cromwell? How are you supposed to get that script? I'm assuming you're referring to the EBS autoscaler?

[rdavison]

@markjschreiber : Oh! I think maybe I understand where the misunderstanding is. Could it be you're implying that the Cromwell server is correctly provisioning the worker nodes on behalf of the user?

[markjschreiber]

When you submit a job to Cromwell with AWS Batch configured as the backend AWS Batch will deploy workers via ECS. Those workers will be deployed according to a launch template that configures the EC2 workers including downloading and starting the EBS auto expander on the EC2 as part of its startup. So it’s standard for the ECS cluster/ Batch compute environment used by Cromwell but not standard for any other ECS node.

…

On Wed, Jul 22, 2020 at 4:04 PM Richard Davison ***@***.***> wrote: @markjschreiber <https://github.com/markjschreiber> : Oh! I think maybe I understand where the misunderstanding is. Could it be you're implying that the Cromwell *server* is correctly provisioning the worker nodes on behalf of the user? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5468 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2E6EKOXJKQMVJWYBPRYK3R45A6FANCNFSM4LW5UP3A> .