Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EC2 Task Runner #503

Open
anna-geller opened this issue Jul 29, 2024 · 3 comments
Open

Add EC2 Task Runner #503

anna-geller opened this issue Jul 29, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@anna-geller
Copy link
Member

Feature description

We should add support for running tasks on Amazon EC2 instances directly (without Batch).

The user will need to specify (we can set sensible defaults e.g. to use Ubuntu AMI and query private subnet ID for the default VPC):

  • the containerImage
  • instanceType e.g. t2.micro
  • imageId - AMI e.g. ami-0abcdef1234567890
  • subnetId
  • list of tags for cost tracking

Kestra should:

  • output insensitive instance details like confirmation of the InstanceId, instanceType etc.
  • capture execution logs
  • terminate the instance after the task finishes and confirm that in the logs
@anna-geller anna-geller added the enhancement New feature or request label Jul 29, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Issues Jul 29, 2024
@anna-geller
Copy link
Member Author

Note that if the user chooses an instance type without Docker preinstalled, we can't run it as a container.

We should then log a friendly error message: "Your chosen AMI doesn't support Docker runtime. Please choose a different AMI for your instance. See the Amazon EC2 documentation for more details."

@myron-overton
Copy link

myron-overton commented Jul 29, 2024

You could have the server alway choose the image. Because this type of task is an extension of the current server and not arbitrary task running. Therefore your build system would have to generate an API for of this extension on each release.

@anna-geller
Copy link
Member Author

no no, actually we ideally want to enable running arbitrary commands -- for an extension of the current server, we have a feature called worker groups

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

2 participants