Gov Cloud AMI

[haviland-nh]

Hello,

Our team is looking to use CML, but can't execute the AWS provider because we're working out of gov-cloud-west-1 and the AMI isn't available. Could we help you to bring it into the region?
Appreciated,

Nearly Human Team

[haviland-nh]

Or if there is a better way to do that, please let us know - happy to work with you.

[dacbd]

It looks to be partially implemented, the option to specify an image/ami exists on the runner resource here(the underlying system)
but the option isn't exposed yet via cml?

[0x2b3bfa0]

👋🏼 Hello! We don't have access to AWS GovCloud regions, but you can just copy our AMI to your region and CML will should detect it automatically.

aws ec2 copy-image \
  --region gov-cloud-west-1 \
  --source-region us-east-1 \
  --source-image-id ami-0a8e187584f56101e
  --name iterative-cml

[0x2b3bfa0]

It looks to be partially implemented, the option to specify an image/ami exists on the runner resource here(the underlying system)
but the option isn't exposed yet via cml?

CML searches for the default image using the name, so it should be enough to copy the AMI from a public region. 🙈

[0x2b3bfa0]

can't execute the AWS provider

We're supposed to fallback to the official Ubuntu 18.04 AMI if the CML one is not available. Didn't it work automatically? 🤔

[haviland-nh]

Ok thank you, we will give that a try on the AMI copy! I'll check our github action just to make sure, but I don't think it did. Is maybe the image search a different result in gov cloud? I'll get back to you with this tomorrow! Thank you again for looking into it.

…

________________________________ From: Helio Machado ***@***.***> Sent: Wednesday, February 16, 2022 8:42:24 PM To: iterative/cml ***@***.***> Cc: Karl Haviland ***@***.***>; Author ***@***.***> Subject: Re: [iterative/cml] Gov Cloud AMI (Issue #895) can't execute the AWS provider We're supposed to fallback to the official Ubuntu 18.04 AMI if the CML one is not available. Didn't it work automatically? 🤔 — Reply to this email directly, view it on GitHub<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fiterative%2Fcml%2Fissues%2F895%23issuecomment-1042483599&data=04%7C01%7C%7Cd23e1e26198842c6de9508d9f1b6bf50%7C690031e043d74af6af72d094c1bb6f59%7C1%7C0%7C637806589483010666%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=coDokNQAHF31%2FWjB2LObLSJ%2BnWLcc0FGi7FBG%2B4oGiQ%3D&reserved=0>, or unsubscribe<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQVQQFMDD3D4RTDZXNOVKU3U3RG77ANCNFSM5OSP3XYQ&data=04%7C01%7C%7Cd23e1e26198842c6de9508d9f1b6bf50%7C690031e043d74af6af72d094c1bb6f59%7C1%7C0%7C637806589483010666%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0tfcx3sn5ERog3RKovWbQtAX7BIdCWE5v%2FMFO8%2FPQdo%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[haviland-nh]

Hello just a few things after attempting to try these items out

• I can't copy the AMI into the region us-gov-west-1. The aws cli outputs the error below

aws ec2 copy-image --region us-gov-west-1 --source-region us-east-1   --source-image-id ami-0a8e187584f56101e   --name iterative-cml

An error occurred (InvalidRequest) when calling the CopyImage operation: Copy image not allowed from specified region.

• Ubuntu 18 can't fall back because the image isn't being found. I believe the owner id is different.

aws ec2 describe-images --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-bionic-18.04*"
{
    "Images": [
        {
            "Architecture": "arm64",
            "CreationDate": "2021-06-04T21:39:19.000Z",
            "ImageId": "ami-001a1191bd4ab2d1a",
            "ImageLocation": "aws-marketplace/ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-arm64-server-20210604-10d15682-9d0b-41dd-93c2-5e7de4f74165",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "345084742485",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",

• Our runner is hitting the error
  Creating...\n\n\t╷\n│ Error: Failed creating the machine: Nor iterative-cml nor Ubuntu Server 18.04 are available in your region

[dacbd]

You could also try making your own ami from an Ubuntu instance and name it “iterative-cml” maybe that would work?

[haviland-nh]

Oh interesting idea! I will try that.

[haviland-nh]

That worked, just a heads up to those who might run into this, we needed to copy without an encrypted snapshot, but we successfully ran a training after following that last idea @dacbd, thank you!

Just to tie a bow in this discussion, in our trial and error, we attempted to use our existing EKS cluster with the kubernetes provider implementation, which is a nice option and maybe even preferable longer term. The documentation on the mechanics for Kubernetes is a bit small and we were wondering where to go to ask more about this topic?
Thanks again,

Nearly Human team

[0x2b3bfa0]

The documentation on the mechanics for Kubernetes is a bit small and we were wondering where to go to ask more about this topic?

Yes, documentation is still a bit scarce. 🙈 Feel free to ask in the #cml channel on our Discord server.

[dacbd]

I feel like the only remaining action point is some documentation on using cml runner with k8s iterative/cml.dev#343