instance_type: you don't have access to t3.medium instances in eu-north-1 - error when deploying

[AbbeKamalov]

I'm trying to launch a cluster using cortex but I'm gettiing following error:

cortex cluster up

cluster name [cortex]: mycluster

aws region [us-east-1]: eu-north-1

using aws access key id ****XSBA and aws secret access key ****KeJE from cache (to use different credentials, specify the --aws-key and --aws-secret flags)

aws instance type [m5.large]: t3.medium

min instances [1]: 1

max instances [5]: 5

verifying your configuration ...

error: instance_type: you don't have access to t3.medium instances in eu-north-1; please request access in the appropriate region (https://console.aws.amazon.com/support/cases#/create?issueType=service-limit-increase&limitType=ec2-instances). If you submitted a request and it was recently approved, please allow ~30 minutes for AWS to reflect this change."

cluster configuration schema can be found here: https://docs.cortex.dev/v/0.20/cluster-management/config

I've requested and have gotten service limit increase (even though I'm just trying to launch 1 instance), but that didn't solve the problem, even after few hours. I've checked and T3.MEDIUM instances are in fact available in EU-NORTH-1 region. How can this be solved?

[deliahu]

@MUTAL thanks for reaching out.

It seems that there is a bug/inconsistency in the AWS service quotas API (which cortex uses behind the scenes) in eu-north-1, but I would like to confirm on your end too.

For me, when I check my AWS EC2 dashboard in eu-north-1, I see that I have 5 vCPUs of quota for "standard" instances:

However, when I run this command, it comes up empty:

$ aws service-quotas list-service-quotas --service-code ec2 --region eu-north-1 | grep -B 14 -A 7 "Standard/OnDemand"

For comparison, in us-west-2, I see that I have 1280 vCPU:

Which agrees with what the CLI command says:

$ aws service-quotas list-service-quotas --service-code ec2 --region us-west-2 | grep -B 14 -A 7 Standard/OnDemand

        {
            "ServiceCode": "ec2",
            "ServiceName": "Amazon Elastic Compute Cloud (Amazon EC2)",
            "QuotaArn": "arn:aws:servicequotas:us-west-2:764403040460:ec2/L-1216C47A",
            "QuotaCode": "L-1216C47A",
            "QuotaName": "Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances",
            "Value": 1280.0,
            "Unit": "None",
            "Adjustable": true,
            "GlobalQuota": false,
            "UsageMetric": {
                "MetricNamespace": "AWS/Usage",
                "MetricName": "ResourceCount",
                "MetricDimensions": {
                    "Class": "Standard/OnDemand",
                    "Resource": "vCPU",
                    "Service": "EC2",
                    "Type": "Resource"
                },
                "MetricStatisticRecommendation": "Maximum"
            }
        },

Can you confirm that you notice the same discrepancy on your account?

If so, one workaround could be to try a different region, for example eu-central-1 (which I have used before, so I expect will work)

[AbbeKamalov]

@MUTAL thanks for reaching out.

It seems that there is a bug/inconsistency in the AWS service quotas API (which cortex uses behind the scenes) in eu-north-1, but I would like to confirm on your end too.

For me, when I check my AWS EC2 dashboard in eu-north-1, I see that I have 5 vCPUs of quota for "standard" instances:

However, when I run this command, it comes up empty:

$ aws service-quotas list-service-quotas --service-code ec2 --region eu-north-1 | grep -B 14 -A 7 "Standard/OnDemand"

For comparison, in us-west-2, I see that I have 1280 vCPU:

Which agrees with what the CLI command says:

$ aws service-quotas list-service-quotas --service-code ec2 --region us-west-2 | grep -B 14 -A 7 Standard/OnDemand

        {
            "ServiceCode": "ec2",
            "ServiceName": "Amazon Elastic Compute Cloud (Amazon EC2)",
            "QuotaArn": "arn:aws:servicequotas:us-west-2:764403040460:ec2/L-1216C47A",
            "QuotaCode": "L-1216C47A",
            "QuotaName": "Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances",
            "Value": 1280.0,
            "Unit": "None",
            "Adjustable": true,
            "GlobalQuota": false,
            "UsageMetric": {
                "MetricNamespace": "AWS/Usage",
                "MetricName": "ResourceCount",
                "MetricDimensions": {
                    "Class": "Standard/OnDemand",
                    "Resource": "vCPU",
                    "Service": "EC2",
                    "Type": "Resource"
                },
                "MetricStatisticRecommendation": "Maximum"
            }
        },

Can you confirm that you notice the same discrepancy on your account?

If so, one workaround could be to try a different region, for example eu-central-1 (which I have used before, so I expect will work)

I tried eu-central-1 but it threw the same error but us-east-1 worked. I wonder why could that be?

[deliahu]

@MUTAL I'm glad to hear us-east-1 is working for you.

Regarding eu-central-1, this should also work, so it would be good to figure out why it's not. Do you mind checking your service quota for the Standard/OnDemand in eu-central-1? You can do this from the AWS console by changing your region, clicking on "Limits" in the EC2 dashboard, and then searching for "standard" (see above for a screen shot). You can also check it by running this command:

aws service-quotas list-service-quotas --service-code ec2 --region eu-central-1 | grep -B 14 -A 7 Standard/OnDemand

If the value is 0, then the error that cortex shows is expected, but it it's not 0, then there is some other issue which we can look into.

[AbbeKamalov]

I checked via EC2 dashboard and it says that my limit for ALL Standard Spot instance is 36vCPUs and for On-Demand All Standard the limit is 42vCPUs.

…

On Wed, 14 Oct 2020 at 19:59, David Eliahu ***@***.***> wrote: @MUTAL <https://github.com/mutal> I'm glad to hear us-east-1 is working for you. Regarding eu-central-1, this should also work, so it would be good to figure out why it's not. Do you mind checking your service quota for the Standard/OnDemand in eu-central-1? You can do this from the AWS console by changing your region, clicking on "Limits" in the EC2 dashboard, and then searching for "standard" (see above for a screen shot). You can also check it by running this command: aws service-quotas list-service-quotas --service-code ec2 --region eu-central-1 | grep -B 14 -A 7 Standard/OnDemand If the value is 0, then the error that cortex shows is expected, but it it's not 0, then there is some other issue which we can look into. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1438 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFOXIRC5YOMOPPIKUHGGATSKXRJLANCNFSM4SPMIFPA> .

[deliahu]

Thanks. Do you mind sending the output of:

aws service-quotas list-service-quotas --service-code ec2 --region eu-central-1 | grep -B 14 -A 7 Standard/OnDemand

[deliahu]

@MUTAL I just wanted to give you a heads up that this fix got released with v0.21.0 today. Please let us know if you continue to run into issues on eu-north-1, or if you’d like us to take a look at the issue you were having on eu-central-1.