GHES : http error 999 during scale up

[m8t88]

Hi all,

we are using GHES 3.10.2 and I tried to get this repo v5.6.1 up and running for the first time. We had an older version of this repos back from july 2022 in place and were thinking to update to the newest version.

We configured a multi-runner with ephemeral org runners and self-build AMI with the following yaml:

matcherConfig:
  exactMatch: true
  labelMatchers:
    - [ self-hosted, linux, x64, ubuntu-2004, 8xl ]
    - [ gha-ubuntu20-8xl ]
  priority: 1 # set ephemeral runner priority to 1
fifo: true
redrive_build_queue:
  enabled: false
  maxReceiveCount: null
runner_config:
  runner_os: linux
  runner_architecture: x64
  runner_run_as: ubuntu
  runner_name_prefix: ubuntu-2004-x64_
  enable_ssm_on_runners: true
  instance_types:
    - c7i.8xlarge
    - m7i.8xlarge
    - m7i-flex.8xlarge
    - m7a.8xlarge
    - c6a.8xlarge
    - c6i.8xlarge
    - m6i.8xlarge
    - m6a.8xlarge
  runners_maximum_count: 1
  enable_ephemeral_runners: true
  enable_organization_runners: true
  enable_job_queued_check: true
  enable_jit_config: false
  enable_on_demand_failover_for_errors: ['InsufficientInstanceCapacity']
  create_service_linked_role_spot: true
  delay_webhook_event: 0
  scale_down_schedule_expression: cron(* * * * ? *)
  runner_metadata_options:
    instance_metadata_tags: disabled
    http_endpoint: enabled
    http_tokens: optional
    http_put_response_hop_limit: 1
  ami_filter:
    name:
      - gha-ubuntu20-runner-amd64-*
    state:
      - available
  ami_owners:
    - "abc"

Before we got an 401 when the upscaler tried to run. Were able to fix that by explicitly setting the GHES URL
ghes_url = "https://our ghes instance"

We are now stuck when the scale up lambda tries to spawn a new instance, it throws a 999 and I have no idea what to check next. CloudWatch shows:

{
    "level": "DEBUG",
    "message": "GHES API URL: https://ou ghes instance/api/v3",
    "service": "runners-scale-up",
    "timestamp": "2024-01-12T10:54:55.193Z",
    "xray_trace_id": "1-65a11a7d-1b109a1647a95e3ef73938cc",
    "module": "gh-auth",
    "region": "eu-central-1",
    "environment": "prod-linux-x64-ubuntu-8xl",
    "aws-request-id": "5603949c-27dd-592a-9ac1-ca873b773cbc",
    "function-name": "prod-linux-x64-ubuntu-8xl-scale-up",
    "runner": {
        "type": "Org",
        "owner": "abc",
        "namePrefix": "ubuntu-2004-x64_"
    },
    "github": {
        "event": "workflow_job",
        "workflow_job_id": "9317507"
    }
}

{
    "level": "WARN",
    "message": "Ignoring error: Request failed with status code 999",
    "service": "runners-scale-up",
    "timestamp": "2024-01-12T10:54:55.340Z",
    "xray_trace_id": "1-65a11a7d-1b109a1647a95e3ef73938cc",
    "region": "eu-central-1",
    "environment": "prod-linux-x64-ubuntu-8xl",
    "aws-request-id": "5603949c-27dd-592a-9ac1-ca873b773cbc",
    "function-name": "prod-linux-x64-ubuntu-8xl-scale-up",
    "module": "lambda.ts"
}

Neither the syncer nor the webhook lambdas are throwing any errors.

I am kind of lost. Does anyone know what might happen and where I could check next?