Module update removes registered runner but does not create a new runner

[jonpas]

Describe the bug

Updating GitLab Runner version by re-deploying a Terraform project using this module with the new "Next GitLab Runner Token Architecture", as the new EC2 instance is created, GitLab Runner is removed from the GitLab instance, but a new one is not registered.

Before "Next GitLab Runner Token Architecture", the GitLab Runner was simply updated.

To Reproduce

Steps to reproduce the behavior:

1. Update runner_gitlab = { runner_version = "16.7.0" } to a newer version (eg. 16.8.0)
2. terraform plan + terraform apply
3. Observe new master runner instance being created on EC2 and old one destroyed
4. Observe runner being deleted from GitLab instance
5. Wait and observe no new runner being created on the GitLab instance
6. Have no runner

Expected behavior

A GitLab Runner to either be recreated or re-use the existing runner registration.

Additional context

Upgrade to "Next GitLab Runner Token Architecture" was performed with a completely fresh setup, no old SSM parameters remained. We deploy using and GitLab's Terraform state handling through GitLab's CI (running on a instance completely separated from any auto-deployed ones from this module).

We have to fully destroy the runner and recreate it to get the runner re-registered. EC2 system logs show nothing different from the past, but I can supply a log if no other ideas are available.

[kayman-mk]

Having old SSM parameters with old tokens is usually the problem, yes. But this shouldn't be the case here.

What you should observe in the logs is, that a new token is fetched from GitLab and stored as SSM parameter. The Runner should be shown in GitLab.

Just checked my configuration

  runner_gitlab = {
    # this is the token used for the API call allowing the Runner to register
    access_token_secure_parameter_store_name = var.gitlab_group_access_token_ssm_name
  }

  runner_gitlab_registration_config = {
   # needed for the new registration process
   type              = "group" # instance or group or project
    group_id          = "1234"
  }

The Runner creates a SSM parameter holding the glrt- token returned in the registration process. Is the SSM parameter present and holding the token?

[jonpas]

There are multiple SSM parameter tokens created:

• *-access-token holding the glpat-* token (GitLab Access Token for registering Runners)
• *-<name>-runner-token holding the glrt- token (no description)
• *-<name>-sentry-dsn holding null (no description)

My configuration is similar except it's an instance runner.

  runner_gitlab = {
    runner_version                           = "16.8.0"
    url                                      = "https://gitlab...."
    access_token_secure_parameter_store_name = "gitlab-runner-access-token"
  }

  runner_gitlab_registration_config = {
    type              = "instance"
    tag_list          = "tag1,tag2"
    description       = "name"
    locked_to_project = "false"
    run_untagged      = "true"
    maximum_timeout   = "3600"
  }

Anyways, after updating to 7.3.1, it has re-registered itself correctly first try. I wonder if #1058 fixed the issue in this case.

[jonpas]

This has happened again with simple update of VPC module to 5.7.0 and GitLab Runner version to 16.10.0, no update to this project.

SSM Parameter Store shows all tokens, with *-<name>-runner-token being updated at the time of update.

EC2 System Log shows nothing about fetching a new token though.

---

Update to 7.4.0 also didn't have any issues, so maybe this is only happening when not updating this project?

[lpsm-nuageit]

is there any way of setting this up without having to register a runner? i already have my runner created in the visual interface

[kayman-mk]

Yes, that is possible. We added a variable in the last release.

There is also a PR ongoing to allow zero downtime updates.