Self-hosted runners disappeared

[BrightRan]

Associated GitHub Community topic: https://github.saobby.my.eu.orgmunity/t/disappearing-self-hosted-runners/137669

The customer has added some self-hosted runners for his repository, but the runners would completely disappear as if he never added any.
When he refreshes, the runners would come back. Some would be Offline but would go back to being Idle after another refresh. Other times when he refreshes the runners disappear again.
When the customer logs into the runner machines to check their status, he can see a lot of connection retries.

2020-10-13 21:12:44Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
2020-10-13 21:14:42Z: Runner reconnected.
2020-10-13 21:15:42Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.

[chingc]

@BrightRan Thank you for creating this ticket. I'd like to add that I'm using v2.273.5 of the runner on a plain Amazon Linux 2 EC2 Instance. I haven't experienced any issues yesterday so perhaps it was an intermittent issue on GitHub or Amazon's end.

[brandan-schmitz]

I have see this issue as well using the save version as @chingc. Today I received an error from my build actions for a project, seems that the runner vanished from the project and left me with no runners registered. Opening the runner itself on my server it showed that the runner was still registered but was getting an unknown disconnect error from GitHub, and that was all that it would do was loop between restarting the runner service to saying it received an unknown disconnect error back to starting the runner service.

I have since wiped the old runner and re-downloaded it and registered it back to the server again as I needed the build pipeline running but not sure when it first occurred on my system.

[jonnikim]

This still seems to be an issue.

We had 5 runners, 4 of them were offline and 1 was idle. I disabled Actions to fix some syntax. Left it alone for ~2+ weeks and came back to see that only the idle one was remaining. The other 4 looks to have been deleted. Re-enabling Actions didn't bring them back either.

I still have the directories for the other 4 runners, but trying to start them throws this error Failed to create a session. The runner registration has been deleted from the server, please re-configure.

[nikola-jokic]

Hi @jonnikim,

If the runner does not get any tasks for 30 days, it is being cleaned up from the service side. That might be the reason why you needed to re-configure your runner again.

@brandan-schmitz, @chingc, does this help?

[mhl-itm-bhg]

I am experiencing a similar issue, when attempting to run the actions-runner (runc.cmd) on my machine I get the following error
Failed to create a session. The runner registration has been deleted from the server, please re-configure.
When attempting to reconfigure the runner (config.cmd) I get the following error
Cannot configure the runner because it is already configured. To reconfigure the runner, run 'config.cmd remove' or './config.sh remove' first.
When I run config.cmd remove I'm asked to enter a runner removal token.

I have no idea where to get this token. Is there any way to reconfigure without being dependent on tokens that disappeared from the repo?

[nikola-jokic]

Hi @mhl-itm-bhg,

You can just remove a file named .runner inside your root directory from where you are executing config.sh.

[nikola-jokic]

Hi everyone,

Since this seems to be resolved, I am going to close this issue. If you experience this issue again, you can create a new issue or write a comment here, and we will re-open it 😄

[shishodiyas]

how can we make it so that the runner doesn't get deleted.

[whutchinson98]

I just experienced this issue. Is there any update on how to prevent this?

[nikola-jokic]

The docs now state:

A self-hosted runner is automatically removed from GitHub if it has not connected to GitHub Actions for more than 14 days.

@shishodiyas, @whutchinson98 you can't. One way you can automate this is to use API to fetch the registration token and register your runner again from a shell script.

[sxtyxmm]

I have a shell script for the same but can you elaborate on the API use.