Intermittent clone failures since 09/11/2024 - kex_exchange_identification: Connection closed by remote host

[Narrator]

Select Topic Area

Bug

Body

Hi 👋

I'm an engineer from the AWS Amplify service - Our customers use GitHub to connect their applications to Amplify and we offer a CI/CD workflow where we clone the customer's repository to allow customers to run their build commands.

Since 09/11, our customers have been intermittently observing this issue while cloning:

 kex_exchange_identification: Connection closed by remote host
 Connection closed by 20.205.243.166 port 22
 fatal: Could not read from remote repository.
 Please make sure you have the correct access rights
 and the repository exists.

The fact that these failures are intermittent is what is concerning to us. Nothing has changed in the build containers we provision for our customers - The SSH settings are the same as they always were.

We run ssh-keyscan github.com and add the host keys to the ~/.ssh/known_hosts file every time before we clone. Our logs show that this call succeeded, but the failure still occurs.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[renanleme]

I'm seeing the same issue on our services, we are running on AWS ECS Fargate and it's also intermitent

Examples:

[2024-09-14, 09:18:26 UTC] git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
[2024-09-14, 09:18:26 UTC]   cmdline: git clone -v --branch=main --filter=blob:none -- git@github.com:my_company/my_repo.git  /tmp/tmp89ax7e8o
[2024-09-14, 09:18:26 UTC]   stderr: 'Cloning into '/tmp/tmp89ax7e8o'...
[2024-09-14, 09:18:26 UTC] kex_exchange_identification: Connection closed by remote host
[2024-09-14, 09:18:26 UTC] Connection closed by 140.82.116.4 port 22
[2024-09-14, 09:18:26 UTC] fatal: Could not read from remote repository.
[2024-09-14, 09:18:26 UTC] Please make sure you have the correct access rights
[2024-09-14, 09:18:26 UTC] and the repository exists.

[2024-09-16 08:09:26,020] git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
[2024-09-16 08:09:26,020]   cmdline: git clone -v --branch=main --filter=blob:none -- git@github.com:my_company/my_repo.git /tmp/tmpm3rvwspu
[2024-09-16 08:09:26,020]   stderr: 'Cloning into '/tmp/tmpm3rvwspu'...
[2024-09-16 08:09:26,020] kex_exchange_identification: Connection closed by remote host
[2024-09-16 08:09:26,020] Connection closed by 140.82.116.4 port 22
[2024-09-16 08:09:26,020] fatal: Could not read from remote repository.
[2024-09-16 08:09:26,020] Please make sure you have the correct access rights
[2024-09-16 08:09:26,020] and the repository exists.

[djhughes]

We have also been experiencing intermittent failures of this same type since 11th September in our CI/CD environments. There have been no changes to our build containers. We are using a version controlled known_hosts file.

error Command failed.
Exit code: 128
Command: git
Arguments: clone ssh://git@github.com/<redacted>/<redacted> /__w/l/l/cache/yarn/v6/.tmp/9633bdeff9fc5e7d34e4017d846876e5
Directory: /__w/l/l/s
Output:
Cloning into '/__w/l/l/cache/yarn/v6/.tmp/9633bdeff9fc5e7d34e4017d846876e5'...
kex_exchange_identification: Connection closed by remote host

15:08:47 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- git@github.com:<redacted>/<redacted>.git +refs/pull/*:refs/remotes/origin/pr/*" returned status code 128:
15:08:47 stdout: 
15:08:47 stderr: kex_exchange_identification: read: Connection reset by peer
15:08:47 Connection reset by 20.26.156.215 port 22
15:08:47 fatal: Could not read from remote repository.
15:08:47 
15:08:47 Please make sure you have the correct access rights
15:08:47 and the repository exists.

[ajax-koriakin-m]

I'm seeing the same issue on our Jenkins installations. We are running on AWS EC2 on-demand as the main node and spot instances as agents, and it's also intermittent.

Example:

hudson.plugins.git.GitException: Command "git fetch --tags --force --progress --prune -- origin +refs/heads/master:refs/remotes/origin/master" returned status code 128:
stdout: 
stderr: kex_exchange_identification: Connection closed by remote host
Connection closed by <server_ip> port 22
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
	at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:406)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:219)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:124)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:71)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
Finished: FAILURE

[thabiseak]

It seems like AWS Amplify users are experiencing intermittent issues when cloning repositories from GitHub, particularly since 09/11. The error message "kex_exchange_identification: Connection closed by remote host" is being encountered, followed by the connection closing by port 22, and the failure to read from the repository.

The AWS engineer points out that their setup has not changed — they still run ssh-keyscan github.com and add host keys to the known_hosts file before cloning, but despite this, the failure persists. Since the issue is intermittent, it's concerning, and they are seeking insights or potential solutions for what might be causing this behavior.

[gitlon]

Github have noted it as an issue on their status page https://www.githubstatus.com/incidents/r3x7x31k7nn1

Disruption with Git SSH
Update
We are seeing up to 2% of Git SSH connections failing.

We have taken suspected problematic hosts out of rotation and are monitoring for recovery and continuing to investigate.
Posted 10 minutes ago. Sep 16, 2024 - 13:38 UTC

[Narrator]

The errors have stopped happening for us as of 6:24am PST this morning.