-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GitHub] Handshake failed: knownhosts: key mismatch #490
Comments
The known_hosts file is in the same secret as the SSH key, please see the docs here https://fluxcd.io/docs/components/source/gitrepositories/#ssh-authentication |
I'm getting the same error on my cluster:
Looks like an issue with GitHub host keys. |
I am also seeing this error in the last 30 minutes on 3 clusters that had been previously working fine |
According to: https://github.blog/2021-09-01-improving-git-protocol-security-github/ Today is the day that host keys get rotated at GitHub. There are two new host keys in the blog post, one for ECDSA and another for Ed25519. |
Ok so rotating the SSH key fixes it. Before: $ k -n flux-system get secret flux-system -o json | jq '.data | map_values(@base64d)'
{
"identity": "-----BEGIN PRIVATE KEY-----\n",
"identity.pub": "ecdsa-sha2-nistp384 \n",
"known_hosts": "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=="
} After: {
"identity": "-----BEGIN PRIVATE KEY-----\n",
"identity.pub": "ecdsa-sha2-nistp384 \n",
"known_hosts": "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg="
} |
Cool, thanks, but I do see the "old" keys when doing keyscan on the nodes. |
GitHub has changed its SSH host keys from DSA to ECDSA! To fix the key mismatch error, you have two options: Update the
Or rotate the SSH keys with
|
Updated |
If you'd like a short program to do it: #!/usr/bin/env bash
set -e -u -o pipefail
# NB: The Ed25519-format key does not work with Flux.
for secret_name in flux-system repo-2 repo-3; do
kubectl --namespace=flux-system \
patch secret "${secret_name}" \
--patch='
stringData:
known_hosts: >
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg='
done
kubectl --namespace=flux-system rollout restart deployment source-controller
kubectl --namespace=flux-system rollout status deployment/source-controller --watch |
Confirmed. Working for us now as well after deleting the secret and bootstrapping again. |
@seh the secret is not mounted inside source-controller, instead the controller reads the secret from Kubernetes API before each Git operations. I don't think you need rollout restart. |
I was finding that it sits in what appears to be due to a backed-off timer, such that it won't try again for a while after several consecutive failures, but restarting it caused it to try again immediately. |
Variant on the above script: https://gist.github.com/ellieayla/76352313c4f5939db6d2268fb70b0d48 Then either wait or request each GitRepository to reconcile. |
Confirm that we are getting this on our cluster as well suddenly. |
Update the documentation and example to use the new SSH public key that GitHub deployed after Nov 16th 2021. See also fluxcd/source-controller#490 Signed-off-by: Alessandro Degano <a.degano@gmail.com>
@stefanprodan maybe add to the comment that if you edit the secrets manually, you should restart the We've stopped the
Edit: the old |
source-controller doesn't alter secrets. It can't even do that, our RBAC allows the controller read-only access to secrets. |
You clearly don't use bootstrap or you've stored the SSH keys in Git. If so, then update the secret in Git as well. |
Unfortunately, this was a predictable incident. It felt wrong to me, as a Flux user, to be providing a known hosts entry as part of the terraform bootstrap process (from this example) for precisely this reason. To prevent another incident of similar scale in the future, why not give the |
Bootstrap does no such thing, Flux itself generates the known_hosts entries. As a Flux user, you are never asked to provide host keys. |
Are multiple |
|
@rtjfarrimond I was referring to |
I understand, but to be clear, in my original comment I was referring to the terraform bootstrap process. Updated the original comment to reflect this. |
How can a |
We have two git sources,
Is |
Sorry, my bad. It looks like we have the secrets for |
The Secret files are not managed or written to by any of the controllers, but only used for read operations. If something is overwriting your Secret, it must come from something within your configuration. |
If the some process were to update the known_hosts runs on the same box with the same user that uses the known_hosts file, where would the vector for a MITM be? |
By it automatically accepting the offered keys. If your network is compromised and |
Yep, that makes sense, I withdraw my bad idea! Thanks :) |
@stefanprodan Here is a PR to update the known_hosts in the terraform example I linked earlier. |
Two things lengthened my fixing of this problem across ~20 clusters:
I had to patch the top-level Kustomization to set "spec.wait" to false, then force Flux to reconcile it. It took many tries before the health checking timeouts expired and Flux finally both updated and then started using the new Secret "data.known_hosts" field value. |
Update the documentation and example to use the new SSH public key that GitHub deployed after Nov 16th 2021. See also fluxcd/source-controller#490 Also update the suggested algorithm for the SSH key to use with GitHub to ensure (compability with libgit2)[https://github.blog/2021-09-01-improving-git-protocol-security-github/#libgit2-and-other-git-clients]. Signed-off-by: Alessandro Degano <a.degano@gmail.com>
Thanks for the suggestion, in my case i also had to:
|
worked for us |
Worked for me as well, thanks! 👯 |
|
In my case bootstrap fails to create new secret
Switching from ssh to https helped |
@kaaboaye your user token doesn’t have permission to create deploy keys, you need to be a repo admin. |
I am having a very similar, if not the same, error while setting up
This is a brand new instantiation which I have just fired up a few minutes ago as of this writing. |
@ninja9k1 - I assume by now that you've resolved this issue for your local The This command should remove the existing key:
You can then either use this command to insert the new key without actually trying to SSH to GitHub:
Or start an SSH connection to
|
An error is still reported ( Handshake failed: knownhosts: key mismatch ) when a new ecdsa hostkey is generated |
I managed to get SFTP working, see here on #2948 |
Update the documentation and example to use the new SSH public key that GitHub deployed after Nov 16th 2021. See also fluxcd/source-controller#490 Also update the suggested algorithm for the SSH key to use with GitHub to ensure (compability with libgit2)[https://github.blog/2021-09-01-improving-git-protocol-security-github/#libgit2-and-other-git-clients]. Signed-off-by: Alessandro Degano <a.degano@gmail.com>
Started getting these errors out of the blue on all clusters.
Doing
find -name known_hosts
in the pod produces nothing.Restarting the pod = same error immediately.
What's going on, where's the
known_hosts
file?The text was updated successfully, but these errors were encountered: