[rawhide] Cannot SSH into node #1775
Assignees
Labels
jira
for syncing to jira
kind/bug
pipeline failure
This issue or pull request is derived from CI failures
Comments
aaradhak
changed the title
gursewak1997
added
the
pipeline failure
|
I followed these steps with the latest SELinux policy, selinux-policy 41.14.1.fc41, which was just released. After running through the steps, I found that the test appears to be passing. Maybe this newer package resolved the issue? Could you try using |
|
The real test is to open a PR against rawhide that drops almost all the overrides (except the json-glib one) and fast-tracks that selinux-policy package and see what CI says. If it's green, we merge :) |
|
With the pins except the |
|
Opened this PR for testing - coreos/fedora-coreos-config#3090 |
|
I tried that locally too and that issue is still persisting: #1735. Is there a reason we cant just remove only the selinux pin? I tested that locally and the build succeeds and the kola tests pass. I also opened a PR for testing: coreos/fedora-coreos-config#3091 |
|
@jlebon Tested by adding |
|
@aaradhak Thanks for testing! OK, so it seems like definitely we have a separate possibly systemd-related problem, on our hands (because the only other package we were holding back apart from selinux-policy was systemd). Let's update this issue to reflect that. |
jlebon
changed the title
|
So here's a test: run |
|
@jlebon Ok, so I followed steps mentioned and it seems that the ssh to core@localhost gets denied due to permission issue. QEMU SSH Port check: Journal log output: Looking at the entries with [preauth], i assume that the connection was attempted but failed during the authentication phase, likely due to the public key not being accepted or the key not being recognized. |
jlebon
added a commit
to jlebon/fedora-coreos-config
that referenced
this issue
Aug 21, 2024
systemd v256 added a new userdb functionality where SSH authorized keys can be part of a User Record. To make this transparently work with sshd authentication, an sshd config dropin that sets an `AuthorizedKeysCommand` directive was added. Unfortunately, it was added with a higher priority than intended, which meant that it overrode the `AuthorizedKeysCommand` directive from `ssh-key-dir`, which is how our `~/.ssh/authorized_keys.d/` magic works today with Ignition and Afterburn. So the end result is that this broke SSH which of course broke kola too. This is tracked in upstream systemd at: systemd/systemd#33648 The dropin was recently reverted in Fedora: https://src.fedoraproject.org/rpms/systemd/c/38291e13c1dec15618b7d09e4217d10076897cdf?branch=rawhide The latest rawhide systemd build with that change is already in the repos: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ff872f0544 So we can just drop all the overrides to pull in the latest systemd. We'll need to keep an eye on the conversation there to make sure that the final solution doesn't re-break FCOS, but we would notice it pretty quickly too. Closes: coreos/fedora-coreos-tracker#1775
jlebon
added a commit
to jlebon/fedora-coreos-config
that referenced
this issue
Aug 21, 2024
systemd v256 added a new userdb functionality where SSH authorized keys can be part of a User Record. To make this transparently work with sshd authentication, an sshd config dropin that sets an `AuthorizedKeysCommand` directive was added. Unfortunately, it was added with a higher priority than intended, which meant that it overrode the `AuthorizedKeysCommand` directive from `ssh-key-dir`, which is how our `~/.ssh/authorized_keys.d/` magic works today with Ignition and Afterburn. So the end result is that this broke SSH which of course broke kola too. This is tracked in upstream systemd at: systemd/systemd#33648 The dropin was recently reverted in Fedora: https://src.fedoraproject.org/rpms/systemd/c/38291e13c1dec15618b7d09e4217d10076897cdf?branch=f41 The latest f41 systemd build with that change is already in the repos: https://bodhi.fedoraproject.org/updates/FEDORA-2024-8d144cc8af So we can just drop all the overrides to pull in the latest systemd. We'll need to keep an eye on the conversation there to make sure that the final solution doesn't re-break FCOS, but we would notice it pretty quickly too. Closes: coreos/fedora-coreos-tracker#1775
|
Should be fixed by coreos/fedora-coreos-config#3115 and coreos/fedora-coreos-config#3116. |
jlebon
added a commit
to jlebon/fedora-coreos-config
that referenced
this issue
Aug 22, 2024
systemd v256 added a new userdb functionality where SSH authorized keys can be part of a User Record. To make this transparently work with sshd authentication, an sshd config dropin that sets an `AuthorizedKeysCommand` directive was added. Unfortunately, it was added with a higher priority than intended, which meant that it overrode the `AuthorizedKeysCommand` directive from `ssh-key-dir`, which is how our `~/.ssh/authorized_keys.d/` magic works today with Ignition and Afterburn. So the end result is that this broke SSH which of course broke kola too. This is tracked in upstream systemd at: systemd/systemd#33648 The dropin was recently reverted in Fedora: https://src.fedoraproject.org/rpms/systemd/c/38291e13c1dec15618b7d09e4217d10076897cdf?branch=rawhide Fast-track the latest rawhide systemd build with that change. We'll need to keep an eye on the conversation there to make sure that the final solution doesn't re-break FCOS, but we would notice it pretty quickly too. Closes: coreos/fedora-coreos-tracker#1775
jlebon
added a commit
to jlebon/fedora-coreos-config
that referenced
this issue
Aug 22, 2024
systemd v256 added a new userdb functionality where SSH authorized keys can be part of a User Record. To make this transparently work with sshd authentication, an sshd config dropin that sets an `AuthorizedKeysCommand` directive was added. Unfortunately, it was added with a higher priority than intended, which meant that it overrode the `AuthorizedKeysCommand` directive from `ssh-key-dir`, which is how our `~/.ssh/authorized_keys.d/` magic works today with Ignition and Afterburn. So the end result is that this broke SSH which of course broke kola too. This is tracked in upstream systemd at: systemd/systemd#33648 The dropin was recently reverted in Fedora: https://src.fedoraproject.org/rpms/systemd/c/38291e13c1dec15618b7d09e4217d10076897cdf?branch=f41 Fast-track the latest f41 systemd build with that change. We'll need to keep an eye on the conversation there to make sure that the final solution doesn't re-break FCOS, but we would notice it pretty quickly too. Closes: coreos/fedora-coreos-tracker#1775
jlebon
added a commit
to coreos/fedora-coreos-config
that referenced
this issue
Aug 22, 2024
systemd v256 added a new userdb functionality where SSH authorized keys can be part of a User Record. To make this transparently work with sshd authentication, an sshd config dropin that sets an `AuthorizedKeysCommand` directive was added. Unfortunately, it was added with a higher priority than intended, which meant that it overrode the `AuthorizedKeysCommand` directive from `ssh-key-dir`, which is how our `~/.ssh/authorized_keys.d/` magic works today with Ignition and Afterburn. So the end result is that this broke SSH which of course broke kola too. This is tracked in upstream systemd at: systemd/systemd#33648 The dropin was recently reverted in Fedora: https://src.fedoraproject.org/rpms/systemd/c/38291e13c1dec15618b7d09e4217d10076897cdf?branch=f41 Fast-track the latest f41 systemd build with that change. We'll need to keep an eye on the conversation there to make sure that the final solution doesn't re-break FCOS, but we would notice it pretty quickly too. Closes: coreos/fedora-coreos-tracker#1775
jlebon
added a commit
to coreos/fedora-coreos-config
that referenced
this issue
Aug 26, 2024
systemd v256 added a new userdb functionality where SSH authorized keys can be part of a User Record. To make this transparently work with sshd authentication, an sshd config dropin that sets an `AuthorizedKeysCommand` directive was added. Unfortunately, it was added with a higher priority than intended, which meant that it overrode the `AuthorizedKeysCommand` directive from `ssh-key-dir`, which is how our `~/.ssh/authorized_keys.d/` magic works today with Ignition and Afterburn. So the end result is that this broke SSH which of course broke kola too. This is tracked in upstream systemd at: systemd/systemd#33648 The dropin was recently reverted in Fedora: https://src.fedoraproject.org/rpms/systemd/c/38291e13c1dec15618b7d09e4217d10076897cdf?branch=rawhide Fast-track the latest rawhide systemd build with that change. We'll need to keep an eye on the conversation there to make sure that the final solution doesn't re-break FCOS, but we would notice it pretty quickly too. Closes: coreos/fedora-coreos-tracker#1775
|
i am still seeing this errors on the s390x build of branched : https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/blue/organizations/jenkins/build-arch/detail/build-arch/106/pipeline/ relevant packages : Most tests fails with Relevant selinux denials : |
|
@jbtrystram That looks like a different bug. Can you make a separate issue for this so we don't get confused? Let's close this one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
jlebon: see #1775 (comment) and following. This is unlikely to be an SELinux issue.
Original report follows.
Describe the bug
The kola
basic.uefitest is failing with AVC denials when using selinux-policy-41.13-1.fc41 RPM. The test failure occurs with the following error:console log:
The avc denials seen here seem to be all permissive=1
Reproduction steps
Expected behavior
cosa runto come upkola test basic.uefi to PASS
Actual behavior
cosa run hangs
kola test basic.uefi fails:
System details
upstream rawhide x86_64 with selinux-policy-41.13-1.fc41
Butane or Ignition config
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: