Test with Java 21

[MarkEWaite]

Test with Java 21

Java 21 was released Sep 19, 2023. We want to announce full support for Java 21 in early October and would like the most used plugins to be compiled and tested with Java 21.

The acceptance test harness and plugin bill of materials tests are already passing with Java 21. This is a further step to improve plugin readiness for use with Java 21 and for development with Java 21.

The change intentionally tests only two Java configurations, Java 17 and Java 21 because we believe that the risk of a regression that only affects Java 11 is low. We generate Java 11 byte code with the Java 17 and the Java 21 builds, so we're already testing Java 11 byte code.

Includes or supersedes pull requests:

• use jsch plugin instead of version bundled in groovy-ssh library to support newer algorithms #101
• Bump plugin from 4.54 to 4.62 #97
• Bump git-changelist-maven-extension from 1.4 to 1.6 #89
• Bump lombok from 1.18.24 to 1.18.26 #86

JENKINS-65533 requests support for current OpenSSH keys. #101 provides that support by switching to the jsch plugin that is now using the mwiede fork of jsch instead of using the original unmaintained jsch.

Test instructions

Run automated tests with Java 21 and Java 17.

Use the plugin with an OpenSSH key generated with ssh-keygen -t ed25519

Submitter checklist

• Link to JIRA ticket in description, if appropriate.
• Change is code complete and matches issue description.
• Appropriate unit or acceptance tests or explanation to why this change has no tests.
• Reviewer's manual test instructions provided in PR description. See Reviewer's first task below.

Reviewer checklist

• Run the changes and verified the change matches the issue description.
• Reviewed the code.
• Verified that the appropriate tests have been written or valid explanation given.
• If applicable, test installing this plugin on the Jenkins instance.

[nrayapati]

Thank you @MarkEWaite and @globalbus!

[Fribb]

Could it be that #101 somehow wasn't released with the latest version 2.0.68.va_d21a_12a_6476?

At least the release version doesn't mention this in the message and I also can't find it in any of the commits.

[MarkEWaite]

@Fribb, the description of this pull request says that it supersedes #101. It includes the relevant contents of #101 and other improvements. Pull request #101 was not merged into the latest version, but the contents of #101 should be included in the most recent release as part of commit 7409b1c . That commit was squash merged into the most recent release as ad21a12

[Fribb]

@MarkEWaite Thank you for the clarification.

While I understand this, I can't get it to work and I can't figure out where the issue might be.
I have the SSH Pipeline Step Plugin 2.0.68.va_d21a_12a_6476 and JSch dependency Plugin 0.2.16-86.v42e010d9484b_ installed

But whenever I try to run a simple test pipeline with the SSH Pipeline Step, it fails with "Auth fail for methods 'publickey,password'". this is with an ed25519 private key

I tested this with running the Agent in Java 11, 17 and 21 now, it is always the same error.

And the Credentials are correct and work, using sh 'ssh -i ${KEYFILE} root@theIP "ls -l"' from a Pipeline works without issues.

def remote = [:]
remote.name = 'ECS-ACN'
remote.host = 'the IP'
remote.port = 22
remote.allowAnyHosts = true

node {
    
    withCredentials([sshUserPrivateKey(credentialsId: 'ecs-customer-live',
                                       keyFileVariable: 'KEYFILE',
                                       passphraseVariable: 'PASSPHRASE',
                                       usernameVariable: 'USERNAME')]) {

        remote.user = USERNAME
        remote.identityFile = KEYFILE
        stage("SSH") {
            sshCommand remote: remote,
                       command: "ls -l"
        }
    }
}

[MarkEWaite]

using sh 'ssh -i ${KEYFILE} root@theIP "ls -l"' from a Pipeline works without issues.

It is almost always preferable to use a shell step to perform a Pipeline operation than to rely on a plugin to perform that operation. Shell steps are easier to test locally. Shell steps are easier to diagnose locally. Since the shell step works, I think you should use the shell step rather than the ssh-steps plugin.

[Fribb]

That shell command was for testing purposes to see if there was an issue with the stored Jenkins credentials. However, I cannot find a specific "shell step" that you are talking about either in the default plugin installation or as an available plugin that I can install or the pipeline syntax, can you explain that or link to where I can find something about this?

Still, that wouldn't really address the overall issue that I have with the SSH Pipeline steps plugin, only to circumvent it. I have a fairly big pipeline that relies on and uses the SSH Pipeline steps Plugin, I would have to painstakingly convert every usage to their equivalent shell command which I would rather avoid.

If this isn't the right place for this, I can understand this, but so far neither on Reddit nor the Jenkins Discourse Posts have gotten any replies for multiple days now.

[MarkEWaite]

If it is important to you that the ssh steps plugin must support your use case, then it is probably important enough for you to offer to become one of the maintainers of the ssh steps plugin. Then you can assure that it addresses your use case, has tests for that use case, and continues to meet your needs.

When I said "shell step", I meant the general purpose "sh", "bat", and "powershell" steps in Jenkins Pipeline.

I'm not a maintainer of the ssh steps plugin. I'm not a user of the ssh steps plugin. I replied to your questions here because I was the outside contributor that submitted this pull request as part of assuring that Jenkins supports Java 21.

[Fribb]

I am not really here to argue. I just saw this pull request which should have addressed the ed25519 private key issue that I currently experience with this plugin which, according to your own description and the pull request, this should have been addressed already.

However, reporting a potential Bug doesn't automatically mean I have the time, motivation or experience to contribute to Jenkins as a whole or the Plugin. This was not some new functionality, this is a potential issue (or rather an existing issue that you linked to yourself: JENKINS-65533)

When I said "shell step", I meant the general purpose "sh", "bat", and "powershell" steps in Jenkins Pipeline.

I used that...

Regardless, as I said, I don't to argue. I just thought that if there is a pull request that should address an issue which doesn't (at least in my case), this would be a good place to get some clarification for it. But apparently, it is not.

[jpasher-lazor]

@Fribb I think there was just some confusion on what you are trying to do versus what the sample script is doing. I can offer a little guidance, since I'm using the SSH Steps plugin and ran into the same original problem.

I've had SSH Steps set up for quite a while, and I had to switch to an ed25519 key back when Ubuntu 22.04 came out, because they removed the older style rsa-sha1 key exchange (RSA is still supported, but older libraries might not recognize the newer identifier). Keep in mind that the original bug fix wasn't to make ed25519 work (that always worked); it was to make it support the new rsa type being announced. With that said, I still have jobs that use the ed25519 key without problems on the same version of SSH Steps and JSch that you are using. I'm also still using Java 11.

My suggestion is to test a few different things to narrow down where the problem is actually occurring, because it's probably not related to the bug with this plugin that was originally fixed.

• Try connecting to a different host (even better if it's a distro/release with a different sshd version)
• Try using an RSA key
• Try using a key without a password

[Fribb]

@jpasher-lazor thanks for the reply but I must ask, what do you mean by "what you are trying to do versus what the sample script is doing"? With the example pipeline, I try to connect to the remote with my keyfile and execute the ls -l command. isn't that what it should be doing or does as it is described?

Try connecting to a different host (even better if it's a distro/release with a different sshd version)

I just took a couple of samples and one of our test systems (running in Docker) has an OpenSSH 6.6.1, one customer server has 7.2, and a different one has 8.2. I can connect to them without issues from the Jenkins agent (Ubuntu 22.04, OpenSSH 8.9) from the command line.

I also ran the same command over the complete list of customers (63 different Ubuntu servers) and none of them could successfully connect. while I haven't checked specifically every one of them, the ones that I checked, and that failed too, all have the ed25519 public key in their authorized_keys for the root user that we use to connect. I mean, it works with the key with a native ssh command so the configuration on the server side might not be the problem, right?

Try using a key without a password

The key was created without a passphrase

I also tried to connect with a username and password, which doesn't throw an error but a simple pwd command doesn't output anything in the logs which is more than weird.

I tried to enable verbose logging for the Plugin and after outputting the log once and not again (which was "fixed" by a restart of Jenkins) I could use the remote.logLevel = 'FINEST' to produced some interesting output

[jsch] CheckSignatures: ssh-ed25519,ssh-ed448
[jsch] ssh-ed25519 is not available.
[jsch] ssh-ed448 is not available.

If I am not mistaken, this is the remote SSH configuration. but looking into /etc/ssh/sshd_config it says

HostKey /etc/ssh/ssh_host_ed25519_key

which should enable it, correct?

Edit: Just to add, I am pretty sure that this isn't because of the servers.