Publish Over SSH fails to connect to Serv-U servers

[omehegan]

Jenkins and plugins versions report

After upgrading Serv-U from version 15.3.0.1375 to 15.3.2.172, Publish Over SSH can no longer connect to the server. The exception is:

2023-05-02 17:32:52.029+0000 [id=5495639]       WARNING j.p.p.BapSshHostConfiguration#connect: Failed to connect session for config [JenkinSFTP]. Message [Session.con
IO Stream Read]
com.jcraft.jsch.JSchException: Session.connect: java.io.IOException: End of IO Stream Read
        at com.jcraft.jsch.Session.connect(Session.java:565)
        at jenkins.plugins.publish_over_ssh.BapSshHostConfiguration.connect(BapSshHostConfiguration.java:403)
        at jenkins.plugins.publish_over_ssh.BapSshHostConfiguration.createClient(BapSshHostConfiguration.java:274)
        at jenkins.plugins.publish_over_ssh.BapSshHostConfiguration.createClient(BapSshHostConfiguration.java:263)
        at jenkins.plugins.publish_over_ssh.descriptor.BapSshPublisherPluginDescriptor.validateConnection(BapSshPublisherPluginDescriptor.java:204)
        at jenkins.plugins.publish_over_ssh.descriptor.BapSshPublisherPluginDescriptor.doTestConnection(BapSshPublisherPluginDescriptor.java:199)
        at jenkins.plugins.publish_over_ssh.descriptor.BapSshHostConfigurationDescriptor.doTestConnection(BapSshHostConfigurationDescriptor.java:102)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
        at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
        at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:78)
        at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
        at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
        at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
        at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
        at com.cloudbees.jenkins.ha.HAHealthCheckFilter.doFilter(HAHealthCheckFilter.java:35)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
        at com.splunk.splunkjenkins.WebPostAccessLogger.doFilter(WebPostAccessLogger.java:39)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at com.cloudbees.analytics.gatherer.InboundLink$1.doFilter(InboundLink.java:113)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at com.cloudbees.devoptics.deliver.jira.ApiProxyFilter.doFilter(ApiProxyFilter.java:59)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at com.cloudbees.jenkins.plugins.requestfilter.BlockingFilter.doFilter(BlockingFilter.java:78)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:95)
        at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:70)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:68)
        at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:43)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
        at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
        at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:118)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
        at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
        at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
        at java.base/java.lang.Thread.run(Unknown Source) 

According to SolarWinds, who support Serv-U, this is caused by the fact that the Jsch library does not strictly adhere to the SFTP RFC. See https://support.solarwinds.com/SuccessCenter/s/article/Unable-to-connect-through-SFTP-using-Java-based-clients-after-upgrading-to-Serv-U-version-15-3-2?language=en_US

They have provided a patch for Serv-U that can fix this issue, but they recommend that users of Jsch update it to a version that fixes this RFC compliance issue. The original Jsch project seems to have been abandoned, but this fork resolves the issue and is backwards-compatible: https://github.com/mwiede/jsch

Just wanted to file this in case it helps others.

What Operating System are you using (both controller, and any agents involved in the problem)?

Windows Server 2012 R2

Reproduction steps

1. Try to publish SSH artifact to Serv-U server version 15.3.2.172 or newer, or configure such a server and click the 'Test Configuration' button.

Expected Results

Successful connection and publish of artifact.

Actual Results

Connection fails with stack trace shown above.

Anything else?

No response