Unable to create new jenkins agent

[placydo]

Jenkins is failing to create new agents with following errors:
io.fabric8.kubernetes.client.KubernetesClientException: at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

and

java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden' at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

We have managed to pinpoint the root cause. The kubernetes-plugin bug is causing it.
As a temporary workaround we had to add explicitly port 443 to kubernetes URL.

It would be great if jenkins operator could set full and explicit URL (just in case):

kubernetes.default.svc.cluster.local:443

Also, important info, is that the problem appeared with no user interaction, looks like the operator updated the plugin.

Our versions:

• Jenkins 2.176.3
• kubernetes plugin 1.15.7
• kubernetes v1.13.10-eks-5ac0f1

Upstream issue:
https://issues.jenkins-ci.org/browse/JENKINS-59000?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true

Last known working version of kubernetes-plugin is 1.15.2 and 403 were spotted on 1.15.3

[tumevoiz]

Hi @placydo
We will check this. As a temporary solution, I recommend reverting the plugin version to the last working version. You can do this under spec.master.basePlugins.

[tomaszsek]

Fixed in https://github.com/jenkinsci/kubernetes-operator/releases/tag/v0.2.1