Jenkins cli is giving 4XX errors, and most of the other features seems to be broken and giving crumb error

[richamishra006]

Kubernetes version: 1.23
Jenkins Operator version: 0.8.0-beta.2

• First we installed the operator via helm, which installed the stable release(0.6.2) and the jenkins pod kept on restarting, and there was no error in the logs.
• So we checked the kubernetes manifest file and it was pointing to 0.8.0-beta.2 version, after switching to this version and adding the script, which we found in one of the github issue (without this it was giving error ), below is the error we were getting

+ echo 'Installing plugins required by Operator - end'
+ echo 'Installing plugins required by user - begin'
+ cat
Installing plugins required by Operator - end
Installing plugins required by user - begin
+ jenkins-plugin-cli --verbose --latest true -f /var/lib/jenkins/user-plugins.txt
File containing list of plugins to be downloaded: /var/lib/jenkins/user-plugins.txt
Reading in plugins from /var/lib/jenkins/user-plugins.txt

No directory to download plugins entered. Will use default of /usr/share/jenkins/ref/plugins
Using update center https://updates.jenkins.io/update-center.json from JENKINS_UC environment variable
Using experimental update center https://updates.jenkins.io/experimental/update-center.json from JENKINS_UC_EXPERIMENTAL environment variable
Using incrementals mirror https://repo.jenkins-ci.org/incrementals from JENKINS_INCREMENTALS_REPO_MIRROR environment variable
No CLI option or environment variable set for plugin info, using default of https://updates.jenkins.io/plugin-versions.json
No war entered. Will use default of /usr/share/jenkins/jenkins.war

Retrieving update center information
Update center URL: https://updates.jenkins.io/update-center.json?version=2.401.1
Returning cached value for: update-center-2.401.1
Returning cached value for: experimental-update-center-2.401.1
Returning cached value for: plugin-versions
Done
+ echo 'Installing plugins required by user - end'
Installing plugins required by user - end
bash: line 1: /usr/bin/tini: No such file or directory

• After the version change and adding the script, we realised that some of the features are broken, like i am unable to access the /manage/cli endpoint and also if we run any jenkins-cli command from the terminal, getting this error

java -jar jenkins-cli.jar -s http://localhost:8082/ -auth jenkins-operator:113d63cfcf93e0aec800c7d0b6be45e97b
io.jenkins.cli.shaded.jakarta.websocket.DeploymentException: Handshake error.
	at io.jenkins.cli.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:658)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:696)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:849)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:493)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:337)
	at hudson.cli.CLI.webSocketConnection(CLI.java:364)
	at hudson.cli.CLI._main(CLI.java:324)
	at hudson.cli.CLI.main(CLI.java:103)
Caused by: io.jenkins.cli.shaded.org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 404.
	at io.jenkins.cli.shaded.org.glassfish.tyrus.client.TyrusClientEngine.processResponse(TyrusClientEngine.java:301)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.container.jdk.client.ClientFilter.processRead(ClientFilter.java:167)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:111)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:113)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:113)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:295)
	at io.jenkins.cli.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:279)
	at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishRead(UnixAsynchronousSocketChannelImpl.java:439)
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:191)
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)
	at java.base/sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:306)
	at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Attaching the screenshot we got after hitting /manage/cli endpoint,

can someone please help here, our end goal is to create a script which will include jenkins cli to manage the jobs, but due to these errors in operator, unable to execute jenkins cli command

[brokenpip3]

I'm sorry but I never used the official jenkins cli so I'm not aware on what should be the right behavior.
If you start jenkins on your laptop you are able to use the cli?
I may need to find time to tests this 'cause like I said never used before.

[brokenpip3]

it seems that is disable here:

kubernetes-operator/pkg/configuration/base/resources/base_configuration_configmap.go

Lines 94 to 96 in 01a3990

	if (jenkins.getDescriptor("jenkins.CLI") != null) {
	jenkins.getDescriptor("jenkins.CLI").get().setEnabled(false)
	}

need to understand why and if we should keep it like that.

[brokenpip3]

probably has been disabled due to this: https://www.jenkins.io/blog/2019/02/17/remoting-cli-removed/

[brokenpip3]

Can you elaborate the use case of using the cli?