Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade Jenkins, and then Jenkins failed to start #83

Closed
watchman-rorschach opened this issue Mar 14, 2022 · 3 comments
Closed

Upgrade Jenkins, and then Jenkins failed to start #83

watchman-rorschach opened this issue Mar 14, 2022 · 3 comments

Comments

@watchman-rorschach
Copy link

watchman-rorschach commented Mar 14, 2022 •
edited
Loading

upgrade Jenkins on the console page of Jenkins，and then Jenkins failed to start

[root@node1 ~]# kubectl describe pods devops-jenkins-5d744f66b9-4bxxh -n  kubesphere-devops-system
Name:         devops-jenkins-5d744f66b9-4bxxh
Namespace:    kubesphere-devops-system
Priority:     0
Node:         node2/172.171.100.155
Start Time:   Mon, 14 Mar 2022 16:05:52 +0800
Labels:       app=devops-jenkins
              chart=jenkins-0.19.3
              component=devops-jenkins-master
              heritage=Helm
              pod-template-hash=5d744f66b9
              release=devops
Annotations:  checksum/config: be5c33be923a1201e22615d80894af4e2c9f8023ccc1d99d1b6a771176c0991a
              cni.projectcalico.org/containerID: 74a969da34ad104a011ec28b068054a02bf2887ef65477b030f70df44796783d
              cni.projectcalico.org/podIP: 10.233.96.73/32
              cni.projectcalico.org/podIPs: 10.233.96.73/32
Status:       Running
IP:           10.233.96.73
IPs:
  IP:           10.233.96.73
Controlled By:  ReplicaSet/devops-jenkins-5d744f66b9
Init Containers:
  copy-default-config:
    Container ID:  docker://67368ded3f29bec5350f577181346033a85e95575cac0411882e3c1f0120d6d3
    Image:         kubesphere/ks-jenkins:v3.2.0-2.249.1
    Image ID:      docker-pullable://kubesphere/ks-jenkins@sha256:9f6c0ffd0ff0ad76b279386f09e0e59fe55933a6e146b1bdfdfb09dbaf21790b
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      /var/jenkins_config/apply_config.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 14 Mar 2022 16:05:57 +0800
      Finished:     Mon, 14 Mar 2022 16:05:57 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     100m
      memory:  1500Mi
    Environment:
      ADMIN_PASSWORD:  <set to the key 'jenkins-admin-password' in secret 'devops-jenkins'>  Optional: false
      ADMIN_USER:      <set to the key 'jenkins-admin-user' in secret 'devops-jenkins'>      Optional: false
    Mounts:
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (rw)
      /var/jenkins_home from jenkins-home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5q57 (ro)
Containers:
  devops-jenkins:
    Container ID:  docker://9f9933cb6c77b14c53b1bc3b5edecf829051eb59e30339ec42ed13fbc04c9f12
    Image:         kubesphere/ks-jenkins:v3.2.0-2.249.1
    Image ID:      docker-pullable://kubesphere/ks-jenkins@sha256:9f6c0ffd0ff0ad76b279386f09e0e59fe55933a6e146b1bdfdfb09dbaf21790b
    Ports:         8080/TCP, 50000/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
      --argumentsRealm.roles.$(ADMIN_USER)=admin
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Mon, 14 Mar 2022 16:42:35 +0800
      Finished:     Mon, 14 Mar 2022 16:46:02 +0800
    Ready:          False
    Restart Count:  9
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:      100m
      memory:   1500Mi
    Liveness:   http-get http://:http/login delay=90s timeout=5s period=10s #success=1 #failure=12
    Readiness:  http-get http://:http/login delay=60s timeout=1s period=10s #success=1 #failure=3
    Environment:
      JAVA_TOOL_OPTIONS:                  -Xms512m -Xmx512m -XX:MaxRAM=2g -Dhudson.slaves.NodeProvisioner.initialDelay=20 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Dhudson.model.LoadStatistics.clock=5000 -Dhudson.model.LoadStatistics.decay=0.2 -Dhudson.slaves.NodeProvisioner.recurrencePeriod=5000 -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=10000 -verbose:gc -Xloggc:/var/jenkins_home/gc-%t.log -XX:NumberOfGCLogFiles=2 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=100m -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions 
      JENKINS_OPTS:                       
      ADMIN_PASSWORD:                     <set to the key 'jenkins-admin-password' in secret 'devops-jenkins'>  Optional: false
      ADMIN_USER:                         <set to the key 'jenkins-admin-user' in secret 'devops-jenkins'>      Optional: false
      CASC_JENKINS_CONFIG:                /var/jenkins_home/casc_configs/
      CASC_MERGE_STRATEGY:                override
      com.sun.jndi.ldap.connect.timeout:  15000
      com.sun.jndi.ldap.read.timeout:     60000
      kubernetes.connection.timeout:      60000
      kubernetes.request.timeout:         60000
      EMAIL_SMTP_HOST:                    mail.example.com
      EMAIL_SMTP_PORT:                    465
      EMAIL_USE_SSL:                      false
      EMAIL_FROM_NAME:                    KubeSphere
      EMAIL_FROM_ADDR:                    admin@example.com
      EMAIL_FROM_PASS:                    P@ssw0rd
    Mounts:
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (ro)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_home/casc_configs from casc-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5q57 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  casc-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      jenkins-casc-config
    Optional:  false
  jenkins-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      devops-jenkins
    Optional:  false
  secrets-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  devops-jenkins
    ReadOnly:   false
  kube-api-access-j5q57:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  41m                   default-scheduler  Successfully assigned kubesphere-devops-system/devops-jenkins-5d744f66b9-4bxxh to node2
  Normal   Pulling    41m                   kubelet            Pulling image "kubesphere/ks-jenkins:v3.2.0-2.249.1"
  Normal   Pulled     41m                   kubelet            Successfully pulled image "kubesphere/ks-jenkins:v3.2.0-2.249.1" in 2.977210537s
  Normal   Created    41m                   kubelet            Created container copy-default-config
  Normal   Started    41m                   kubelet            Started container copy-default-config
  Normal   Pulling    41m                   kubelet            Pulling image "kubesphere/ks-jenkins:v3.2.0-2.249.1"
  Normal   Pulled     41m                   kubelet            Successfully pulled image "kubesphere/ks-jenkins:v3.2.0-2.249.1" in 2.988911433s
  Normal   Created    41m                   kubelet            Created container devops-jenkins
  Normal   Started    41m                   kubelet            Started container devops-jenkins
  Warning  BackOff    11m (x12 over 13m)    kubelet            Back-off restarting failed container
  Warning  Unhealthy  6m1s (x126 over 40m)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  61s (x120 over 39m)   kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500

the log of jenkins

2022-03-14 08:55:11.425+0000 [id=20]	INFO	jenkins.model.Jenkins#cleanUp: Jenkins stopped

 2022-03-14 08:55:42.762+0000 [id=13]	WARNING	h.ExpressionFactory2$JexlExpression#evaluate: Caught exception evaluating: it.stackTraceString in /login. Reason: java.lang.reflect.InvocationTargetException

 java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong.

 	at jenkins.model.Jenkins.get(Jenkins.java:777)

 	at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.getDescriptor(DataBoundConfigurator.java:217)

 	at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.getName(DataBoundConfigurator.java:205)

 	at io.jenkins.plugins.casc.ConfiguratorException.getMessage(ConfiguratorException.java:70)

 	at java.lang.Throwable.getLocalizedMessage(Throwable.java:392)

 	at java.lang.Throwable.toString(Throwable.java:481)

 	at hudson.Functions.doPrintStackTrace(Functions.java:1700)

 	at hudson.Functions.doPrintStackTrace(Functions.java:1691)

 	at hudson.Functions.doPrintStackTrace(Functions.java:1691)

 	at hudson.Functions.doPrintStackTrace(Functions.java:1691)

 	at hudson.Functions.doPrintStackTrace(Functions.java:1691)

 	at hudson.Functions.doPrintStackTrace(Functions.java:1691)

 	at hudson.Functions.doPrintStackTrace(Functions.java:1691)

 	at hudson.Functions.printThrowable(Functions.java:1675)

 	at hudson.util.ErrorObject.getStackTraceString(ErrorObject.java:51)

 Caused: java.lang.reflect.InvocationTargetException

 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 	at java.lang.reflect.Method.invoke(Method.java:498)

 	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)

 	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)

 	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)

 	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)

 	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)

 	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)

 	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)

 	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)

 	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)

 	at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24)

 	at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)

 	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

 	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

 	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

 	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

 	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

 	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

 	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

 	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

 	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

 	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

 	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

 	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

 	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

 	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:64)

 	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:54)

 	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)

 	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)

 	at org.kohsuke.stapler.ScriptRequestDispatcher.forward(ScriptRequestDispatcher.java:97)

 	at hudson.util.ErrorObject.doDynamic(ErrorObject.java:56)

 	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)

 	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)

 	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)

 	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)

 	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)

 	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:481)

 	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)

 	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)

 	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)

 	at org.kohsuke.stapler.Stapler.service(Stapler.java:240)

 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)

 	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)

 	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

 	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:117)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

 	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

 	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

 	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

 	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

 	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)

 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)

 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)

 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)

 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)

 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)

 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)

 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)

 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)

 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

 	at org.eclipse.jetty.server.Server.handle(Server.java:501)

 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)

 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)

 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)

 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)

 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

 	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)

 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)

 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)

 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)

 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)

 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)

 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)

 	at java.lang.Thread.run(Thread.java:748)
@watchman-rorschach
Copy link
Author

Invalid configuration

@JohnNiang
Copy link
Member

Invalid configuration

Please refer to

@wali97
Copy link

wali97 commented Dec 16, 2022

@watchman-rorschach Hi,did you find a fix for this problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JohnNiang @watchman-rorschach @wali97