Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

che-master-ci job produces broken "dashboard.war" sometimes #14497

Closed
4 of 23 tasks
dmytro-ndp opened this issue Sep 10, 2019 · 17 comments
Closed
4 of 23 tasks

che-master-ci job produces broken "dashboard.war" sometimes #14497

dmytro-ndp opened this issue Sep 10, 2019 · 17 comments
Labels
kind/bug Outline of a bug - must adhere to the bug report template. severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code.

Comments

@dmytro-ndp
Copy link
Contributor

dmytro-ndp commented Sep 10, 2019

Describe the bug

There were at least two times when https://ci.codenvycorp.com/job/che-master-ci/ job produced broken dashboard.war artifact:

Both of the times it were restarting of build after previous build failure: #14519

Because of that, chectl of next channel failed on starting nightly Eclipse Che:

Che server log
Using embedded assembly.
Listening for transport dt_socket at address: 8000
2019-09-10 07:40:14,513[main]             [INFO ] [o.a.c.s.VersionLoggerListener 89]    - Server version:        Apache Tomcat/8.5.35
2019-09-10 07:40:14,515[main]             [INFO ] [o.a.c.s.VersionLoggerListener 91]    - Server built:          Nov 3 2018 17:39:20 UTC
2019-09-10 07:40:14,516[main]             [INFO ] [o.a.c.s.VersionLoggerListener 93]    - Server number:         8.5.35.0
2019-09-10 07:40:14,516[main]             [INFO ] [o.a.c.s.VersionLoggerListener 95]    - OS Name:               Linux
2019-09-10 07:40:14,516[main]             [INFO ] [o.a.c.s.VersionLoggerListener 97]    - OS Version:            4.9.64
2019-09-10 07:40:14,517[main]             [INFO ] [o.a.c.s.VersionLoggerListener 99]    - Architecture:          amd64
2019-09-10 07:40:14,517[main]             [INFO ] [o.a.c.s.VersionLoggerListener 101]   - Java Home:             /usr/lib/jvm/java-1.8-openjdk/jre
2019-09-10 07:40:14,517[main]             [INFO ] [o.a.c.s.VersionLoggerListener 103]   - JVM Version:           1.8.0_191-b12
2019-09-10 07:40:14,518[main]             [INFO ] [o.a.c.s.VersionLoggerListener 105]   - JVM Vendor:            Oracle Corporation
2019-09-10 07:40:14,518[main]             [INFO ] [o.a.c.s.VersionLoggerListener 107]   - CATALINA_BASE:         /home/user/eclipse-che/tomcat
2019-09-10 07:40:14,518[main]             [INFO ] [o.a.c.s.VersionLoggerListener 109]   - CATALINA_HOME:         /home/user/eclipse-che/tomcat
2019-09-10 07:40:14,518[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties
2019-09-10 07:40:14,519[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
2019-09-10 07:40:14,519[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:MaxRAMFraction=2
2019-09-10 07:40:14,519[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UseParallelGC
2019-09-10 07:40:14,519[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:MinHeapFreeRatio=10
2019-09-10 07:40:14,520[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:MaxHeapFreeRatio=20
2019-09-10 07:40:14,520[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:GCTimeRatio=4
2019-09-10 07:40:14,520[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:AdaptiveSizePolicyWeight=90
2019-09-10 07:40:14,521[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UnlockExperimentalVMOptions
2019-09-10 07:40:14,521[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UseCGroupMemoryLimitForHeap
2019-09-10 07:40:14,521[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dsun.zip.disableMemoryMapping=true
2019-09-10 07:40:14,521[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Xms20m
2019-09-10 07:40:14,521[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.docker.network=bridge
2019-09-10 07:40:14,522[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dport.http=8080
2019-09-10 07:40:14,522[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.home=/home/user/eclipse-che/
2019-09-10 07:40:14,523[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.logs.dir=/data/logs
2019-09-10 07:40:14,523[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.logs.level=INFO
2019-09-10 07:40:14,523[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml
2019-09-10 07:40:14,524[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
2019-09-10 07:40:14,524[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
2019-09-10 07:40:14,524[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
2019-09-10 07:40:14,524[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n
2019-09-10 07:40:14,525[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote
2019-09-10 07:40:14,525[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.ssl=false
2019-09-10 07:40:14,525[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
2019-09-10 07:40:14,525[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.local.conf.dir=/etc/conf
2019-09-10 07:40:14,526[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dignore.endorsed.dirs=
2019-09-10 07:40:14,526[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.base=/home/user/eclipse-che//tomcat
2019-09-10 07:40:14,526[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.home=/home/user/eclipse-che//tomcat
2019-09-10 07:40:14,526[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
2019-09-10 07:40:14,654[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 560]  - Initializing ProtocolHandler ["http-nio-8080"]
2019-09-10 07:40:14,667[main]             [INFO ] [o.a.t.util.net.NioSelectorPool 67]   - Using a shared selector for servlet write/read
2019-09-10 07:40:14,693[main]             [INFO ] [o.a.catalina.startup.Catalina 649]   - Initialization processed in 480 ms
2019-09-10 07:40:14,713[main]             [INFO ] [c.m.JmxRemoteLifecycleListener 336]  - The JMX Remote Listener has configured the registry on port [32001] and the server on port [32101] for the [Platform] server
2019-09-10 07:40:14,713[main]             [INFO ] [o.a.c.core.StandardService 416]      - Starting service [Catalina]
2019-09-10 07:40:14,713[main]             [INFO ] [o.a.c.core.StandardEngine 259]       - Starting Servlet Engine: Apache Tomcat/8.5.35
2019-09-10 07:40:15,228[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
2019-09-10 07:40:17,083[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [1,855] ms
2019-09-10 07:40:17,084[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war]
2019-09-10 07:40:17,178[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war] has finished in [94] ms
2019-09-10 07:40:17,179[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
2019-09-10 07:40:17,270[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [91] ms
2019-09-10 07:40:17,273[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
2019-09-10 07:40:17,276[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/dashboard]
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:225)
	at java.util.zip.ZipFile.(ZipFile.java:155)
	at java.util.jar.JarFile.(JarFile.java:166)
	at java.util.jar.JarFile.(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
	at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
	at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
	at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:40:17,278[ost-startStop-1]  [ERROR] [o.a.c.core.ContainerBase 757]        - ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/dashboard]]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@51421ff1]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4907)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5042)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 10 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.WarResourceSet@8bb86e1]
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
	at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:714)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: error in opening zip file
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:142)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
	... 16 common frames omitted
Caused by: java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:225)
	at java.util.zip.ZipFile.(ZipFile.java:155)
	at java.util.jar.JarFile.(JarFile.java:166)
	at java.util.jar.JarFile.(JarFile.java:130)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:139)
	... 17 common frames omitted
2019-09-10 07:40:17,279[ost-startStop-1]  [ERROR] [o.a.c.startup.HostConfig 988]        - Error deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/dashboard]]
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:40:17,280[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war] has finished in [6] ms
2019-09-10 07:40:17,280[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
2019-09-10 07:40:17,282[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/docs]
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:225)
	at java.util.zip.ZipFile.(ZipFile.java:155)
	at java.util.jar.JarFile.(JarFile.java:166)
	at java.util.jar.JarFile.(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
	at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
	at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
	at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:40:17,283[ost-startStop-1]  [ERROR] [o.a.c.core.ContainerBase 757]        - ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/docs]]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@2a40e9d6]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4907)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5042)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 10 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.WarResourceSet@4adff0a4]
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
	at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:714)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: error in opening zip file
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:142)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
	... 16 common frames omitted
Caused by: java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:225)
	at java.util.zip.ZipFile.(ZipFile.java:155)
	at java.util.jar.JarFile.(JarFile.java:166)
	at java.util.jar.JarFile.(JarFile.java:130)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:139)
	... 17 common frames omitted
2019-09-10 07:40:17,286[ost-startStop-1]  [ERROR] [o.a.c.startup.HostConfig 988]        - Error deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/docs]]
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:40:17,286[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war] has finished in [6] ms
2019-09-10 07:40:17,287[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2019-09-10 07:40:17,289[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/api]
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:225)
	at java.util.zip.ZipFile.(ZipFile.java:155)
	at java.util.jar.JarFile.(JarFile.java:166)
	at java.util.jar.JarFile.(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
	at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
	at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
	at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:40:17,293[ost-startStop-1]  [ERROR] [o.a.c.core.ContainerBase 757]        - ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@3f64db54]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4907)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5042)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 10 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.WarResourceSet@7e18eef1]
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
	at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:714)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: error in opening zip file
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:142)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
	... 16 common frames omitted
Caused by: java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:225)
	at java.util.zip.ZipFile.(ZipFile.java:155)
	at java.util.jar.JarFile.(JarFile.java:166)
	at java.util.jar.JarFile.(JarFile.java:130)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:139)
	... 17 common frames omitted
2019-09-10 07:40:17,294[ost-startStop-1]  [ERROR] [o.a.c.startup.HostConfig 988]        - Error deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:40:17,294[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [7] ms
2019-09-10 07:40:17,297[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 588]  - Starting ProtocolHandler ["http-nio-8080"]
2019-09-10 07:40:17,305[main]             [INFO ] [o.a.catalina.startup.Catalina 700]   - Server startup in 2611 ms
 

https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/che-regular-tests/view/K8S/job/CHE-7-Happy-Path-Tests/1606/console

00:02:41.104 + /usr/local/bin/chectl server:start --k8spodreadytimeout=180000 -t=/mnt/hudson_workspace/workspace/CHE-7-Happy-Path-Tests/deploy/ --platform=minikube --listr-renderer=verbose
00:02:41.105 [02:23:40] ✈️  Minikube preflight checklist [started]
00:02:41.106 [02:23:40] Verify if kubectl is installed [started]
00:02:41.107 [02:23:40] Verify if kubectl is installed [completed]
00:02:41.107 [02:23:40] Verify if minikube is installed [started]
00:02:41.108 [02:23:40] Verify if minikube is installed [completed]
00:02:41.108 [02:23:40] Verify if minikube is running [started]
00:02:41.109 [02:23:40] Verify if minikube is running [completed]
00:02:41.109 [02:23:40] Start minikube [started]
00:02:41.110 [02:23:40] Start minikube [skipped]
00:02:41.110 [02:23:40] → Minikube is already running.
00:02:41.110 [02:23:40] Verify if minikube ingress addon is enabled [started]
00:02:41.111 [02:23:40] Verify if minikube ingress addon is enabled [completed]
00:02:41.112 [02:23:40] Enable minikube ingress addon [started]
00:02:41.112 [02:23:40] Enable minikube ingress addon [skipped]
00:02:41.113 [02:23:40] → Ingress addon is already enabled.
00:02:41.113 [02:23:40] Retrieving minikube IP and domain for ingress URLs [started]
00:02:41.114 [02:23:40] Retrieving minikube IP and domain for ingress URLs...10.0.148.57.nip.io. [title changed]
00:02:41.115 [02:23:40] Retrieving minikube IP and domain for ingress URLs...10.0.148.57.nip.io. [completed]
00:02:41.115 [02:23:40] ✈️  Minikube preflight checklist [completed]
00:02:41.116 [02:23:40] 🏃‍  Running Helm to install Che [started]
00:02:41.116 [02:23:40] Verify if helm is installed [started]
00:02:41.117 [02:23:40] Verify if helm is installed [completed]
00:02:41.118 [02:23:40] Create Tiller Role Binding [started]
00:02:41.198 [02:23:40] Create Tiller Role Binding...done. [title changed]
00:02:41.199 [02:23:40] Create Tiller Role Binding...done. [completed]
00:02:41.200 [02:23:40] Create Tiller Service Account [started]
00:02:41.328 [02:23:40] Create Tiller Service Account...done. [title changed]
00:02:41.329 [02:23:40] Create Tiller Service Account...done. [completed]
00:02:41.329 [02:23:40] Create Tiller RBAC [started]
00:02:41.498 [02:23:40] Create Tiller RBAC [completed]
00:02:41.498 [02:23:40] Create Tiller Service [started]
00:02:49.567 [02:23:48] Create Tiller Service...done. [title changed]
00:02:49.569 [02:23:48] Create Tiller Service...done. [completed]
00:02:49.570 [02:23:48] Preparing Che Helm Chart [started]
00:02:49.571 [02:23:48] Preparing Che Helm Chart...done. [title changed]
00:02:49.573 [02:23:48] Preparing Che Helm Chart...done. [completed]
00:02:49.574 [02:23:48] Updating Helm Chart dependencies [started]
00:02:54.570 [02:23:53] Updating Helm Chart dependencies...done. [title changed]
00:02:54.572 [02:23:53] Updating Helm Chart dependencies...done. [completed]
00:02:54.573 [02:23:53] Deploying Che Helm Chart [started]
00:02:55.192 [02:23:54] Deploying Che Helm Chart...done. [title changed]
00:02:55.193 [02:23:54] Deploying Che Helm Chart...done. [completed]
00:02:55.195 [02:23:54] 🏃‍  Running Helm to install Che [completed]
00:02:55.197 [02:23:54] ✅  Post installation checklist [started]
00:02:55.197 [02:23:54] Devfile registry pod bootstrap [started]
00:02:55.198 [02:23:54] scheduling [started]
00:02:55.231 [02:23:54] scheduling...done. [title changed]
00:02:55.232 [02:23:54] scheduling...done. [completed]
00:02:55.232 [02:23:54] downloading images [started]
00:03:03.347 [02:24:02] downloading images...done. [title changed]
00:03:03.348 [02:24:02] downloading images...done. [completed]
00:03:03.349 [02:24:02] starting [started]
00:03:06.898 [02:24:06] starting...done. [title changed]
00:03:06.899 [02:24:06] starting...done. [completed]
00:03:06.900 [02:24:06] Devfile registry pod bootstrap [completed]
00:03:06.901 [02:24:06] Plugin registry pod bootstrap [started]
00:03:06.903 [02:24:06] scheduling [started]
00:03:06.904 [02:24:06] scheduling...done. [title changed]
00:03:06.905 [02:24:06] scheduling...done. [completed]
00:03:06.906 [02:24:06] downloading images [started]
00:03:06.912 [02:24:06] downloading images...done. [title changed]
00:03:06.913 [02:24:06] downloading images...done. [completed]
00:03:06.914 [02:24:06] starting [started]
00:03:10.469 [02:24:09] starting...done. [title changed]
00:03:10.470 [02:24:09] starting...done. [completed]
00:03:10.470 [02:24:09] Plugin registry pod bootstrap [completed]
00:03:10.472 [02:24:09] Che pod bootstrap [started]
00:03:10.472 [02:24:09] scheduling [started]
00:03:10.474 [02:24:09] scheduling...done. [title changed]
00:03:10.475 [02:24:09] scheduling...done. [completed]
00:03:10.476 [02:24:09] downloading images [started]
00:03:10.477 [02:24:09] downloading images...done. [title changed]
00:03:10.478 [02:24:09] downloading images...done. [completed]
00:03:10.479 [02:24:09] starting [started]
00:06:12.658 [02:27:12] starting [failed]
00:06:12.659 [02:27:12] → ERR_TIMEOUT: Timeout set to pod ready timeout 180000
00:06:12.659 [02:27:12] Che pod bootstrap [failed]
00:06:12.660 [02:27:12] → ERR_TIMEOUT: Timeout set to pod ready timeout 180000
00:06:12.661 [02:27:12] ✅  Post installation checklist [failed]
00:06:12.661 [02:27:12] → ERR_TIMEOUT: Timeout set to pod ready timeout 180000
00:06:12.663 Error: ERR_TIMEOUT: Timeout set to pod ready timeout 180000
00:06:12.664     at KubeHelper.<anonymous> (/usr/local/lib/chectl/lib/api/kube.js:538:19)
00:06:12.665     at Generator.next (<anonymous>)
00:06:12.665     at fulfilled (/usr/local/lib/chectl/node_modules/tslib/tslib.js:107:62)

Che version

  • latest
  • nightly
  • other: please specify

Steps to reproduce

Expected behavior

Runtime

  • kubernetes (include output of kubectl version)
  • Openshift (include output of oc version)
  • minikube (include output of minikube version and kubectl version)
  • minishift (include output of minishift version and oc version)
  • docker-desktop + K8S (include output of docker version and kubectl version)
  • other: (please specify)

Screenshots

Installation method

  • chectl
  • che-operator
  • minishift-addon
  • I don't know

Environment

  • my computer
    • Windows
    • Linux
    • macOS
  • Cloud
    • Amazon
    • Azure
    • GCE
    • other (please specify)
  • CRW Central CI
@dmytro-ndp dmytro-ndp added kind/bug Outline of a bug - must adhere to the bug report template. area/chectl Issues related to chectl, the CLI of Che labels Sep 10, 2019
@benoitf
Copy link
Contributor

benoitf commented Sep 10, 2019

I don't think it's a chectl issue

@benoitf benoitf added status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. and removed area/chectl Issues related to chectl, the CLI of Che labels Sep 10, 2019
@benoitf
Copy link
Contributor

benoitf commented Sep 10, 2019

@dmytro-ndp please provide the logs of che-server

@benoitf
Copy link
Contributor

benoitf commented Sep 10, 2019

$ docker run --rm -it --entrypoint=bash eclipse/che-server:nightly
bash-4.4# cd /home/user/eclipse-che/
bash-4.4# cd tomcat/webapps/
bash-4.4# more api.war 
<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>
bash-4.4# ls -la
total 6816
drwxrwxrwx    1 root     root          4096 Sep  9 19:04 .
drwxrwxrwx    1 root     root          4096 Sep  9 19:04 ..
-rw-rw-r--    1 root     root       6207764 Sep  9 19:04 ROOT.war
-rw-rw-r--    1 root     root           162 Sep  9 19:04 api.war
-rw-rw-r--    1 root     root           162 Sep  9 19:04 dashboard.war
-rw-rw-r--    1 root     root           162 Sep  9 19:04 docs.war
-rw-rw-r--    1 root     root        710909 Sep  9 19:04 swagger.war
-rw-rw-r--    1 root     root         33849 Sep  9 19:04 workspace-loader.war

so probably the image is not built correctly due to maven requests issues

@benoitf benoitf removed their assignment Sep 10, 2019
@rhopp
Copy link
Contributor

rhopp commented Sep 10, 2019

Che server logs:

$ kubectl logs che-849b855dd8-4496q -n che
Using embedded assembly.
Listening for transport dt_socket at address: 8000
2019-09-10 07:45:01,141[main]             [INFO ] [o.a.c.s.VersionLoggerListener 89]    - Server version:        Apache Tomcat/8.5.35
2019-09-10 07:45:01,151[main]             [INFO ] [o.a.c.s.VersionLoggerListener 91]    - Server built:          Nov 3 2018 17:39:20 UTC
2019-09-10 07:45:01,151[main]             [INFO ] [o.a.c.s.VersionLoggerListener 93]    - Server number:         8.5.35.0
2019-09-10 07:45:01,152[main]             [INFO ] [o.a.c.s.VersionLoggerListener 95]    - OS Name:               Linux
2019-09-10 07:45:01,152[main]             [INFO ] [o.a.c.s.VersionLoggerListener 97]    - OS Version:            4.15.0
2019-09-10 07:45:01,153[main]             [INFO ] [o.a.c.s.VersionLoggerListener 99]    - Architecture:          amd64
2019-09-10 07:45:01,159[main]             [INFO ] [o.a.c.s.VersionLoggerListener 101]   - Java Home:             /usr/lib/jvm/java-1.8-openjdk/jre
2019-09-10 07:45:01,160[main]             [INFO ] [o.a.c.s.VersionLoggerListener 103]   - JVM Version:           1.8.0_191-b12
2019-09-10 07:45:01,160[main]             [INFO ] [o.a.c.s.VersionLoggerListener 105]   - JVM Vendor:            Oracle Corporation
2019-09-10 07:45:01,161[main]             [INFO ] [o.a.c.s.VersionLoggerListener 107]   - CATALINA_BASE:         /home/user/eclipse-che/tomcat
2019-09-10 07:45:01,161[main]             [INFO ] [o.a.c.s.VersionLoggerListener 109]   - CATALINA_HOME:         /home/user/eclipse-che/tomcat
2019-09-10 07:45:01,162[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties
2019-09-10 07:45:01,162[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
2019-09-10 07:45:01,162[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:MaxRAMFraction=2
2019-09-10 07:45:01,163[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UseParallelGC
2019-09-10 07:45:01,163[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:MinHeapFreeRatio=10
2019-09-10 07:45:01,164[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:MaxHeapFreeRatio=20
2019-09-10 07:45:01,164[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:GCTimeRatio=4
2019-09-10 07:45:01,164[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:AdaptiveSizePolicyWeight=90
2019-09-10 07:45:01,165[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UnlockExperimentalVMOptions
2019-09-10 07:45:01,165[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UseCGroupMemoryLimitForHeap
2019-09-10 07:45:01,165[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dsun.zip.disableMemoryMapping=true
2019-09-10 07:45:01,166[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Xms20m
2019-09-10 07:45:01,166[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.docker.network=bridge
2019-09-10 07:45:01,166[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dport.http=8080
2019-09-10 07:45:01,166[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.home=/home/user/eclipse-che/
2019-09-10 07:45:01,167[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.logs.dir=/data/logs
2019-09-10 07:45:01,168[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.logs.level=INFO
2019-09-10 07:45:01,168[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml
2019-09-10 07:45:01,168[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
2019-09-10 07:45:01,169[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
2019-09-10 07:45:01,170[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
2019-09-10 07:45:01,170[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n
2019-09-10 07:45:01,170[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote
2019-09-10 07:45:01,176[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.ssl=false
2019-09-10 07:45:01,177[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
2019-09-10 07:45:01,177[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.local.conf.dir=/etc/conf
2019-09-10 07:45:01,177[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dignore.endorsed.dirs=
2019-09-10 07:45:01,178[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.base=/home/user/eclipse-che//tomcat
2019-09-10 07:45:01,178[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.home=/home/user/eclipse-che//tomcat
2019-09-10 07:45:01,178[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
2019-09-10 07:45:01,290[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 560]  - Initializing ProtocolHandler ["http-nio-8080"]
2019-09-10 07:45:01,301[main]             [INFO ] [o.a.t.util.net.NioSelectorPool 67]   - Using a shared selector for servlet write/read
2019-09-10 07:45:01,310[main]             [INFO ] [o.a.catalina.startup.Catalina 649]   - Initialization processed in 562 ms
2019-09-10 07:45:01,328[main]             [INFO ] [c.m.JmxRemoteLifecycleListener 336]  - The JMX Remote Listener has configured the registry on port [32001] and the server on port [32101] for the [Platform] server
2019-09-10 07:45:01,329[main]             [INFO ] [o.a.c.core.StandardService 416]      - Starting service [Catalina]
2019-09-10 07:45:01,329[main]             [INFO ] [o.a.c.core.StandardEngine 259]       - Starting Servlet Engine: Apache Tomcat/8.5.35
2019-09-10 07:45:01,992[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
2019-09-10 07:45:03,745[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [1,752] ms
2019-09-10 07:45:03,746[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
2019-09-10 07:45:03,751[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/dashboard]
java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:103)
        at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
        at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
        at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
        at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
        at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
        at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
        at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
        at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:45:03,754[ost-startStop-1]  [ERROR] [o.a.c.core.ContainerBase 757]        - ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/dashboard]]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@3ff46973]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4907)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5042)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 10 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.WarResourceSet@599d1a39]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
        at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:714)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: error in opening zip file
        at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:142)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        ... 16 common frames omitted
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
        at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:139)
        ... 17 common frames omitted
2019-09-10 07:45:03,755[ost-startStop-1]  [ERROR] [o.a.c.startup.HostConfig 988]        - Error deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/dashboard]]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:45:03,755[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war] has finished in [9] ms
2019-09-10 07:45:03,756[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2019-09-10 07:45:03,759[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/api]
java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:103)
        at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
        at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
        at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
        at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
        at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
        at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
        at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
        at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:45:03,760[ost-startStop-1]  [ERROR] [o.a.c.core.ContainerBase 757]        - ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@514c210]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4907)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5042)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 10 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.WarResourceSet@19b79fa4]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
        at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:714)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: error in opening zip file
        at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:142)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        ... 16 common frames omitted
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
        at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:139)
        ... 17 common frames omitted
2019-09-10 07:45:03,761[ost-startStop-1]  [ERROR] [o.a.c.startup.HostConfig 988]        - Error deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:45:03,761[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [5] ms
2019-09-10 07:45:03,762[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
2019-09-10 07:45:03,764[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/docs]
java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:103)
        at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
        at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
        at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
        at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
        at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
        at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
        at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
        at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:45:03,766[ost-startStop-1]  [ERROR] [o.a.c.core.ContainerBase 757]        - ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/docs]]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@35958e34]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4907)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5042)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 10 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.WarResourceSet@4d85dbf6]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
        at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:714)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: error in opening zip file
        at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:142)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        ... 16 common frames omitted
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
        at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:139)
        ... 17 common frames omitted
2019-09-10 07:45:03,767[ost-startStop-1]  [ERROR] [o.a.c.startup.HostConfig 988]        - Error deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/docs]]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-09-10 07:45:03,767[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war] has finished in [5] ms
2019-09-10 07:45:03,768[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
2019-09-10 07:45:03,868[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [99] ms
2019-09-10 07:45:03,868[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war]
2019-09-10 07:45:04,098[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1020]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war] has finished in [230] ms
2019-09-10 07:45:04,104[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 588]  - Starting ProtocolHandler ["http-nio-8080"]
2019-09-10 07:45:04,120[main]             [INFO ] [o.a.catalina.startup.Catalina 700]   - Server startup in 2808 ms

@dmytro-ndp dmytro-ndp changed the title chectl failed on start of nighlty Eclipse Che on minikube 1.1.1 Eclipse Che can't deploy "dashboard.war" on start Sep 10, 2019
@benoitf
Copy link
Contributor

benoitf commented Sep 10, 2019

it's a fault on CI system that reach a too many requests on maven server when building che

@benoitf
Copy link
Contributor

benoitf commented Sep 10, 2019

chectl server:start --platform=minikube
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Verify if minikube ingress addon is enabled
    ↓ Enable minikube ingress addon [skipped]
      → Ingress addon is already enabled.
    ✔ Retrieving minikube IP and domain for ingress URLs...192.168.99.134.nip.io.
  ✔ 🏃‍  Running Helm to install Che
    ✔ Verify if helm is installed
    ✔ Create Tiller Role Binding...it already exists.
    ✔ Create Tiller Service Account...it already exists.
    ✔ Create Tiller RBAC
    ✔ Create Tiller Service...it already exists.
    ✔ Preparing Che Helm Chart...done.
    ✔ Updating Helm Chart dependencies...done.
    ✔ Deploying Che Helm Chart...done.
  ✔ ✅  Post installation checklist
    ✔ Devfile registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Plugin registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Che pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Retrieving Che Server URL...http://che-che.192.168.99.134.nip.io
    ✔ Che status check
Command server:start has completed successfully.

@benoitf benoitf closed this as completed Sep 10, 2019
@benoitf benoitf removed the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Sep 10, 2019
@tolusha tolusha reopened this Sep 12, 2019
@tolusha
Copy link
Contributor

tolusha commented Sep 12, 2019

I've just faced the issue above

Che server nightly version

2019-09-12 07:32:09,885[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/api]
--
  | java.util.zip.ZipException: error in opening zip file
  | at java.util.zip.ZipFile.open(Native Method)
  | at java.util.zip.ZipFile.<init>(ZipFile.java:225)
  | at java.util.zip.ZipFile.<init>(ZipFile.java:155)
  | at java.util.jar.JarFile.<init>(JarFile.java:166)
  | at java.util.jar.JarFile.<init>(JarFile.java:103)
  | at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)

@tolusha tolusha added the severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code. label Sep 12, 2019
@sunix
Copy link
Contributor

sunix commented Sep 12, 2019

Same here ...

2019-09-12 08:18:55,217[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 957]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2019-09-12 08:18:55,259[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/api]
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:225)
	at java.util.zip.ZipFile.<init>(ZipFile.java:155)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
	at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
	at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
	at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-12 08:18:55,292[ost-startStop-1]  [ERROR] [o.a.c.core.ContainerBase 757]        - ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@5a3d8fc0]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4907)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5042)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 10 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.WarResourceSet@4c3c6231]
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
	at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:714)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: error in opening zip file
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:142)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
	... 16 common frames omitted
Caused by: java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:225)
	at java.util.zip.ZipFile.<init>(ZipFile.java:155)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:130)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
	at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
	at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:139)
	... 17 common frames omitted
2019-09-12 08:18:55,294[ost-startStop-1]  [ERROR] [o.a.c.startup.HostConfig 988]        - Error deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]]
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-12 08:18:55,297[ost-startStop-1]  [INFO ] [

@dmytro-ndp
Copy link
Contributor Author

dmytro-ndp commented Sep 12, 2019

Rebuild of che-server:nightly image on CI fixed the issue previously.
I am trying to rebuild the images, along with @skabashnyuk.

@l0rd
Copy link
Contributor

l0rd commented Sep 12, 2019

Thanks @dmytro-ndp and @skabashnyuk for your help on this.

If I understand correctly the problem is that file api.war is 162k and its content is:

<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>

Do you happen to know why this happens? And why for api.war only?

@dmytro-ndp
Copy link
Contributor Author

dmytro-ndp commented Sep 12, 2019

@l0rd: there was broken artifact found when deploying dashboard.war:

2019-09-12 08:18:55,259[ost-startStop-1]  [ERROR] [o.a.c.startup.ContextConfig 744]     - Exception fixing docBase for context [/api]
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:225)
	at java.util.zip.ZipFile.<init>(ZipFile.java:155)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
	at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:130)
	at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:609)
	at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:742)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

There is something similar happened here #14519

Actually, rebuild of maven artifacts and che-server image https://ci.codenvycorp.com/view/nightly-builds/job/all-che-docker-images-nightly/1154/ fixed the problem.

@dmytro-ndp
Copy link
Contributor Author

Reopened for further investigation so as it happened twice for the last week.

@dmytro-ndp dmytro-ndp reopened this Sep 12, 2019
@dmytro-ndp
Copy link
Contributor Author

There is something wrong with build of Che by the job https://ci.codenvycorp.com/job/che-master-ci/

@dmytro-ndp dmytro-ndp changed the title Eclipse Che can't deploy "dashboard.war" on start che-master-ci job produces broken "dashboard.war" sometimes Sep 12, 2019
@skabashnyuk
Copy link
Contributor

skabashnyuk commented Sep 12, 2019

the job is ok. after this pr #14518 this job will fail if it saw inconsistencies in artifact checksums.

@l0rd
Copy link
Contributor

l0rd commented Sep 12, 2019

@skabashnyuk you mean that it will fail earlier (at build time vs runtime) but it may still fail right?

@skabashnyuk
Copy link
Contributor

you mean that it will fail earlier (at build time vs runtime) but it may still fail right?

yes. on build time. It should reduce to minimum chances to have inconsistent artifacts.

@l0rd
Copy link
Contributor

l0rd commented Sep 13, 2019

I guess we can close this one then and eventually re-open a new one if we see some build error.

@l0rd l0rd closed this as completed Sep 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Outline of a bug - must adhere to the bug report template. severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code.
Projects
None yet
Development

No branches or pull requests

7 participants