Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to open Eclipse Che Dashboard on minikube v1.25.1 with DevWorkspace enabled #20984

Closed
Tracked by #21094
tolusha opened this issue Jan 5, 2022 · 18 comments
Closed
Tracked by #21094

Failed to open Eclipse Che Dashboard on minikube v1.25.1 with DevWorkspace enabled #20984

tolusha opened this issue Jan 5, 2022 · 18 comments
Labels
area/install Issues related to installation, including offline/air gap and initial setup kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.
Milestone
7.46

Comments

@tolusha
Copy link
Contributor

tolusha commented Jan 5, 2022

Describe the bug

Failed to open

Che version

next (development version)

Steps to reproduce

chectl update next
chectl server:deploy --platform minikube
chectl dashboard:open

Dashboard is not opened. Che server logs contains stacktrace.

Expected behavior

Dashboard is opened

Runtime

minikube

Screenshots

No response

Installation method

chectl/next

Environment

Linux

Eclipse Che Logs

2022-01-05 14:42:16,013[io-8080-exec-10]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2022-01-05 14:42:16
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/admin-che/secrets/workspace-credentials-secret. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:623)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:493)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:475)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:807)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:188)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:155)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:88)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.configurator.CredentialsSecretConfigurator.configure(CredentialsSecretConfigurator.java:43)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.configureNamespace(KubernetesNamespaceFactory.java:570)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.getOrCreate(KubernetesNamespaceFactory.java:334)
	at org.eclipse.che.workspace.infrastructure.kubernetes.provision.NamespaceProvisioner.provision(NamespaceProvisioner.java:42)
	at org.eclipse.che.workspace.infrastructure.kubernetes.api.server.KubernetesNamespaceService.provision(KubernetesNamespaceService.java:95)


### Additional context

_No response_
@tolusha tolusha added kind/bug Outline of a bug - must adhere to the bug report template. team/platform severity/P1 Has a major impact to usage or development of the system. labels Jan 5, 2022
@max-cx
Copy link

max-cx commented Jan 5, 2022 •
edited
Loading

For me, this happened with Che 7.41.1 on minikube v1.24.0, Kubernetes v1.22.3, Docker 20.10.8, chectl 7.41.1, kubectl (client:v1.23.1/server:v1.22.3).

@tolusha
Copy link
Contributor Author

tolusha commented Jan 6, 2022

7.41.1 works fine

@tolusha tolusha changed the title Failed to open Eclipse Che Dashboard Failed to open Eclipse Che Dashboard on minikube with DevWorkspace enabled Jan 26, 2022
@tolusha
Copy link
Contributor Author

tolusha commented Jan 26, 2022

@sparkoo
Could you have a look pls.
It should work out of box but instead we have the similar problem like this #21049

@sparkoo
Copy link
Member

sparkoo commented Jan 28, 2022

hmm I've quickly looked at this, I can reproduce, but I don't know what's wrong. Token looks ok, permissions looks ok. I can't get the secret even if I try to use token with curl. My bet is something with kubernetes apiserver configuration with dex, because it looks like kubernetes does not know the user from the token. Maybe something new in latest minikube versions?

@tolusha
Copy link
Contributor Author

tolusha commented Jan 28, 2022

Which version minikube did you use for testing?
I've tried with minikube version: v1.22.0 and got the same result
che-serer logs:

2022-01-28 12:53:36,911[nio-8080-exec-7]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2022-01-28 12:53:36
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/admin-che/secrets/workspace-credentials-secret. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:623)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:493)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:475)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:807)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:188)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:155)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:88)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.configurator.CredentialsSecretConfigurator.configure(CredentialsSecretConfigurator.java:43)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.configureNamespace(KubernetesNamespaceFactory.java:570)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.getOrCreate(KubernetesNamespaceFactory.java:334)
	at org.eclipse.che.workspace.infrastructure.kubernetes.provision.NamespaceProvisioner.provision(NamespaceProvisioner.java:42)
	at org.eclipse.che.workspace.infrastructure.kubernetes.api.server.KubernetesNamespaceService.provision(KubernetesNamespaceService.java:95)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

Untitled

@sparkoo
Copy link
Member

sparkoo commented Jan 28, 2022 •
edited
Loading

I have v1.25.0

I would suspect this patch eclipse-che/che-operator#1291. Can you test before and after? Or test stable version if it works ok? I think it's something with mismatching oidc configuration. client_id, secret, claim or something like that does not match kubernetes and Che configurations

@tolusha tolusha added area/install Issues related to installation, including offline/air gap and initial setup team/deploy sprint/current and removed team/platform labels Jan 31, 2022
@tolusha tolusha mentioned this issue Jan 31, 2022
Closed
18 tasks
@tolusha tolusha added this to the 7.44 milestone Feb 1, 2022
@tolusha tolusha mentioned this issue Feb 2, 2022
Closed
@tolusha
Copy link
Contributor Author

tolusha commented Feb 2, 2022

Can't reproduce after switching to minikube v1.24.0 and cleanin up old resources

@tolusha tolusha closed this as completed Feb 2, 2022
@tolusha tolusha modified the milestones: 7.44, 7.43 Feb 2, 2022
@l0rd
Copy link
Contributor

l0rd commented Feb 2, 2022

@tolusha I am reproducing it on a fresh minikube v1.25.1. What do you mean with "cleaning up old resources"?

@tolusha
Copy link
Contributor Author

tolusha commented Feb 2, 2022

It works well for v1.24.0
Yes, v1.25.1 is still an issue.

"cleaning up old resources"

I mean rm -rf ./minikube but not sure if it really helped.
But yes, switching to minikube v1.24.0 fixed issue.

I will reopen to adopt to v1.25.x

@tolusha tolusha reopened this Feb 2, 2022
@l0rd l0rd modified the milestones: 7.43, 7.44 Feb 3, 2022
@l0rd
Copy link
Contributor

l0rd commented Feb 3, 2022

Moving to 7.44 as 7.43 has been released.

@barrygear
Copy link

I've been having this issue with clusters created in AWS EKS. Verified that the Dex installation is correct. It creates the namespace with the user name but then fails while checking for the Secret. I've given ClusterAdmin to all forms of claims from Dex and its clearly not using any of them. aws eks 1.21, dex 2.30.0, che 7.43.0, using che operator, and dev-workspace.

@l0rd l0rd changed the title Failed to open Eclipse Che Dashboard on minikube with DevWorkspace enabled Failed to open Eclipse Che Dashboard on minikube v1.25.1 with DevWorkspace enabled Feb 4, 2022
@l0rd l0rd pinned this issue Feb 4, 2022
@tolusha tolusha unpinned this issue Feb 4, 2022
This was referenced Feb 10, 2022
Closed
Closed
@slieer
Copy link

slieer commented Feb 15, 2022

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:39:34Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
minikube version
minikube version: v1.25.1
commit: 3e64b11ed75e56e4898ea85f96b2e4af0301f43d
chectl version
chectl/7.43.0 linux-x64 node-v16.13.2

chectl server:status
Eclipse Che Version : 7.43.0
Eclipse Che Url : https://192.168.49.2.nip.io/dashboard/

image

@dmytro-ndp dmytro-ndp mentioned this issue Feb 15, 2022
Closed
@dmytro-ndp dmytro-ndp added severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code. and removed severity/P1 Has a major impact to usage or development of the system. labels Feb 16, 2022
@tolusha
Copy link
Contributor Author

tolusha commented Feb 17, 2022

We have to wait minikube v1.26.0 to test.
Multiple issues which prevent deployment Eclipse Che on minikube v1.25.x

kubernetes/minikube#13543
kubernetes/minikube#13552
kubernetes/minikube#13519
kubernetes/minikube#13503
kubernetes/minikube#13625

@amol-m-deshpande
Copy link

@tolusha
I have been trying the eclipse che on minikube.
Installation is done

  ✔ Retrieving Che self-signed CA certificate...OK
  ✔ Prepare post installation output...done
  ✔ Show important messages
    ✔ Eclipse Che 'next' has been successfully deployed.
    ✔ Documentation             : https://www.eclipse.org/che/docs/
    ✔ -------------------------------------------------------------------------------
    ✔ Users Dashboard           : https://192.168.49.2.nip.io/dashboard/
    ✔ -------------------------------------------------------------------------------
    ✔ Plug-in Registry          : https://192.168.49.2.nip.io/plugin-registry/v3/
    ✔ Devfile Registry          : https://192.168.49.2.nip.io/devfile-registry/
    ✔ -------------------------------------------------------------------------------
    ✔ Dex user credentials      : che@eclipse.org:admin
    ✔ Dex user credentials      : user1@che:password
    ✔ Dex user credentials      : user2@che:password
    ✔ Dex user credentials      : user3@che:password
    ✔ Dex user credentials      : user4@che:password
    ✔ Dex user credentials      : user5@che:password
    ✔ -------------------------------------------------------------------------------
Command server:deploy has completed successfully in 09:12.

When I go to http://192.168.49.2.nip.io/dashboard/ on chrome, nothing is loading up
for minikube version: 1.24.0 and Eclipse Che Version: next

@tolusha
Copy link
Contributor Author

tolusha commented Feb 17, 2022

yes, v1.24.0 works fine.
The issue with v1.25.x

@amol-m-deshpande
Copy link

amol-m-deshpande commented Feb 17, 2022 •
edited
Loading

@tolusha For minikube v1.24.0, the output shows deploy has completed successfully, but when I go to this https://192.168.49.2.nip.io/dashboard/
The dashboard doesn't load up, I get the blank screen

@tolusha
Copy link
Contributor Author

tolusha commented Feb 17, 2022

hmm. It works fine on my side
Pls check che serverlogs
kubectl logs deploy/che -n eclipse-che any stacktraces ?

@amol-m-deshpande
Copy link

amol-m-deshpande commented Feb 17, 2022 •
edited
Loading

@tolusha here are the logs, unable to load the dashboard

Using embedded assembly in /home/user/eclipse-che.
Found a custom cert. Adding it to java trust store /home/user/cacerts based on /opt/java/openjdk/lib/security/cacerts
Trust this certificate? [no]:  Certificate was added to keystore
Certificate was added to keystore
Imported 1 certificates from /public-certs/dex-ca.ca.crt
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
17-Feb-2022 12:48:03.728 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/10.0.14
17-Feb-2022 12:48:03.732 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Dec 2 2021 22:01:36 UTC
17-Feb-2022 12:48:03.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.14.0
17-Feb-2022 12:48:03.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
17-Feb-2022 12:48:03.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.10.76-linuxkit
17-Feb-2022 12:48:03.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
17-Feb-2022 12:48:03.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /opt/java/openjdk
17-Feb-2022 12:48:03.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.11+9
17-Feb-2022 12:48:03.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            AdoptOpenJDK
17-Feb-2022 12:48:03.750 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.752 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che/tomcat/conf/logging.properties
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMPercentage=85.0
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStore=/home/user/cacerts
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStorePassword=changeit
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
17-Feb-2022 12:48:03.788 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/logs/
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che/tomcat/conf/tomcat-logger.xml
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/home/user/eclipse-che/tomcat/conf/
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
17-Feb-2022 12:48:03.791 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.791 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.791 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che/tomcat/temp
17-Feb-2022 12:48:04.831 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
17-Feb-2022 12:48:04.936 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1901] milliseconds
17-Feb-2022 12:48:05.050 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
17-Feb-2022 12:48:05.051 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.0.14]
17-Feb-2022 12:48:05.253 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
17-Feb-2022 12:48:11.864 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [6,610] ms
17-Feb-2022 12:48:11.866 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2022-02-17 12:48:19,748[main]             [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63]   - This=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory@24a10870 obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@70bc4941 environment={}
2022-02-17 12:48:21,748[main]             [INFO ] [.e.c.a.d.WsMasterServletModule 52]   - Running in native-user mode ...
2022-02-17 12:48:30,261[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-11380, cluster=RemoteSubscriptionChannel, physical address=172.17.0.16:7800
-------------------------------------------------------------------
2022-02-17 12:48:33,840[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-11380: no members discovered after 3474 ms: creating cluster as coordinator
2022-02-17 12:48:33,902[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-61836, cluster=WorkspaceLocks, physical address=172.17.0.16:7801
-------------------------------------------------------------------
2022-02-17 12:48:37,144[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-61836: no members discovered after 3205 ms: creating cluster as coordinator
2022-02-17 12:48:37,157[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-7602, cluster=WorkspaceStateCache, physical address=172.17.0.16:7802
-------------------------------------------------------------------
2022-02-17 12:48:40,250[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-7602: no members discovered after 3027 ms: creating cluster as coordinator
2022-02-17 12:48:40,757[main]             [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 13.3)
2022-02-17 12:48:40,846[main]             [INFO ] [o.f.c.i.util.VersionPrinter 44]      - Flyway 4.2.0 by Boxfuse
2022-02-17 12:48:40,855[main]             [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 13.3)
2022-02-17 12:48:40,960[main]             [INFO ] [i.f.CustomSqlMigrationResolver 158]  - Searching for SQL scripts in locations [classpath:che-schema]
2022-02-17 12:48:41,140[main]             [INFO ] [o.f.c.i.command.DbValidate 44]       - Successfully validated 63 migrations (execution time 00:00.184s)
2022-02-17 12:48:41,223[main]             [INFO ] [o.f.c.i.m.MetaDataTableImpl 44]      - Creating Metadata table: "public"."schema_version"
2022-02-17 12:48:41,321[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Current version of schema "public": << Empty Schema >>
2022-02-17 12:48:41,521[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.0.0.8.1 - 1__init.sql
2022-02-17 12:48:41,907[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.0.0.9.1 - 1__add_index_on_workspace_temporary.sql
2022-02-17 12:48:41,931[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.0.0.9.2 - 2__update_local_links_in_environments.sql
2022-02-17 12:48:42,022[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.2.0.1 - 1__increase_project_attributes_values_length.sql
2022-02-17 12:48:42,108[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.4.0.1 - 1__drop_user_to_account_relation.sql
2022-02-17 12:48:42,144[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.4.0.2 - 2__create_missed_account_indexes.sql
2022-02-17 12:48:42,215[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.6.0.1 - 1__add_exec_agent_where_terminal_agent_is_present.sql
2022-02-17 12:48:42,307[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.7.0.1 - 1__add_factory.sql
2022-02-17 12:48:42,509[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.7.0.2 - 2__remove_match_policy.sql
2022-02-17 12:48:42,535[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.8.0.1 - 1__add_foreigh_key_indexes.sql
2022-02-17 12:48:42,680[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.11.0.1 - 1__optimize_user_search.sql
2022-02-17 12:48:42,731[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.19.0.0.1 - 0.1__add_permissions.sql
2022-02-17 12:48:42,924[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.19.0.0.2 - 0.2__add_resources.sql
2022-02-17 12:48:43,018[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.19.0.0.3 - 0.3__add_organization.sql
2022-02-17 12:48:43,108[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.1 - 1__add_path_to_serverconf.sql
2022-02-17 12:48:43,124[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.2 - 2__rename_agents_to_installers.sql
2022-02-17 12:48:43,144[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.3 - 3__add_installer.sql
2022-02-17 12:48:43,171[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.3.1 - 3.1__remove_old_recipe_permissions.sql
2022-02-17 12:48:43,181[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.4 - 4__remove_old_recipe.sql
2022-02-17 12:48:43,194[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.5 - 5__add_machine_env.sql
2022-02-17 12:48:43,205[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.6 - 6__remove_snapshots.sql
2022-02-17 12:48:43,217[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.7 - 7__add_machine_volumes.sql
2022-02-17 12:48:43,236[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.8 - 8__add_serverconf_attributes.sql
2022-02-17 12:48:43,269[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.9 - 9__increase_externalmachine_env_value_length.sql
2022-02-17 12:48:43,288[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.10 - 10__move_dockerimage_recipe_location_to_content.sql
2022-02-17 12:48:43,313[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.11 - 11__increase_workspace_attributes_values_length.sql
2022-02-17 12:48:43,324[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.12 - 12__remove_stack_sources.sql
2022-02-17 12:48:43,330[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.3.0.1 - 1__add_fk_indexes.sql
2022-02-17 12:48:43,344[main]             [WARN ] [o.f.c.i.dbsupport.JdbcTemplate 48]   - DB: identifier "che_index_factory_on_projects_loaded_action_value_action_entity_id" will be truncated to "che_index_factory_on_projects_loaded_action_value_action_entity" (SQL State: 42622 - Error Code: 0)
2022-02-17 12:48:43,350[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.3.0.1.1 - 1.1__add_fk_indexes.sql
2022-02-17 12:48:43,370[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.4.0.1 - 1__add_workspace_expirations.sql
2022-02-17 12:48:43,383[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.4.0.2 - 2__add_signature_key.sql
2022-02-17 12:48:43,401[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.4.0.3 - 3__add_k8s_runtimes.sql
2022-02-17 12:48:43,453[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.10.0.1 - 1__add_workspace_cfg_attributes.sql
2022-02-17 12:48:43,475[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.10.0.2 - 2__change_signature_key_pair_id.sql
2022-02-17 12:48:43,490[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.11.0.1 - 1__add_signature_key_constraints.sql
2022-02-17 12:48:43,507[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.12.0.1 - 1__rename_project_attributes_values_field.sql
2022-02-17 12:48:43,523[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.15.0.1 - 1__remove_not_null_constraint_from_env_name_fields.sql
2022-02-17 12:48:43,542[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.15.0.2 - 2__add_commands_to_k8s_runtime.sql
2022-02-17 12:48:43,582[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.16.0.1 - 1__increase_workspace_config_attributes_values_length.sql
2022-02-17 12:48:43,593[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.16.0.2 - 2__create_workspace_activity_table.sql
2022-02-17 12:48:43,619[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.16.0.3 - 3__bootstrap_ws_activity_data.sql
2022-02-17 12:48:43,634[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.17.0.1 - 1__convert_enums_to_strings.sql
2022-02-17 12:48:43,657[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.4.0.1 - 1__add_devfile.sql
2022-02-17 12:48:43,773[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.5.0.1 - 1__devfile_command_reference.sql
2022-02-17 12:48:43,792[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.6.0.1 - 1__add_devfile_component_prefs.sql
2022-02-17 12:48:43,824[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.7.0.1 - 1__add_registry_url_to_devfile_component.sql
2022-02-17 12:48:43,838[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.8.0.2.0.1 - 1__devfile_metadata.sql
2022-02-17 12:48:43,846[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.8.0.2.0.2 - 2__devfile_make_some_fields_optional.sql
2022-02-17 12:48:43,855[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.1.0.1 - 1__change_devfile_component_preferences_type.sql
2022-02-17 12:48:43,866[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.1.0.1.1 - 1.1__remove_stack_permissions.sql
2022-02-17 12:48:43,930[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.1.0.2 - 2__remove_stacks.sql
2022-02-17 12:48:43,946[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.2.0.1 - 1__remove_installers.sql
2022-02-17 12:48:43,964[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.4.0.1 - 1__add_devfile_source_sparse_checkout_dir.sql
2022-02-17 12:48:43,976[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.4.0.2 - 2__add_preview_url_to_devfile_command.sql
2022-02-17 12:48:43,995[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.6.0.1 - 1__drop_che_workspace_expiration.sql
2022-02-17 12:48:44,009[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.10.0.1 - 1__add_devfile_plugin_editor_component_cpu_limit_request.sql
2022-02-17 12:48:44,020[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.10.0.2 - 2__add_devfile_plugin_editor_component_ram_request.sql
2022-02-17 12:48:44,029[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.11.0.1 - 1__update_inconsistent_stopped_workspace_activities.sql
2022-02-17 12:48:44,044[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.16.0.1 - 1__add_devfile_component_automount_workspace_secrets.sql
2022-02-17 12:48:44,058[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.20.0.1 - 1__userdevfile.sql
2022-02-17 12:48:44,093[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.20.0.1.1 - 1.1__add_userdevfile_permissions.sql
2022-02-17 12:48:44,131[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.21.0.1 - 1__remove_installers.sql
2022-02-17 12:48:44,145[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.26.0.1 - 1__remove_factory_button_and_image.sql
2022-02-17 12:48:44,164[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Successfully applied 63 migrations to schema "public" (execution time 00:02.942s).
2022-02-17 12:48:46,713[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-13566, cluster=EclipseLinkCommandChannel, physical address=172.17.0.16:7803
-------------------------------------------------------------------
2022-02-17 12:48:49,768[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-13566: no members discovered after 3040 ms: creating cluster as coordinator
2022-02-17 12:48:49,823[main]             [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 182]   - Configured factories for environments: '[kubernetes, no-environment]'
2022-02-17 12:48:49,824[main]             [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 183]   - Registered infrastructure 'kubernetes'
2022-02-17 12:48:49,860[main]             [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 694]   - Infrastructure is tracking 0 active runtimes that need to be stopped
2022-02-17 12:48:49,921[main]             [INFO ] [o.e.c.m.oidc.OIDCInfoProvider 71]    - Retrieving OpenId configuration from endpoint: http://dex.dex:5556/.well-known/openid-configuration
2022-02-17 12:48:50,049[main]             [INFO ] [o.e.c.m.oidc.OIDCInfoProvider 80]    - openid configuration = {issuer=https://dex.192.168.49.2.nip.io, authorization_endpoint=https://dex.192.168.49.2.nip.io/auth, token_endpoint=https://dex.192.168.49.2.nip.io/token, jwks_uri=https://dex.192.168.49.2.nip.io/keys, userinfo_endpoint=https://dex.192.168.49.2.nip.io/userinfo, device_authorization_endpoint=https://dex.192.168.49.2.nip.io/device/code, grant_types_supported=[authorization_code, refresh_token, urn:ietf:params:oauth:grant-type:device_code], response_types_supported=[code], subject_types_supported=[public], id_token_signing_alg_values_supported=[RS256], code_challenge_methods_supported=[S256, plain], scopes_supported=[openid, email, groups, profile, offline_access], token_endpoint_auth_methods_supported=[client_secret_basic, client_secret_post], claims_supported=[iss, sub, aud, iat, exp, email, email_verified, locale, name, preferred_username, at_hash]}
2022-02-17 12:48:50,142[main]             [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36]   - Eclipse Che Api Core: Build info '7.43.0' scmRevision '9a0a73c1500f1398796e157bbb4fdafd19c51c7c' implementationVersion '7.43.0'
2022-02-17 12:48:50,228[main]             [INFO ] [o.e.c.c.metrics.MetricsServer 46]    - Metrics server started at port 8087 successfully 
17-Feb-2022 12:48:50.821 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [38,955] ms
17-Feb-2022 12:48:50.822 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
17-Feb-2022 12:48:51.138 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [316] ms
17-Feb-2022 12:48:51.147 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
17-Feb-2022 12:48:51.210 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [46294] milliseconds

@debovema debovema mentioned this issue Feb 18, 2022
Closed
@tolusha tolusha mentioned this issue Feb 21, 2022
Closed
4 tasks
@tolusha tolusha added severity/P1 Has a major impact to usage or development of the system. and removed severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code. sprint/current team/deploy labels Feb 21, 2022
@benoitf benoitf modified the milestones: 7.44, 7.45 Feb 28, 2022
@benoitf benoitf modified the milestones: 7.45, 7.46 Mar 17, 2022
@tolusha tolusha closed this as completed Apr 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/install Issues related to installation, including offline/air gap and initial setup kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.
Projects
None yet
Milestone
7.46
Development

No branches or pull requests
9 participants
@benoitf @l0rd @sparkoo @dmytro-ndp @slieer @tolusha @barrygear @max-cx @amol-m-deshpande