Multi-user Che with IBM Cloud Private's OIDC server failing due to self-signed certs #12863

johnmcollier · 2019-03-11T18:06:42Z

Description

I'm trying to get multi-user Che running on IBM Cloud Private (ICP) 3.1.2, using ICP's own OIDC server as the OIDC provider. The server is running at https://<master-ip>:8443/oidc/endpoint/OP. I've made sure to set the following values in the Che helm chart:

customOidcProvider: "https://<master-ip>:8443/oidc/endpoint/OP"
tls:
  enabled: true
  useCertManager: true
  useStaging: false
  secretName: che-tls

However, when I install Che, the following errors are shown in the Che pod's logs:

1) Error injecting constructor, java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://<master-ip>:8443/oidc/endpoint/OP/.well-known/openid-configuration
  at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:70)
  at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.class(KeycloakSettings.java:53)
  while locating org.eclipse.che.multiuser.keycloak.server.KeycloakSettings
    for the 1st parameter of org.eclipse.che.multiuser.keycloak.server.KeycloakProfileRetriever.<init>(KeycloakProfileRetriever.java:39)
  at org.eclipse.che.multiuser.keycloak.server.KeycloakProfileRetriever.class(KeycloakProfileRetriever.java:32)
  while locating org.eclipse.che.multiuser.keycloak.server.KeycloakProfileRetriever
    for the 1st parameter of org.eclipse.che.multiuser.keycloak.server.dao.KeycloakProfileDao.<init>(KeycloakProfileDao.java:38)
  while locating org.eclipse.che.multiuser.keycloak.server.dao.KeycloakProfileDao
  while locating org.eclipse.che.api.user.server.spi.ProfileDao
    for the 2nd parameter of org.eclipse.che.multiuser.keycloak.server.KeycloakUserManager.<init>(KeycloakUserManager.java:58)
  at org.eclipse.che.multiuser.keycloak.server.KeycloakUserManager.class(KeycloakUserManager.java:58)
  while locating org.eclipse.che.multiuser.keycloak.server.KeycloakUserManager
  while locating org.eclipse.che.multiuser.api.account.personal.PersonalAccountUserManager
  while locating org.eclipse.che.api.user.server.UserManager
Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://<master-ip>:8443/oidc/endpoint/OP/.well-known/openid-configuration
	at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings.<init>(KeycloakSettings.java:104)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakSettings$$FastClassByGuice$$e0d0786b.newInstance(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at org.everrest.guice.servlet.EverrestGuiceContextListener.getInjector(EverrestGuiceContextListener.java:140)
	at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45)
	at org.everrest.guice.servlet.EverrestGuiceContextListener.contextInitialized(EverrestGuiceContextListener.java:85)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The last error makes it seem like its failing on the self-signed cert that ICP uses for OIDC. So I guess I have two questions:

Has anyone had luck integrating Che with ICP's OIDC server? If so, how?
Has anyone been able to run a multi-user Che with a self-signed certificate for TLS?

I'm wondering if I need to update the che-tls secret? Maybe have it contain the self-signed certs? I'm not sure.

The text was updated successfully, but these errors were encountered:

tetchel · 2019-03-11T18:10:21Z

Looks like #12634 is the issue for supporting self-signed tls certs

ghost · 2019-03-12T13:44:28Z

@johnmcollier you need to add self signed cert to workspace master.

It looks like there's no such an in k8s helm charts, but here's OpenShift template: https://github.com/eclipse/che/blob/master/deploy/openshift/templates/che-server-template.yaml#L158

So, you need to create a secret called self-signed-certificate and ca.crt needs to contain cert, as well as env to Che deployment. Che server entrypoint script will add it to Java trust store on start up https://github.com/eclipse/che/blob/master/dockerfiles/che/entrypoint.sh#L293

Hope that helps!

johnmcollier · 2019-03-12T15:01:09Z

@eivantsov Thanks a ton! I'll give that a try

johnmcollier · 2019-03-12T22:22:05Z

@eivantsov Thanks! Your suggestion worked and the self-signed cert can now be used.

I'm now running into a separate issue when I try to access the Che dashboard (having made sure to register the OIDC endpoint before installing):

The OAuth service provider could not redirect the request because the redirect URI was not valid. Contact your system administrator to resolve the problem.

The URL shown in my browser is: https://<master-ip>:8443/oidc/endpoint/OP/authorize?client_id=che-public&redirect_uri=https%3A%2F%2Fche-che.<proxy-ip>.nip.io%2Fdashboard%2F&state=fb6ed178-27e9-4495-88cb-5fd6076e0e35&response_mode=fragment&response_type=code&scope=openid%20email%20profile&nonce=14f5c699-a152-42a6-b284-2938a9c5c1de

It seems to be complaining about the redirect URI specifically?

I set "redirect_uris":["https://che-che.<proxy-ip>.nip.io/*"], when registering the OIDC client, are there any other redirect/callback URIs that I need to add for Che?

Edit: What's interesting is that if I change the URL to:
https://<master-ip>:8443/oidc/endpoint/OP/authorize?client_id=che-public&redirect_uri=https://che-che.<proxy-ip>.nip.io/dashboard&state=fb6ed178-27e9-4495-88cb-5fd6076e0e35&response_mode=fragment&response_type=code&scope=openid%20email%20profile&nonce=14f5c699-a152-42a6-b284-2938a9c5c1de

I can access the OIDC login page just fine. The redirect afterward still fails, as it's using the previous URL. It seems like the OIDC server doesn't like something to do with the redirect_uri's encoding?

johnmcollier · 2019-03-13T02:35:48Z

As a follow up, I ended up needing to set "allow_regexp_redirects": true and setting redirect_uris to a regex matching the encoded redirect_uri when registering the OIDC client for Che. With that, I'm able to log in.

I now get stuck in some kind of redirect loop after logging in. But that seems to be a separate problem. It looks like it's appending code=X1sfByo5JhaGJDa3oOngV3PcYTsfSE&state=1d465b80-3fe2-4e23-8de2-a95a3e88d1a2 to the URL each time it redirects, infinitely.

ghost · 2019-03-13T06:12:32Z

@johnmcollier with Keycloak we set the following redirect uri in client settings (my local example):

http://codeready-che.172.17.0.1.xip.io/*

What is <proxy IP> in your case? In my example it's docker0 local IP.

johnmcollier · 2019-03-15T04:26:29Z

@eivantsov Turns out the issue was with the server running the OIDC OP on my cluster. There was a bug preventing it from parsing redirect_uri's with parameters attached to it. As a temporary workaround, using a regexp matching for the redirect_uri worked for me.

Closing this issue as my issues with the self-signed certs, and redirect_uris have been resolved.

However, I'm still unable to get multi-user working, but it now appears to be a separate issue with how the client script is interacting with my cluster's OIDC OP (related to what my colleague @tetchel highlighted in #12881). So we're looking at modifying OIDCKeycloak.js instead.

Anyways, thanks for your help! Much appreciated!

davidwindell · 2019-07-11T01:45:00Z

@johnmcollier I've just tried to setup Gitlab as an OpenID connect provider for Che. I'm also seeing the same thing with ?code being added to the URL and an infinite redirect loop. Do I need wildcard support to be implemented to fix this do you think? https://gitlab.com/gitlab-org/gitlab-ce/issues/48707

johnmcollier · 2019-07-11T20:30:44Z

@davidwindell I believe so, that's what I ended up needing to use to get it working.

ghost added the kind/question Questions that haven't been identified as being feature requests or bugs. label Mar 12, 2019

johnmcollier mentioned this issue Mar 12, 2019

Add option to Che deployment.yaml to allow self-signed certificates for OIDC #12874

Merged

johnmcollier closed this as completed Mar 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-user Che with IBM Cloud Private's OIDC server failing due to self-signed certs #12863

Multi-user Che with IBM Cloud Private's OIDC server failing due to self-signed certs #12863

johnmcollier commented Mar 11, 2019 •

edited

Loading

tetchel commented Mar 11, 2019

ghost commented Mar 12, 2019 •

edited by ghost

Loading

johnmcollier commented Mar 12, 2019

johnmcollier commented Mar 12, 2019 •

edited

Loading

johnmcollier commented Mar 13, 2019 •

edited

Loading

ghost commented Mar 13, 2019

johnmcollier commented Mar 15, 2019

davidwindell commented Jul 11, 2019

johnmcollier commented Jul 11, 2019

Multi-user Che with IBM Cloud Private's OIDC server failing due to self-signed certs #12863

Multi-user Che with IBM Cloud Private's OIDC server failing due to self-signed certs #12863

Comments

johnmcollier commented Mar 11, 2019 • edited Loading

Description

tetchel commented Mar 11, 2019

ghost commented Mar 12, 2019 • edited by ghost Loading

johnmcollier commented Mar 12, 2019

johnmcollier commented Mar 12, 2019 • edited Loading

johnmcollier commented Mar 13, 2019 • edited Loading

ghost commented Mar 13, 2019

johnmcollier commented Mar 15, 2019

davidwindell commented Jul 11, 2019

johnmcollier commented Jul 11, 2019

johnmcollier commented Mar 11, 2019 •

edited

Loading

ghost commented Mar 12, 2019 •

edited by ghost

Loading

johnmcollier commented Mar 12, 2019 •

edited

Loading

johnmcollier commented Mar 13, 2019 •

edited

Loading