OpenShift/OKD cluster - Use with external/outside domain #716

eduardolucioac · 2021-06-24T16:19:58Z

eduardolucioac
Jun 24, 2021

I have an OpenShift/OKD cluster already up and running and using the following domain...

certain.domain

Within this domain I have these (among others) features working...

console-openshift-console.apps.okdsubdom.certain.domain
oauth-openshift.apps.okdsubdom.certain.domain
prometheus-k8s-openshift-monitoring.apps.okdsubdom.certain.domain
grafana-openshift-monitoring.apps.okdsubdom.certain.domain
alertmanager-main-openshift-monitoring.apps.okdsubdom.certain.domain

QUESTION: I own this other (external/outside) domain...

another.domain

.... and I need the resources mentioned above to respond to it as below...

console-openshift-console.apps.okdsubdom.another.domain
oauth-openshift.apps.okdsubdom.another.domain
prometheus-k8s-openshift-monitoring.apps.okdsubdom.another.domain
grafana-openshift-monitoring.apps.okdsubdom.another.domain
alertmanager-main-openshift-monitoring.apps.okdsubdom.another.domain

.

IMPORTANT: Our main difficulty these days is the Web Console because it doesn't use relative paths for its URLs. That is, its (absolute path) URLs are in the domain certain.domain and we can't find a simple way to make it use and consume the URL another.domain.

Thanks! =D

sgreene570 · 2021-06-24T17:25:18Z

sgreene570
Jun 24, 2021

I believe you can customize the console route in 4.6
https://docs.openshift.com/container-platform/4.6/web_console/customizing-the-web-console.html#customizing-the-web-console-url_customizing-web-console

And I think the functionality that youre describing is in the pipeline:
openshift/enhancements#577

6 replies

eduardolucioac Jun 29, 2021
Author

Using the command...

oc patch consoles.operator.openshift.io cluster --type merge --patch '{"spec":{"route":{"hostname":"console.okdsubdom.another.domain"}}}'

... we can now access application (Web Console) in domain "okdsubdom.another.domain" as in this example...

curl -k https://console-openshift-console.okdsubdom.another.domain

But now... A new problem has arisen "The application is currently not serving requests at this endpoint." ...

[root@okd4-services ~]# curl -k https://console-openshift-console.okdsubdom.another.domain
[...]
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>

NOTE: This problem only occurs if I try to access the application in the domain "okdsubdom.another.domain". Accessing by the domain "okdsubdom.certain.domain" everything works normally.

sgreene570 Jun 29, 2021

@eduardolucioac im not sure if I follow, are you saying that patching the console route to the new hostname does not work, or are you saying that you can only get the console route to work for one hostname at a time? If the latter, you could try making a second route manually for the console (use the operator-generated route as an example, and just swap out the hostname field).

eduardolucioac Jun 29, 2021
Author

"you saying that patching the console route to the new hostname does not work" -> Yes it works. But, the following error happens if I try to access this new path: "The application is currently not serving requests at this endpoint.".

"or are you saying that you can only get the console route to work for one hostname at a time?" -> No. Both work, but the new path has the above flaw.

"you could try making a second route manually for the console (use the operator-generated route as an example, and just swap out the hostname field)" -> Could you give more details and information about what you meant?

Thanks! =D

eduardolucioac Jun 29, 2021
Author

for the console, create DNS entries for your custom console fqdn in both the internal and external DNS zones, and run:
$ oc patch consoles.operator.openshift.io cluster --type=merge \
    --patch '{"spec":{"route":{"hostname":"console.acme.org","secret":{"name":"console-tls-cert"}}}}'

@LorbusChris

Can you give us more information about this part of the setup (below)?

"secret":{"name":"console-tls-cert"}

NOTE: I'm trying to find out if the "Application is not available" problem is related to this configuration.

LorbusChris Jun 29, 2021
Maintainer

openshift/release#18946 might contain some pointers useful to you.

eduardolucioac · 2021-06-29T16:14:45Z

eduardolucioac
Jun 29, 2021
Author

From google group for OKD ("mr.o...@gmail.com")

Hi
It seems that you add replace a route and didn't add new domain (okdsubdom.another.domain) to your ingress controllers.
you can simply add new ingressController for the new domain (okdsubdom.another.domain) and add another route for the console and it's going to work.
For better understanding of what you need to create you can see the existing ingressController for initial domain (okdsubdom.certain.domain).

1 reply

eduardolucioac Jun 29, 2021
Author

From google group for OKD

mr.o...@gmail.com

"It seems that you add replace a route" -> I don't know if I got it wrong, but I actually added one more route.

"add new ingressController for the new domain" -> Could you give us some more details about this?

eduardolucioac · 2021-06-29T19:33:44Z

eduardolucioac
Jun 29, 2021
Author

For some reason this works...

curl -k -H "Host: console-openshift-console.apps.okdsubdom.certain.domain" https://console-openshift-console.apps.okdsubdom.another.domain

It could be a clue.

[Ref(s).: http://5.9.10.113/55655089/openshift-and-istio-gateway-traffic-configuration-in-order-to-access-from-a-doma ]

10 replies

eduardolucioac Jun 30, 2021
Author

@sgreene570

Please @sgreene570 take a look at this link and not the "BUG REPORT INFORMATION"...

"Customizing the web console URL" doesn't work.

NOTE: For some reason I didn't understand @vrutkovs considered it as "spam". So I'm trying to get in touch with the community (or with himself) to try to understand what conduct was infringed.

IMPORTANT: I considered doing a "BUG REPORT", because I followed all the procedures in the documentation, but the procedure still didn't work.

eduardolucioac Jun 30, 2021
Author

@eduardolucioac please don't open duplicate discussions.

Did you see my comment in the above thread? Was it helpful?

openshift/release#18946 might contain some pointers useful to you.

Ow! Sorry! I'm really having a hard time solving this problem. So it ended up going unnoticed...

I will give you some feedback! =D

Thanks a lot!

"please don't open duplicate discussions." -> It really wasn't my intention. I considered doing a "BUG REPORT", because I followed all the procedures in the documentation, but the procedure still didn't work.

FYI: After finishing this whole process I will consolidate the entire installation script (it's already around 3000 lines) for UPI/bare metal in the repository okd_bare_metal and return to the community the time that it made available to me.

eduardolucioac Jun 30, 2021
Author

oc get route -n openshift-console -o yaml
oc get ingresscontroller default -n openshift-ingress-operator -o yaml
I think

@sgreene570

There are the info...

Thanks! =D

[root@okd4-services ~]# oc get ingresscontroller default -n openshift-ingress-operator -o yaml
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  creationTimestamp: "2021-06-25T18:37:32Z"
  finalizers:
  - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
  generation: 1
  managedFields:
  - apiVersion: operator.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"ingresscontroller.operator.openshift.io/finalizer-ingresscontroller": {}
      f:spec:
        .: {}
        f:replicas: {}
      f:status:
        .: {}
        f:availableReplicas: {}
        f:conditions: {}
        f:domain: {}
        f:endpointPublishingStrategy:
          .: {}
          f:type: {}
        f:observedGeneration: {}
        f:selector: {}
        f:tlsProfile:
          .: {}
          f:ciphers: {}
          f:minTLSVersion: {}
    manager: ingress-operator
    operation: Update
    time: "2021-06-25T18:45:24Z"
  name: default
  namespace: openshift-ingress-operator
  resourceVersion: "999607"
  selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
  uid: bbc27e6d-682a-4d24-98f6-b3b8606286ad
spec:
  replicas: 2
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2021-06-25T18:45:02Z"
    reason: Valid
    status: "True"
    type: Admitted
  - lastTransitionTime: "2021-06-25T19:30:15Z"
    status: "True"
    type: PodsScheduled
  - lastTransitionTime: "2021-06-29T00:46:11Z"
    message: The deployment has Available status condition set to True
    reason: DeploymentAvailable
    status: "True"
    type: DeploymentAvailable
  - lastTransitionTime: "2021-06-29T00:46:11Z"
    message: Minimum replicas requirement is met
    reason: DeploymentMinimumReplicasMet
    status: "True"
    type: DeploymentReplicasMinAvailable
  - lastTransitionTime: "2021-06-29T00:46:39Z"
    message: All replicas are available
    reason: DeploymentReplicasAvailable
    status: "True"
    type: DeploymentReplicasAllAvailable
  - lastTransitionTime: "2021-06-25T18:45:24Z"
    message: The configured endpoint publishing strategy does not include a managed load balancer
    reason: EndpointPublishingStrategyExcludesManagedLoadBalancer
    status: "False"
    type: LoadBalancerManaged
  - lastTransitionTime: "2021-06-25T18:45:24Z"
    message: No DNS zones are defined in the cluster dns config.
    reason: NoDNSZones
    status: "False"
    type: DNSManaged
  - lastTransitionTime: "2021-06-29T00:46:11Z"
    status: "True"
    type: Available
  - lastTransitionTime: "2021-06-29T00:46:41Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2021-06-29T00:46:40Z"
    message: Canary route checks for the default ingress controller are successful
    reason: CanaryChecksSucceeding
    status: "True"
    type: CanaryChecksSucceeding
  domain: apps.mbr.okd.local
  endpointPublishingStrategy:
    type: HostNetwork
  observedGeneration: 1
  selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default
  tlsProfile:
    ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - ECDHE-ECDSA-AES128-GCM-SHA256
    - ECDHE-RSA-AES128-GCM-SHA256
    - ECDHE-ECDSA-AES256-GCM-SHA384
    - ECDHE-RSA-AES256-GCM-SHA384
    - ECDHE-ECDSA-CHACHA20-POLY1305
    - ECDHE-RSA-CHACHA20-POLY1305
    - DHE-RSA-AES128-GCM-SHA256
    - DHE-RSA-AES256-GCM-SHA384
    minTLSVersion: VersionTLS12

[root@okd4-services ~]# oc get route -n openshift-console -o yaml
apiVersion: v1
items:
- apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    annotations:
      openshift.io/host.generated: "true"
    creationTimestamp: "2021-06-25T18:49:31Z"
    labels:
      app: console
    managedFields:
    - apiVersion: route.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:app: {}
        f:spec:
          f:port:
            .: {}
            f:targetPort: {}
          f:tls:
            .: {}
            f:insecureEdgeTerminationPolicy: {}
            f:termination: {}
          f:to:
            f:kind: {}
            f:name: {}
            f:weight: {}
          f:wildcardPolicy: {}
      manager: console
      operation: Update
      time: "2021-06-25T18:49:31Z"
    - apiVersion: route.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:ingress: {}
      manager: openshift-router
      operation: Update
      time: "2021-06-25T19:29:53Z"
    name: console
    namespace: openshift-console
    resourceVersion: "39788"
    uid: 749de7bd-ab64-44d3-becd-c82038de97ed
  spec:
    host: console-openshift-console.apps.mbr.okd.local
    port:
      targetPort: https
    tls:
      insecureEdgeTerminationPolicy: Redirect
      termination: reencrypt
    to:
      kind: Service
      name: console
      weight: 100
    wildcardPolicy: None
  status:
    ingress:
    - conditions:
      - lastTransitionTime: "2021-06-25T19:29:53Z"
        status: "True"
        type: Admitted
      host: console-openshift-console.apps.mbr.okd.local
      routerCanonicalHostname: apps.mbr.okd.local
      routerName: default
      wildcardPolicy: None
- apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    annotations:
      include.release.openshift.io/ibm-cloud-managed: "true"
      include.release.openshift.io/self-managed-high-availability: "true"
      include.release.openshift.io/single-node-developer: "true"
    creationTimestamp: "2021-06-25T18:49:05Z"
    managedFields:
    - apiVersion: route.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:include.release.openshift.io/ibm-cloud-managed: {}
            f:include.release.openshift.io/self-managed-high-availability: {}
            f:include.release.openshift.io/single-node-developer: {}
        f:spec:
          f:port:
            .: {}
            f:targetPort: {}
          f:tls:
            .: {}
            f:insecureEdgeTerminationPolicy: {}
            f:termination: {}
          f:to:
            f:kind: {}
            f:name: {}
            f:weight: {}
          f:wildcardPolicy: {}
      manager: cluster-version-operator
      operation: Update
      time: "2021-06-25T18:49:05Z"
    - apiVersion: route.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:ingress: {}
      manager: openshift-router
      operation: Update
      time: "2021-06-25T19:29:52Z"
    name: downloads
    namespace: openshift-console
    resourceVersion: "39765"
    uid: c380e750-475f-4429-a51f-2dde23123a84
  spec:
    host: downloads-openshift-console.apps.mbr.okd.local
    port:
      targetPort: http
    tls:
      insecureEdgeTerminationPolicy: Redirect
      termination: edge
    to:
      kind: Service
      name: downloads
      weight: 100
    wildcardPolicy: None
  status:
    ingress:
    - conditions:
      - lastTransitionTime: "2021-06-25T19:29:52Z"
        status: "True"
        type: Admitted
      host: downloads-openshift-console.apps.mbr.okd.local
      routerCanonicalHostname: apps.mbr.okd.local
      routerName: default
      wildcardPolicy: None
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

LorbusChris Jun 30, 2021
Maintainer

It's not a bug, it's a configuration error by the user, and it's a duplicate and it also shows up in our notifications, thus it may be marked as spam.
If it does indeed turn out to be a bug, we can repurpose this discussion here for it until a proper bug is filed on the Bugzilla.

Please refrain from trying to contact or tag individual developers for cases like this. Also, please be mindful about the fact that the help you receive here comes at the cost of time that contributors spend to help you. It is not LOW/ZERO COST to us (to quote from your GH description). Maybe that needs to be clarified in the Community Charter.

We'd definitely appreciate any PRs with improvements to our docs, or guides. If you're interested to contribute to development and/or testing, you're also very welcome to join the regular OKD WG meetings.

eduardolucioac Jun 30, 2021
Author

@LorbusChris

Please understand my words as positively as possible. =D

"It's not a bug, it's a configuration error by the user, and it's a duplicate and it also shows up in our notifications, thus it may be marked as spam."
[...]
"Please refrain from trying to contact or tag individual developers for cases like this. Also, please be mindful about the fact that the help you receive here comes at the cost of time that contributors spend to help you."

Yes, I agree with everything you said and I have already retracted all of this in the email group and in part here. However, I ask you once more to understand and forgive me. It was not my intention to cause inconvenience, but it's really not easy to find help about OKD 4.X in other communities so I ran out of options.

About "the fact that the help you receive here comes at the cost oftime that contributors spend to help you" I have been working for years with free software and have hundreds of free contributions to people who use free software including in the company I work that produces free software - may not really seem, but I have 110 responses and 3726 of total reputation, only on Stack Exchange, there is much more out there. If you look at my GH you will see that all my projects, without exception, are free software projects. So... Yes! I own "services" for free software and am very proud of it. =D

In this post, for example, I demonstrate how to install the OpenShift OKD/Origin.

It is not LOW/ZERO COST to us (to quote from your GH description). Maybe that needs to be clarified in the Community Charter.

We know this very well even because we make our contributions to the free software world, as this is an ecosystem. In this ecosystem we contribute and we enjoy, is in this way that free software is maintained and spread. All of us here, without exception, enjoy and contribute to free software (including its bundled services). This is a concept known and accepted by everyone who enjoys the benefits of free software. Finally, let it be clear that, yes, free software is a business model. A better, more efficient and more inclusive business model.

We'd definitely appreciate any PRs with improvements to our docs, or guides. If you're interested to contribute to development and/or testing, you're also very welcome to join the regular OKD WG meetings.

Everything in my power to help I will do. You can be sure. Take a look at the "FYI" I wrote for you.

Finally, thank you very much! I'm very grateful for everyone's efforts and I'm very grateful for everything they helped us, including you @LorbusChris !

=D

fortinj66 · 2021-06-30T15:13:51Z

fortinj66
Jun 30, 2021

It looks like if you do not provide "secret":{"name":"<secret-name>"} the console operator will not create the appropriate new route needed to connect to the console.

secret-name is created with:

oc create secret tls console-tls --key=./<cert key> --cert=./<cert pem> -n openshift-config

where cert is the certificate for your domain

once you get that completed the console operator will create a new route:

Initial State

[root@os-utils-d03 poc-upi-4.7]# oc get  consoles.operator.openshift.io cluster -o yaml | yq e .spec -
logLevel: Normal
managementState: Managed
operatorLogLevel: Normal

[root@os-utils-d03 poc-upi-4.7]# oc get routes
NAME        HOST/PORT                                                    PATH   SERVICES    PORT    TERMINATION          WILDCARD
console     console-openshift-console.apps.poc-upi-4.os.maeagle.corp            console     https   reencrypt/Redirect   None
downloads   downloads-openshift-console.apps.poc-upi-4.os.maeagle.corp          downloads   http    edge/Redirect        None

Without secret

[root@os-utils-d03 poc-upi-4.7]#
[root@os-utils-d03 poc-upi-4.7]# oc patch consoles.operator.openshift.io cluster --type merge --patch '{"spec":{"route":{"hostname":"console-upi.maeagle.corp"}}}'
console.operator.openshift.io/cluster patched
[root@os-utils-d03 poc-upi-4.7]#
[root@os-utils-d03 poc-upi-4.7]# oc get  consoles.operator.openshift.io cluster -o yaml | yq e .spec -
logLevel: Normal
managementState: Managed
operatorLogLevel: Normal
route:
  hostname: console-upi.maeagle.corp
[root@os-utils-d03 poc-upi-4.7]# oc get routes
NAME        HOST/PORT                                                    PATH   SERVICES    PORT    TERMINATION          WILDCARD
console     console-openshift-console.apps.poc-upi-4.os.maeagle.corp            console     https   reencrypt/Redirect   None
downloads   downloads-openshift-console.apps.poc-upi-4.os.maeagle.corp          downloads   http    edge/Redirect        None

With secret

[root@os-utils-d03 poc-upi-4.7]#
[root@os-utils-d03 poc-upi-4.7]# oc patch consoles.operator.openshift.io cluster --type merge --patch '{"spec":{"route":{"hostname":"console-upi.maeagle.corp","secret":{"name":"console-tls"}}}}'
console.operator.openshift.io/cluster patched
[root@os-utils-d03 poc-upi-4.7]#
[root@os-utils-d03 poc-upi-4.7]# oc get routes
NAME             HOST/PORT                                                    PATH   SERVICES           PORT                    TERMINATION          WILDCARD
console          console-openshift-console.apps.poc-upi-4.os.maeagle.corp            console-redirect   custom-route-redirect   edge/Redirect        None
console-custom   console-upi.maeagle.corp                                            console            https                   reencrypt/Redirect   None
downloads        downloads-openshift-console.apps.poc-upi-4.os.maeagle.corp          downloads          http                    edge/Redirect        No

7 replies

eduardolucioac Jul 1, 2021
Author

@fortinj66

Your answer was just perfect! Thank you very much!

Now I can access via the new route, however the old route no longer exists (it is redirecting). This apparently does not match with the model situations presented by you...

[root@okd4-services ~]# curl -k https://console-openshift-console.apps.mbr.okd.local
<a href="https://console-openshift-console.brlight.net/">Moved Permanently</a>.

Another thing that seems strange to me is the output of the command below, which also does not match the model situations presented by you...

[root@okd4-services ~]# oc get routes
No resources found in openshift-console-operator namespace.

QUESTION: Is this situation normal?

[]'s

fortinj66 Jul 1, 2021

Use the openshift-console namespace

fortinj66 Jul 1, 2021

And yes, the original route will now redirect.

eduardolucioac Jul 1, 2021
Author

@fortinj66

With this command below I should display all routes. Now it looks normal and consistent with what you presented...

[root@okd4-services ~]# oc get routes --all-namespaces
NAMESPACE                  NAME                HOST/PORT                                                   PATH   SERVICES            PORT                    TERMINATION            WILDCARD
openshift-authentication   oauth-openshift     oauth-openshift.apps.mbr.okd.local                                 oauth-openshift     6443                    passthrough/Redirect   None
openshift-console          console             console-openshift-console.apps.mbr.okd.local                       console-redirect    custom-route-redirect   edge/Redirect          None
openshift-console          console-custom      console-openshift-console.brlight.net                              console             https                   reencrypt/Redirect     None
openshift-console          downloads           downloads-openshift-console.apps.mbr.okd.local                     downloads           http                    edge/Redirect          None
openshift-ingress-canary   canary              canary-openshift-ingress-canary.apps.mbr.okd.local                 ingress-canary      8080                    edge/Redirect          None
openshift-monitoring       alertmanager-main   alertmanager-main-openshift-monitoring.apps.mbr.okd.local          alertmanager-main   web                     reencrypt/Redirect     None
openshift-monitoring       grafana             grafana-openshift-monitoring.apps.mbr.okd.local                    grafana             https                   reencrypt/Redirect     None
openshift-monitoring       prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.mbr.okd.local             prometheus-k8s      web                     reencrypt/Redirect     None
openshift-monitoring       thanos-querier      thanos-querier-openshift-monitoring.apps.mbr.okd.local             thanos-querier      web                     reencrypt/Redirect     None

FURTHER QUESTION: My question may seem silly, as I still know little about OpenShift (OKD)... But why does my oc get routes command show a different output than yours?

Thanks! =D

eduardolucioac Jul 1, 2021
Author

openshift-console

Works like a charm! =D

Thanks! =D

[root@okd4-services ~]# oc project openshift-console
Now using project "openshift-console" on server "https://api.mbr.okd.local:6443".
[root@okd4-services ~]# oc get routes
NAME             HOST/PORT                                        PATH   SERVICES           PORT                    TERMINATION          WILDCARD
console          console-openshift-console.apps.mbr.okd.local            console-redirect   custom-route-redirect   edge/Redirect        None
console-custom   console-openshift-console.brlight.net                   console            https                   reencrypt/Redirect   None
downloads        downloads-openshift-console.apps.mbr.okd.local          downloads          http                    edge/Redirect        None

eduardolucioac · 2021-06-30T16:01:50Z

eduardolucioac
Jun 30, 2021
Author

Did you see my comment in the above thread? Was it helpful?

openshift/release#18946 might contain some pointers useful to you.

Just to consolidate the information provided by @LorbusChris...

The link JoinBuildFarm.md: Add info about secure route exposal is an unapproved merge request...

This change extends the doc with info on how to expose
registry/console/api/apps with Let's Encrypt TLS certs.
[...]

This is the md (complete file) with the changes Join Build Farm .

Observe the Secure and expose routes section.

Very good! I believe it will help!
Let's hope they approve.

0 replies

eduardolucioac · 2021-07-02T00:48:41Z

eduardolucioac
Jul 2, 2021
Author

Below is a complete script to customizing the 'Web Console" URL...

The purpose of this post is to consolidate everyone's contributions into one answer. Thank's guys! =D

Adjust the OKD cluster's internal DNS according to the templates...

NOTE: These models and examples are compatible with the DNS BIND 9. Adjust to your reality.

Define a new DNS zone configuration...

MODEL

read -r -d '' FILE_CONTENT << 'HEREDOC'
BEGIN
$TTL    604800
@   IN  SOA <OKD_SERVICES_NM>.<ANOTHER_DOMAIN>. admin.<ANOTHER_DOMAIN>. (
        1       ; Serial
        604800  ; Refresh
        86400   ; Retry
        2419200 ; Expire
        604800  ; Negative Cache TTL
)

; Name servers - "NS" records.
    IN  NS  <OKD_SERVICES_NM>

; Name servers - "A" records.
<OKD_SERVICES_NM>.<ANOTHER_DOMAIN>. IN A <OKD_SERVICES_IP_OKD_INT_LAN>

; Openshift internal cluster IPs - "A" records.
*.<ANOTHER_DOMAIN>.           IN  A   <OKD_SERVICES_IP_OKD_INT_LAN>
<CONSOLE_NM>.<ANOTHER_DOMAIN>.    IN  A   <OKD_SERVICES_IP_OKD_INT_LAN>

END
HEREDOC
echo -n "${FILE_CONTENT:6:-3}" > '/etc/named/zones/fw.<ANOTHER_DOMAIN>'

EXAMPLE

read -r -d '' FILE_CONTENT << 'HEREDOC'
BEGIN
$TTL    604800
@   IN  SOA okd-services.another.dm. admin.another.dm. (
        1       ; Serial
        604800  ; Refresh
        86400   ; Retry
        2419200 ; Expire
        604800  ; Negative Cache TTL
)

; Name servers - "NS" records.
    IN  NS  okd-services

; Name servers - "A" records.
okd-services.another.dm. IN A 10.3.0.14

; Openshift internal cluster IPs - "A" records.
*.another.dm.           IN  A   10.3.0.14
console-nm.another.dm.    IN  A   10.3.0.14

END
HEREDOC
echo -n "${FILE_CONTENT:6:-3}" > '/etc/named/zones/fw.another.dm'

Configure the DNS to consume the new zone...

MODEL

read -r -d '' FILE_CONTENT << 'HEREDOC'
BEGIN
zone "<OKD_INT_DM>" {
    type master;
    file "/etc/named/zones/db.<OKD_INT_DM>"; // Zone file path.
};

zone "<OKD_INT_LAN_FST_3_OCTS_INV>.in-addr.arpa" {
    type master;
    file "/etc/named/zones/db.<OKD_INT_LAN_FST_3_OCTS>"; // <OKD_INT_LAN_FST_3_OCTS>.0/24 subnet.
};

zone "<ANOTHER_DOMAIN>" {
    type master;
    file "/etc/named/zones/fw.<ANOTHER_DOMAIN>"; // Zone file path.
};

END
HEREDOC
echo -n "${FILE_CONTENT:6:-3}" > '/etc/named/named.conf.local'

EXAMPLE

read -r -d '' FILE_CONTENT << 'HEREDOC'
BEGIN
zone "okdint.dm" {
    type master;
    file "/etc/named/zones/db.okdint.dm"; // Zone file path.
};

zone "0.3.10.in-addr.arpa" {
    type master;
    file "/etc/named/zones/db.10.3.0"; // 10.3.0.0/24 subnet.
};

zone "another.dm" {
    type master;
    file "/etc/named/zones/fw.another.dm"; // Zone file path.
};

END
HEREDOC
echo -n "${FILE_CONTENT:6:-3}" > '/etc/named/named.conf.local'

Create a self-signed SSL certificate (for your LAN)...

MODEL

mkdir -p "/etc/tls/<ANOTHER_DOMAIN>"
cd "/etc/tls/<ANOTHER_DOMAIN>"
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
    -keyout <ANOTHER_DOMAIN>.key -out <ANOTHER_DOMAIN>.crt -subj "/CN=<OKD_SERVICES_NM>.<ANOTHER_DOMAIN>" \
    -addext "subjectAltName=DNS:*.<ANOTHER_DOMAIN>,IP:<OKD_SERVICES_IP>"

EXAMPLE

mkdir -p "/etc/tls/another.dm"
cd "/etc/tls/another.dm"
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
    -keyout another.dm.key -out another.dm.crt -subj "/CN=okd-services.another.dm" \
    -addext "subjectAltName=DNS:*.another.dm,IP:10.2.0.19"

[Ref(s).: https://stackoverflow.com/questions/10175812/how-to-generate-a-self-signed-ssl-certificate-using-openssl ,
https://stackoverflow.com/questions/54081191/self-signed-ssl-certificate-on-private-ip-is-invalid-in-chrome ]

Create a new secret for your new "Web Console" domain...

MODEL

cd "/etc/tls/<ANOTHER_DOMAIN>"
oc create secret tls tsl-<ANOTHER_DOMAIN_WITH_HYPHEN> --key=./<YOUR_DOMAIN>.key --cert=./<YOUR_DOMAIN>.crt -n openshift-config

EXAMPLE

cd "/etc/tls/another.dm"
oc create secret tls tsl-another-dm --key=./another.dm.key --cert=./another.dm.crt -n openshift-config

Configure your cluster to use your new "Web Console" domain...

oc edit consoles.operator.openshift.io

MODEL

spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  route:
    hostname: <CONSOLE_NM>.<ANOTHER_DOMAIN>
    secret:
      name: tsl-<ANOTHER_DOMAIN_WITH_HYPHEN>

EXAMPLE

spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  route:
    hostname: console-nm.another.dm
    secret:
      name: tsl-another-dm

[Ref(s).: https://docs.openshift.com/container-platform/4.6/web_console/customizing-the-web-console.html#customizing-the-web-console-url_customizing-web-console ]

TIPS:

Show "openshift-console" routes...

oc project openshift-console
oc get routes

Test the new route...

curl -k https://console-nm.another.dm

Check original route...

curl -k https://console-openshift-console.apps.mbr.okdint.dm

NOTE: The "console-openshift-console.apps" value is a "standard" and the "mbr.okdint.dm" value is arbitrary. Then adjust the above route to your original settings.

NOTE: The expected output for the above command is: <a href="https://console-nm.another.dm/">Moved Permanently</a>. .

Done! =D

1 reply

binnes Jul 5, 2021
Maintainer

Please note it is better to use the ingress operator now. Web console config via the console-operator config is now deprecated : https://docs.okd.io/latest/web_console/customizing-the-web-console.html#customizing-the-web-console-url_customizing-web-console

eduardolucioac · 2021-07-02T01:44:50Z

eduardolucioac
Jul 2, 2021
Author

@fortinj66

I can now access the Web Console using the new domain (in this case "mydomain.net") and login using the procedures presented by you.

However, I noticed that several other routes were not made available for the new domain "mydomain.net" (see command below), especially the routes...

https://grafana-openshift-monitoring.apps.mbr.okd.local
https://prometheus-k8s-openshift-monitoring.apps.mbr.okd.local
https://alertmanager-main-openshift-monitoring.apps.mbr.okd.local

... which are links accessible from the Web Console itself

[root@okd4-services ~]# oc get --all-namespaces -o jsonpath='{range .items[*]}{range .status.ingress[*]}{.host}{"\n"}{end}{end}' routes
oauth-openshift.apps.mbr.okd.local
console-openshift-console.apps.mbr.okd.local
console-openshift-console.mydomain.net
downloads-openshift-console.apps.mbr.okd.local
canary-openshift-ingress-canary.apps.mbr.okd.local
alertmanager-main-openshift-monitoring.apps.mbr.okd.local
grafana-openshift-monitoring.apps.mbr.okd.local
prometheus-k8s-openshift-monitoring.apps.mbr.okd.local
thanos-querier-openshift-monitoring.apps.mbr.okd.local

QUESTION: How did you solve these cases?

Thanks! =D

3 replies

fortinj66 Jul 2, 2021

You'll need to create new routes using the new domain... Should be fairly easy...

https://docs.okd.io/latest/networking/routes/route-configuration.html

eduardolucioac Jul 2, 2021
Author

You'll need to create new routes using the new domain... Should be fairly easy...

https://docs.okd.io/latest/networking/routes/route-configuration.html

@fortinj66

Regarding what you guided me I have an essential question... Once I create these new routes, will the "Web Console" start using them? Let me explain... These are URLs that are in Web Console links...

https://grafana-openshift-monitoring.apps.mbr.okd.local
https://prometheus-k8s-openshift-monitoring.apps.mbr.okd.local
https://alertmanager-main-openshift-monitoring.apps.mbr.okd.local

... which are in the domain "okd.local". So if I create these new routes in the domain "mydomain.net" the Web Console will use them?

Thanks! =D

PLUS: When you say "You'll need to create new routes using the new domain" you're specifically talking about this Creating a route through an Ingress object ?

binnes Jul 5, 2021
Maintainer

Hello @eduardolucioac,

When you refer to Web Console links are you talking about links available in the top menu bar (3x3 square grid icon) or links for applications such as shown in the Administrator -> Networking -> routes pages or on the developer Topology view?

If it is the links in the top menu bar, then these are controlled by the ConsoleLink CRC and will probably need manually updating (https://docs.okd.io/latest/web_console/customizing-the-web-console.html#creating-custom-links_customizing-web-console) after you create the new routes.

eduardolucioac · 2021-07-11T23:15:30Z

eduardolucioac
Jul 11, 2021
Author

The purpose of this post is consolidate all procedures needed to provide (to try...) access to OpenShift (OKD)'s Web components (especially the Web Console) from a domain different from its internal domain (base domain/installation domain).

[]'s

Hi people!

Greetings! =D

Here is a new way to access OpenShift (OKD)'s Web components (especially the Web Console) from a domain different from its internal domain (base domain/installation domain). There is only one small detail to make it 100%. I think this may be interesting for you, so please take a look!

In the procedure described here I achieved successful login on a new URL (new domain) made available for the "oauth-openshift" resource and I was also able to normally use all other Web resources in new URLs (same new domain).

The only problem is that the Web Console and its components ("alertmanager", "grafana", "prometheus", etc) always redirect to the OLD "oauth-openshift"'s URL when login is required, so we have to adjust the domain manually every time.

To solve this we just need to make the Web resources ("console", "alertmanager", "grafana", "prometheus", etc) redirect to the new URL and not to the old one.

Note that I didn't change the baseDomain I just changed its routes.

As you can see, its has been proven that login to web applications can occur from a domain other than the baseDomain. I believe that the effort you would (dev team) have to expend to allow this would be minimal. Its could be a new feature for OpenShift (OKD)! =D

Anyway, I have to say it was a great effort to reach that answer... =D

NOTE I: There is no other effective answer to this on the internet... I've tried every possible channel!
NOTE II: The deploy was made over Bare Metal/UPI (User Provisioned Infrastructure). I basically follow these steps here: Guide: Installing an OKD 4.5 Cluster.

PROCEDURES:

I found all namespaces that have routes...

[root@okd4-services okd_bare_metal]# oc get routes --all-namespaces
NAMESPACE                  NAME                HOST/PORT                                                   PATH   SERVICES            PORT    TERMINATION            WILDCARD
openshift-authentication   oauth-openshift     oauth-openshift.apps.mbr.okd.local                                 oauth-openshift     6443    passthrough/Redirect   None
openshift-console          console             console-openshift-console.apps.mbr.okd.local                       console             https   reencrypt/Redirect     None
openshift-console          downloads           downloads-openshift-console.apps.mbr.okd.local                     downloads           http    edge/Redirect          None
openshift-ingress-canary   canary              canary-openshift-ingress-canary.apps.mbr.okd.local                 ingress-canary      8080    edge/Redirect          None
openshift-monitoring       alertmanager-main   alertmanager-main-openshift-monitoring.apps.mbr.okd.local          alertmanager-main   web     reencrypt/Redirect     None
openshift-monitoring       grafana             grafana-openshift-monitoring.apps.mbr.okd.local                    grafana             https   reencrypt/Redirect     None
openshift-monitoring       prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.mbr.okd.local             prometheus-k8s      web     reencrypt/Redirect     None
openshift-monitoring       thanos-querier      thanos-querier-openshift-monitoring.apps.mbr.okd.local             thanos-querier      web     reencrypt/Redirect     None

I found all the resources related to each namespace and took note of the route configuration paths...

[root@okd4-services okd_bare_metal]# oc -n openshift-authentication get all
[...]
NAME                                       HOST/PORT                            PATH   SERVICES          PORT   TERMINATION            WILDCARD
route.route.openshift.io/oauth-openshift   oauth-openshift.apps.mbr.okd.local          oauth-openshift   6443   passthrough/Redirect   None

[root@okd4-services okd_bare_metal]# oc -n openshift-console get all
[...]
NAME                                 HOST/PORT                                        PATH   SERVICES    PORT    TERMINATION          WILDCARD
route.route.openshift.io/console     console-openshift-console.apps.mbr.okd.local            console     https   reencrypt/Redirect   None
route.route.openshift.io/downloads   downloads-openshift-console.apps.mbr.okd.local          downloads   http    edge/Redirect        None

[root@okd4-services okd_bare_metal]# oc -n openshift-ingress-canary get all
[...]
NAME                              HOST/PORT                                            PATH   SERVICES         PORT   TERMINATION     WILDCARD
route.route.openshift.io/canary   canary-openshift-ingress-canary.apps.mbr.okd.local          ingress-canary   8080   edge/Redirect   None

[root@okd4-services okd_bare_metal]# oc -n openshift-monitoring get all
[...]
NAME                                         HOST/PORT                                                   PATH   SERVICES            PORT    TERMINATION          WILDCARD
route.route.openshift.io/alertmanager-main   alertmanager-main-openshift-monitoring.apps.mbr.okd.local          alertmanager-main   web     reencrypt/Redirect   None
route.route.openshift.io/grafana             grafana-openshift-monitoring.apps.mbr.okd.local                    grafana             https   reencrypt/Redirect   None
route.route.openshift.io/prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.mbr.okd.local             prometheus-k8s      web     reencrypt/Redirect   None
route.route.openshift.io/thanos-querier      thanos-querier-openshift-monitoring.apps.mbr.okd.local             thanos-querier      web     reencrypt/Redirect   None

In each route I updated the host (there are two) property to a new domain...

oc edit -n openshift-console route.route.openshift.io/console
oc edit -n openshift-console route.route.openshift.io/downloads
oc edit -n openshift-ingress-canary route.route.openshift.io/canary
oc edit -n openshift-monitoring route.route.openshift.io/alertmanager-main
oc edit -n openshift-monitoring route.route.openshift.io/grafana
oc edit -n openshift-monitoring route.route.openshift.io/prometheus-k8s
oc edit -n openshift-monitoring route.route.openshift.io/thanos-querier

In other words, I modified something like this...

apiVersion: route.openshift.io/v1
kind: Route
metadata:
[...]
spec:
  host: route-name.apps.mbr.okd.local
[...]
status:
[...]
    host: route-name.apps.mbr.okd.local
[...]

... to something like this...

apiVersion: route.openshift.io/v1
kind: Route
metadata:
[...]
spec:
  host: route-name.apps.mbr.mydomain.net
[...]
status:
[...]
    host: route-name.apps.mbr.mydomain.net
[...]

In the case of route route.route.openshift.io/oauth-openshift, we will not change it. We will create a new route for its oauth-openshift resource.

Open for editing (but we won't edit it) the route configuration for the oauth-openshift resource...

oc edit -n openshift-authentication route.route.openshift.io/oauth-openshift

... , copy its content and edit the following properties according to your reality modifing from something like this...

[...]
metadata:
[...]
  name: oauth-openshift
[...]
spec:
  host: oauth-openshift.apps.mbr.okd.local
[...]
status:
[...]
    host: oauth-openshift.apps.mbr.okd.local
[...]

... to something like this...

[...]
metadata:
[...]
  name: oauth-openshift-mydomain-net
[...]
spec:
  host: oauth-openshift.apps.mbr.mydomain.net
[...]
status:
[...]
    host: oauth-openshift.apps.mbr.mydomain.net
[...]

... and create the new route...

cat <<EOF | oc create -f -
---
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  creationTimestamp: "2021-06-25T18:45:29Z"
  labels:
    app: oauth-openshift
  name: oauth-openshift-mydomain-net
  namespace: openshift-authentication
  resourceVersion: "6304382"
  uid: 44d8fdcd-42eb-40f8-8312-789dc4bb84df
spec:
  host: oauth-openshift.apps.mbr.mydomain.net
  port:
    targetPort: 6443
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: passthrough
  to:
    kind: Service
    name: oauth-openshift
    weight: 100
  wildcardPolicy: None
status:
  ingress:
  - conditions:
    - lastTransitionTime: "2021-07-10T02:46:57Z"
      status: "True"
      type: Admitted
    host: oauth-openshift.apps.mbr.mydomain.net
    routerCanonicalHostname: apps.mbr.okd.local
    routerName: default
    wildcardPolicy: None
EOF

PROBLEM:

The Web Console and its components ("alertmanager", "grafana", "prometheus", etc) always redirect to this URL...

https://oauth-openshift.apps.mbr.okd.local

... when login is required.

NOTE: The "okd.local" is the cluster's base domain (installation).

Is it possible to configure them to calling another URL? Like this URL below?

https://oauth-openshift.apps.mbr.domain.my

PLUS I: As I said in other spaces of our community, it is very common to have cases where the external domain (internet, WAN) of a company is different from its internal domain (private, LAN) or even cases where a company has several domains (eg: company.com , company.org, company.net...). So I say again that not being able to make web resources (especially the Web Console) work in a domain other than its internal domain is something that can become dramatic if we try to expose OpenShift (OKD) to external access (internet, WAN) a cluster already in use if its internal domain (eg: okd.local) has no external equivalent.

PLUS II:
I - We've already tried this procedure Customizing the web console URL , but it doesn't update all routes and does not update URLs in the Web Console and also has the same login issues above;
II - I also tried this procedure Creating a route through an Ingress object , but equally it doesn't seem to me something that will solve the problems pointed out.

Thanks! =D

0 replies

eduardolucioac · 2021-07-24T04:34:34Z

eduardolucioac
Jul 24, 2021
Author

Does anyone know if there are plans to implement features to allow access to OpenShift (OKD)'s Web components (especially the Web Console) from a different domain from its internal domain (base domain/installation domain)? If so, where can I follow this process?

[]'s to everyone! =D

0 replies

OpenShift/OKD cluster - Use with external/outside domain #716

eduardolucioac Jun 24, 2021

Replies: 9 comments · 28 replies

sgreene570 Jun 24, 2021

eduardolucioac Jun 29, 2021 Author

sgreene570 Jun 29, 2021

eduardolucioac Jun 29, 2021 Author

eduardolucioac Jun 29, 2021 Author

LorbusChris Jun 29, 2021 Maintainer

eduardolucioac Jun 29, 2021 Author

eduardolucioac Jun 29, 2021 Author

eduardolucioac Jun 29, 2021 Author

eduardolucioac Jun 30, 2021 Author

eduardolucioac Jun 30, 2021 Author

eduardolucioac Jun 30, 2021 Author

LorbusChris Jun 30, 2021 Maintainer

eduardolucioac Jun 30, 2021 Author

fortinj66 Jun 30, 2021

eduardolucioac Jul 1, 2021 Author

fortinj66 Jul 1, 2021

fortinj66 Jul 1, 2021

eduardolucioac Jul 1, 2021 Author

eduardolucioac Jul 1, 2021 Author

eduardolucioac Jun 30, 2021 Author

eduardolucioac Jul 2, 2021 Author

binnes Jul 5, 2021 Maintainer

eduardolucioac Jul 2, 2021 Author

fortinj66 Jul 2, 2021

eduardolucioac Jul 2, 2021 Author

binnes Jul 5, 2021 Maintainer

eduardolucioac Jul 11, 2021 Author

eduardolucioac Jul 24, 2021 Author

eduardolucioac
Jun 24, 2021

Replies: 9 comments 28 replies

sgreene570
Jun 24, 2021

eduardolucioac Jun 29, 2021
Author

eduardolucioac Jun 29, 2021
Author

eduardolucioac Jun 29, 2021
Author

LorbusChris Jun 29, 2021
Maintainer

eduardolucioac
Jun 29, 2021
Author

eduardolucioac Jun 29, 2021
Author

eduardolucioac
Jun 29, 2021
Author

eduardolucioac Jun 30, 2021
Author

eduardolucioac Jun 30, 2021
Author

eduardolucioac Jun 30, 2021
Author

LorbusChris Jun 30, 2021
Maintainer

eduardolucioac Jun 30, 2021
Author

fortinj66
Jun 30, 2021

eduardolucioac Jul 1, 2021
Author

eduardolucioac Jul 1, 2021
Author

eduardolucioac Jul 1, 2021
Author

eduardolucioac
Jun 30, 2021
Author

eduardolucioac
Jul 2, 2021
Author

binnes Jul 5, 2021
Maintainer

eduardolucioac
Jul 2, 2021
Author

eduardolucioac Jul 2, 2021
Author

binnes Jul 5, 2021
Maintainer

eduardolucioac
Jul 11, 2021
Author

eduardolucioac
Jul 24, 2021
Author