[ENH] - Handle the case when Traefik is already installed

[mcg1969]

Feature description

I would like to be able to install Nebari on a stock k3s cluster. Unfortunately, by default, it has Traefik already installed, and it seems your terraform doesn't like that. The documentation doesn't seem to allow for this possibility or how to bypass it.

As a temporary workaround I remove the duplicate CRDs, but it would be better if I could avoid having to use the terraform resources that are duplicate in this case.

[terraform]: ╷
[terraform]: │ Error: namespaces "default" already exists
[terraform]: │ 
[terraform]: │   with module.kubernetes-initialization.kubernetes_namespace.main,
[terraform]: │   on modules/initialization/main.tf line 1, in resource "kubernetes_namespace" "main":
[terraform]: │    1: resource "kubernetes_namespace" "main" {
[terraform]: │ 
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.ingress_route,
[terraform]: │   on modules/traefik_crds/main.tf line 1, in resource "kubernetes_manifest" "ingress_route":
[terraform]: │    1: resource "kubernetes_manifest" "ingress_route" {
[terraform]: │ 
[terraform]: │ resource "/ingressroutes.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.ingress_route_tcp,
[terraform]: │   on modules/traefik_crds/main.tf line 200, in resource "kubernetes_manifest" "ingress_route_tcp":
[terraform]: │  200: resource "kubernetes_manifest" "ingress_route_tcp" {
[terraform]: │ 
[terraform]: │ resource "/ingressroutetcps.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.ingress_route_udp,
[terraform]: │   on modules/traefik_crds/main.tf line 348, in resource "kubernetes_manifest" "ingress_route_udp":
[terraform]: │  348: resource "kubernetes_manifest" "ingress_route_udp" {
[terraform]: │ 
[terraform]: │ resource "/ingressrouteudps.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.middleware,
[terraform]: │   on modules/traefik_crds/main.tf line 423, in resource "kubernetes_manifest" "middleware":
[terraform]: │  423: resource "kubernetes_manifest" "middleware" {
[terraform]: │ 
[terraform]: │ resource "/middlewares.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.middlewaretcp,
[terraform]: │   on modules/traefik_crds/main.tf line 1073, in resource "kubernetes_manifest" "middlewaretcp":
[terraform]: │ 1073: resource "kubernetes_manifest" "middlewaretcp" {
[terraform]: │ 
[terraform]: │ resource "/middlewaretcps.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.serverstransports,
[terraform]: │   on modules/traefik_crds/main.tf line 1132, in resource "kubernetes_manifest" "serverstransports":
[terraform]: │ 1132: resource "kubernetes_manifest" "serverstransports" {
[terraform]: │ 
[terraform]: │ resource "/serverstransports.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.tls_option,
[terraform]: │   on modules/traefik_crds/main.tf line 1209, in resource "kubernetes_manifest" "tls_option":
[terraform]: │ 1209: resource "kubernetes_manifest" "tls_option" {
[terraform]: │ 
[terraform]: │ resource "/tlsoptions.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.tls_stores,
[terraform]: │   on modules/traefik_crds/main.tf line 1287, in resource "kubernetes_manifest" "tls_stores":
[terraform]: │ 1287: resource "kubernetes_manifest" "tls_stores" {
[terraform]: │ 
[terraform]: │ resource "/tlsstores.traefik.containo.us" already exists
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: Cannot create resource that already exists
[terraform]: │ 
[terraform]: │   with module.traefik-crds.kubernetes_manifest.traefik_service,
[terraform]: │   on modules/traefik_crds/main.tf line 1334, in resource "kubernetes_manifest" "traefik_service":
[terraform]: │ 1334: resource "kubernetes_manifest" "traefik_service" {
[terraform]: │ 
[terraform]: │ resource "/traefikservices.traefik.containo.us" already exists
[terraform]: ╵

Value and/or benefit

It seems to me that it would be quite common to want to deploy Nebari onto cluster with existing ingress definitions.

Anything else?

No response

[dcmcand]

Hi @mcg1969, you are correct, we don't currently support this feature. I have tagged it as an enhancement request. Thanks for submitting this, I think it would be a good addition, but we will need to discuss it with the rest of the team.

[mcg1969]

I did indeed assume it was an enhancement request!

[Adam-D-Lewis]

We discussed on the community call today, and the community seems open to this, but it's not currently prioritized by anyone. Is this something you'd be interested in working on @mcg1969?

[mcg1969]

I won't be able to do a lot of development, I don't think. But one of my team members might be doing some investigation to "post-process" the terraform output at some point. If we get there we can move it upstream. We certainly wouldn't keep it to ourselves

[mcg1969]

In the meanwhile I have good workarounds. The Terraform-installed ingress created by nebari is compatible with other applications that are running on our k3s instances—it's really just the stock ingress with some added ports exposed.

So I can either stand up k3s with the ingress disabled to start; or I can skip the ingress stage, but make sure I've exposed all the same ports.