Traefik CRD ingress fails during IngressRoute creation

[Jasper-Ben]

Terraform Version and Provider Version

Terraform v0.12.29
provider.kubernetes-alpha v0.1.0

Kubernetes Version

Client: v1.18.5
Server: v1.18.3

Affected Resource(s)

kubernetes_manifest

Terraform Configuration File (snippet)

Complete config can be found here (encrypted for PGP key D5F43B6432EA77F3)

data "rancher2_cluster" "cluster" {
    provider  = rancher2
    name      = var.cluster_name
}

resource "rancher2_project" "traefik" {
    provider    = rancher2
    name        = "traefik"
    cluster_id  = data.rancher2_cluster.cluster.id
    depends_on  = [
        data.rancher2_cluster.cluster,
    ]
}

resource "rancher2_namespace" "traefik" {
    provider    = rancher2
    name        = "traefik"
    project_id  = rancher2_project.traefik.id
    depends_on  = [
        rancher2_project.traefik,
    ]
}

resource "kubernetes_service_account" "traefik-ingress-controller" {
    metadata {
        name = "traefik-ingress-controller"
        namespace = "kube-system"
    }
}

resource "kubernetes_manifest" "traefik-clusterrole" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"  = "rbac.authorization.k8s.io/v1"
        "kind"        = "ClusterRole"
        "metadata"    = {
            "name"      = "traefik-ingress-controller"
        }
        "rules"       = [
            { 
                "apiGroups" = [
                    "",
                ]
                "resources" = [
                    "services",
                    "endpoints",
                    "secrets",
                ]
                "verbs" = [
                    "get",
                    "list",
                    "watch",
                ]
            },
            {
                "apiGroups" = [
                    "extensions",
                ]
                "resources" = [
                    "ingresses"
                ]
                "verbs" = [
                    "get",
                    "list",
                    "watch",
                ]
            },
            {
                "apiGroups" = [
                    "extensions"
                ]
                "resources" = [
                    "ingresses/status"
                ]
                "verbs"     = [
                    "update"
                ]
            },
            {
                "apiGroups" = [
                    "traefik.containo.us"
                ]
                "resources" = [
                    "middlewares",
                    "ingressroutes",
                    "traefikservices",
                    "ingressroutetcps",
                    "ingressrouteudps",
                    "tlsoptions",
                    "tlsstores",
                ]
                "verbs" = [
                    "get",
                    "list",
                    "watch",
                ]
            }
        ]
    }
    depends_on = [
        kubernetes_service_account.traefik-ingress-controller,
    ]
}

resource "kubernetes_manifest" "traefik-role-binding" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "rbac.authorization.k8s.io/v1"
        "kind"          = "ClusterRoleBinding"
        "metadata"      = {
            "name" = "traefik-ingress-controller"
        }
        "roleRef" = {
            "apiGroup"  = "rbac.authorization.k8s.io"
            "kind"      = "ClusterRole"
            "name"      = "traefik-ingress-controller"
        }
        "subjects" = [
            {
                "kind"      = "ServiceAccount"
                "name"      = "traefik-ingress-controller"
                "namespace" = "kube-system"
            }
        ]
    }
    depends_on = [
        kubernetes_manifest.traefik-clusterrole,
        kubernetes_service_account.traefik-ingress-controller,
    ]
}

resource "kubernetes_manifest" "crd-traefik-ingressroutes" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "apiextensions.k8s.io/v1"
        "kind"          = "CustomResourceDefinition"
        "metadata"      = {
            "name"      = "ingressroutes.traefik.containo.us"
        }
        "spec"          = {
            "group"     = "traefik.containo.us"
            "version"   = "v1alpha1"
            "names"     = {
                "kind"      = "IngressRoute"
                "plural"    = "ingressroutes"
                "singular"  = "ingressroute"
            }
            "scope"     = "Namespaced"
        }
    }
}

resource "kubernetes_manifest" "crd-traefik-middlewares" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "apiextensions.k8s.io/v1"
        "kind"          = "CustomResourceDefinition"
        "metadata"      = {
            "name"      = "middlewares.traefik.containo.us"
        }
        "spec"          = {
            "group"     = "traefik.containo.us"
            "version"   = "v1alpha1"
            "names"     = {
                "kind"      = "Middleware"
                "plural"    = "middlewares"
                "singular"  = "middleware"
            }
            "scope"     = "Namespaced"
        }
    }
}

resource "kubernetes_manifest" "crd-traefik-ingressroutetcps" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "apiextensions.k8s.io/v1"
        "kind"          = "CustomResourceDefinition"
        "metadata"      = {
            "name"      = "ingressroutetcps.traefik.containo.us"
        }
        "spec"          = {
            "group"     = "traefik.containo.us"
            "version"   = "v1alpha1"
            "names"     = {
                "kind"      = "IngressRouteTCP"
                "plural"    = "ingressroutetcps"
                "singular"  = "ingressroutetcp"
            }
            "scope"     = "Namespaced"
        }
    }
}

resource "kubernetes_manifest" "crd-traefik-ingressrouteudps" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "apiextensions.k8s.io/v1"
        "kind"          = "CustomResourceDefinition"
        "metadata"      = {
            "name"      = "ingressrouteudps.traefik.containo.us"
        }
        "spec"          = {
            "group"     = "traefik.containo.us"
            "version"   = "v1alpha1"
            "names"     = {
                "kind"      = "IngressRouteUDP"
                "plural"    = "ingressrouteudps"
                "singular"  = "ingressrouteudp"
            }
            "scope"     = "Namespaced"
        }
    }
}

resource "kubernetes_manifest" "crd-traefik-tlsoptions" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "apiextensions.k8s.io/v1"
        "kind"          = "CustomResourceDefinition"
        "metadata"      = {
            "name"      = "tlsoptions.traefik.containo.us"
        }
        "spec"          = {
            "group"     = "traefik.containo.us"
            "version"   = "v1alpha1"
            "names"     = {
                "kind"      = "TLSOption"
                "plural"    = "tlsoptions"
                "singular"  = "tlsoption"
            }
            "scope"     = "Namespaced"
        }
    }
}

resource "kubernetes_manifest" "crd-traefik-tlsstores" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "apiextensions.k8s.io/v1"
        "kind"          = "CustomResourceDefinition"
        "metadata"      = {
            "name"      = "tlsstores.traefik.containo.us"
        }
        "spec"          = {
            "group"     = "traefik.containo.us"
            "version"   = "v1alpha1"
            "names"     = {
                "kind"      = "TLSStore"
                "plural"    = "tlsstores"
                "singular"  = "tlsstore"
            }
            "scope"     = "Namespaced"
        }
    }
}

resource "kubernetes_manifest" "crd-traefik-traefikservices" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "apiextensions.k8s.io/v1"
        "kind"          = "CustomResourceDefinition"
        "metadata"      = {
            "name"      = "traefikservices.traefik.containo.us"
        }
        "spec"          = {
            "group"     = "traefik.containo.us"
            "version"   = "v1alpha1"
            "names"     = {
                "kind"      = "TraefikService"
                "plural"    = "traefikservices"
                "singular"  = "traefikservice"
            }
            "scope"     = "Namespaced"
        }
    }
}

resource "rancher2_catalog" "traefik" {
    provider    = rancher2
    name        = "traefik"
    url         = "https://containous.github.io/traefik-helm-chart"
    kind        = "helm"
    version     = "helm_v3"
    scope       = "project"
    refresh     = true
    project_id  = rancher2_project.traefik.id
    depends_on  = [
        rancher2_project.traefik,
    ]
}

resource "rancher2_app" "traefik" {
    provider            = rancher2
    catalog_name        = rancher2_catalog.traefik.id
    name                = "traefik"
    project_id          = rancher2_project.traefik.id
    template_name       = "traefik"
    template_version    = "8.9.1"
    target_namespace    = rancher2_namespace.traefik.name
    wait                = false
    depends_on = [
        rancher2_catalog.traefik,
        rancher2_namespace.traefik,
        kubernetes_manifest.traefik-role-binding,
        kubernetes_manifest.crd-traefik-ingressroutes,
        kubernetes_manifest.crd-traefik-middlewares,
        kubernetes_manifest.crd-traefik-ingressroutetcps,
        kubernetes_manifest.crd-traefik-ingressrouteudps,
        kubernetes_manifest.crd-traefik-tlsoptions,
        kubernetes_manifest.crd-traefik-tlsstores,
        kubernetes_manifest.crd-traefik-traefikservices,
    ]
}

resource "kubernetes_manifest" "traefik-dashboard" {
    provider = kubernetes-alpha
    manifest = {
        "apiVersion"    = "traefik.containo.us/v1alpha1"
        "kind"          = "IngressRoute"
        "metadata"      = {
            "name" = "traefik-dashboard"
        }
        "spec"          = {
            "entryPoints" = [
                "web"
            ]
            "routes" = [
                {
                    "match"     = "Host(`traefik.localhost`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))"
                    "kind"      = "Rule"
                    "services"  = [
                        {
                            "name" = "api@internal"
                            "kind" = "TraefikService"
                        }
                    ]
                }
            ]
        }
    }
    depends_on = [
        rancher2_app.traefik
    ]
}

Debug Output

https://gist.github.com/Jasper-Ben/3b9004c5c666eaf39521d5703542b6e0

Expected Behavior

Terraform should create all resources, including traefik-dashboard.

Actual Behavior

Terraform errors during the creation of the traefik-dashboard resource:

Error: rpc error: code = Unknown desc = update dry-run for '/traefik-dashboard' failed: the server could not find the requested resource

Everything else seems to work as expected. Manually adding the dashboard as yaml file via kubectl, as described here succeeds and the dashboard is available at localhost:9000/dashboard/#/.

Steps to Reproduce (more info in the complete config)

1. Setup testcluster using rancher
2. create and include module traefik, as seen above.
3. run terraform apply -target module.traefik.rancher2_namespace.traefik && terraform apply (due to Resource dependency doesn't work #52)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Jasper-Ben]

Followup:
Today I suddently encountered a new issue. Without changing any of the configuration, I suddenly get the following errors during terraform apply:

Error: rpc error: code = Unknown desc = update dry-run for '/ingressroutetcps.traefik.containo.us' failed: failed to create typed patch object: .spec.version: field not declared in schema



Error: rpc error: code = Unknown desc = update dry-run for '/ingressrouteudps.traefik.containo.us' failed: failed to create typed patch object: .spec.version: field not declared in schema



Error: rpc error: code = Unknown desc = update dry-run for '/tlsstores.traefik.containo.us' failed: failed to create typed patch object: .spec.version: field not declared in schema


Error: rpc error: code = Unknown desc = update dry-run for '/tlsoptions.traefik.containo.us' failed: failed to create typed patch object: .spec.version: field not declared in schema



Error: rpc error: code = Unknown desc = update dry-run for '/middlewares.traefik.containo.us' failed: failed to create typed patch object: .spec.version: field not declared in schema



Error: rpc error: code = Unknown desc = update dry-run for '/traefikservices.traefik.containo.us' failed: failed to create typed patch object: .spec.version: field not declared in schema



Error: rpc error: code = Unknown desc = update dry-run for '/ingressroutes.traefik.containo.us' failed: failed to create typed patch object: .spec.version: field not declared in schema

[ruby232]

What I did to solve this situation was:

1. Create a first plam that creates the CRDs and apply them.
2. Then create the other plan that uses the CRDs.

[Jasper-Ben]

Hi @ruby232, thank you for your input! That's what I tried, but even after I applied all the CRDs first and tried to add the dashboard afterwards, terraform would still insist that the resource did not exist. That's before I suddenly was not able to create the CRDs in the first place, as described in #94 (comment).

For my part, I have decided to stick to the nginx ingress for now, until this provider is stable enough. I'll happily answer any maintainer questions and help with testing, if the need arises 😄

[ruby232]

@Jasper-Ben
I tried to decode the settings you put in and I couldn't.
It throws me this error:

gpg -d terraform.tar.gz.gpg
gpg: WARNING: unsafe permissions on homedir '/home/myesr/.gnupg'
gpg: encrypted with RSA key, ID D5F43B6432EA77F3
gpg: decryption failed: No secret key

[Jasper-Ben]

@Jasper-Ben
I tried to decode the settings you put in and I couldn't.
It throws me this error:

gpg -d terraform.tar.gz.gpg
gpg: WARNING: unsafe permissions on homedir '/home/myesr/.gnupg'
gpg: encrypted with RSA key, ID D5F43B6432EA77F3
gpg: decryption failed: No secret key

@ruby232
The complete setup is encrypted for the hasicorp pgp key, so only an employee can decode it. Although the setup should be free from sensitive information, I did not want to publicly post the complete configuration, in case I missed something.

I stripped it down to the parts only relevant for traefik, you can find it here: terraform.tar.gz

How to use:

1. install docker & terraform & kubernetes-alpha provider
2. start a local rancher instance: sudo docker run -d --restart=unless-stopped -p 8080:80 -p 8443:443 rancher/rancher:stable
3. access rancher in browser: https://localhost:8443 and go through the initial setup
4. add a local cluster: Global -> Add Cluster -> From existing nodes (Custom) -> Cluster Name = rd-o -> Next -> Tick all node options (etcd, Control Plane, Worker) -> run the command in the terminal
5. Add an API token: Top right on profile -> API & Keys -> Add Key
6. export the bearer-token: export TF_VAR_rancher_token_key="token-XXXXX
7. download and extract the tar.gz. Go into the terraform-root folder.
8. run terraform init
9. run terraform apply -target local_file.kubeconfig && terraform apply

[grobie]

Duplicate of #48.

The issue is unrelated to the Traefik CRD but the result of not setting the namespace attribute in your kubernetes_manifest resources.

[dak1n1]

Closing since this is a duplicate of #48. (To work around the issue, make sure that each kubernetes_manifest has a namespace field specified.)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!