Don't limit CODER_NAMESPACE to a single namespace

[hh]

From https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ :

• Namespaces are a way to divide cluster resources between multiple users
• via Resource Quotas : https://kubernetes.io/docs/concepts/policy/resource-quotas/
• and RBAC : https://kubernetes.io/docs/reference/access-authn-authz/rbac/

There are benefits to deploying per-user namespaces:

• Ability to give the user control over their own namespace via RBAC (deploying other objects / API Isolation)
• Ability to persist expensive objects like cert-manager certs / let encrypt (some objects take a lot of time)
• Ability to isolate traffic between multiple users / namespaces

We create a namespace per user, and do not destroy it when a workspace is torn down. This allows expensive objects (like cert-manager/letsencrypt certs/dns) to persist and be reused for multiple workspaces (from the same user) to access them.

Some resources we use per user/namespace:

• Issuer (Cert-Manager w/ DNS01 for wildcard)
• Certificate (this can take 40 seconds to provision from Lets Encrypt)
• tls-secret (generated by TLS Cert from Certificate)
• wildcard ingress (each user get's there own namespace AND *.username.coder.website [accessible without coder])
• RoleBinding w/ admin over their own namespace (we allow them to create whatever other resources they want within their namespace) : RBAC
• We use Resource Quotas to ensure one user doesn't take over all the resources on a node

[bpmct]

I see. How are you currently provisioning per-user resources? This is actually a feature we've considered doing in Coder.

I think there may be some limitations with Helm around provisioning multi-namespace resources. @hh - Would it be possible to also provision a coder-logstream-kube per-user/namespace as well? That may be a nice workaround.

[hh]

Ideally everything runs within the namespace we create for them, what token would the coder-logstream-kube pod use?

https://github.com/cloudnative-coop/space-templates/blob/canon/equipod/namedspaced.tf#L6-L15

resource "null_resource" "namespace" {
  # install kubectl
  provisioner "local-exec" {
    command = "~/kubectl version --client || (curl -L https://dl.k8s.io/release/v1.27.3/bin/linux/amd64/kubectl -o ~/kubectl && chmod +x ~/kubectl)"
  }
  provisioner "local-exec" {
    command = "~/kubectl create ns ${local.spacename}"
  }
  provisioner "local-exec" {
    command = "~/kubectl -n ${local.spacename} apply -f ${path.module}/manifests/admin-sa.yaml"
  }
}

https://raw.githubusercontent.com/cloudnative-coop/space-templates/canon/equipod/manifests/admin-sa.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
automountServiceAccountToken: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: admin
rules:
  - apiGroups:
      - ""
    resources:
      - "*"
    verbs:
      - "*"
  - apiGroups:
      - "*"
    resources:
      - "*"
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: admin
subjects:
  - kind: ServiceAccount
    name: admin

[bpmct]

I see. Currently, coder-logstream-kube runs within the namespace and doesn't require a token! It uses the token from each workspace's pod spec (which is scoped to only send agent logs/stats for the specific workspace).

helm install coder-logstream-kube coder-logstream-kube/coder-logstream-kube \
    --namespace coder \
    --set url=<your-coder-url>

[sreya]

@hh are you still looking to do a single logstream-kube deployment for multiple workspaces? Wondering if this is worth supporting or not. You'll have to be ok with a cluster role/binding if you don't want to limit to a single namespace.

[matifali]

from #28

v0.0.9-rc.0 - use-case: one logstream-kube deployment watching pods in multiple namespaces.

if the namespace value is unset, logstream-kube should default to watching pods in all namespaces (assuming the proper permissions). this is not currently the case. my values, installed in the coder namespace:

USER-SUPPLIED VALUES:
url: https://eric-aks.demo.coder.com

my workspace is running in the coder-workspaces namespace, with the following role and rolebinding deployed:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: coder-logstream-kube-role
rules:
- apiGroups: [""]
  resources: ["pods", "events"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["apps"]
  resources: ["replicasets", "events"]
  verbs: ["get", "watch", "list"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: coder-logstream-kube-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: coder-logstream-kube-role
subjects:
- kind: ServiceAccount
  name: coder-logstream-kube
  namespace: coder

[dmildh-absci]

We also use a namespace per user and are a paying customer so would be nice to be have single deployment to collect logs from all the namespaces

[dmildh-absci]

For now I ended up creating another null resource the includes the following kubectl manifest (since we already download kubectl for username namespace creation). You could also just download helm each time as well too.

resource "local_file" "coder_log_stream" {
  filename = "/tmp/coder-log-stream.yml"
  content  = templatefile("${path.module}/templates/coder-log-stream.yml.tpl", {
    CODER_URL = data.kubernetes_config_map.coder_url.data.url,
    CODER_NAMESPACE = data.coder_workspace_owner.me.name
  })
}

resource "null_resource" "coder_log_stream" {
  depends_on = [
    null_resource.create_ns,
    local_file.coder_log_stream
  ]
  triggers = {
    always_run = timestamp()
  }
  provisioner "local-exec" {
    command = "/tmp/kubectl apply -n ${data.coder_workspace_owner.me.name} -f /tmp/coder-log-stream.yml"
  }
}

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: coder-logstream-kube-role
rules:
- apiGroups: [""]
  resources: ["pods", "events"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["apps"]
  resources: ["replicasets", "events"]
  verbs: ["get", "watch", "list"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coder-logstream-kube
  labels: 
    app.kubernetes.io/managed-by: coder-template

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: coder-logstream-kube-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: coder-logstream-kube-role
subjects:
- kind: ServiceAccount
  name: coder-logstream-kube

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coder-logstream-kube
spec:
  # This must remain at 1 otherwise duplicate logs can occur!
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: coder-logstream-kube
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: coder-logstream-kube
    spec:
      serviceAccountName: coder-logstream-kube
      restartPolicy: Always
      imagePullSecrets:
       - name: "dockerhubregcreds"
      nodeSelector:
        oke.oraclecloud.com/pool.name: cluster-apps
      containers:
        - name: coder-logstream-kube
          image: ghcr.io/coder/coder-logstream-kube:v0.0.11
          imagePullPolicy: IfNotPresent
          command:
            - /coder-logstream-kube
          resources: {}
          env:
            - name: CODER_URL
              value: ${CODER_URL}
            - name: CODER_NAMESPACE
              value: ${CODER_NAMESPACE}