Unable to use a self-signed certificate on Google Kubernetes Engine

[lucasvaltl]

Bug description

We are unable to use self-signed certs on GKE due to a limitation on the GKE side: you cannot allow containerd to trust other certificates without restarting containerd.

This issue was created to gather data about this problem.

Steps to reproduce

Try to use self-signed certs on a self-hosted installation running on GKE

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[denismaggior8]

Actually, I managed Gitpod to work on GKE with custom CA certs, using a combination of a DaemonSet, a Secret and a ConfigMap.
What is your preferred way with which to contribute with my manifests? Adding @csweichel in the loop.

[devenes]

Hi @denismaggior8, I tried to install private CA certificates on GKE. It works with Ubuntu nodes but not Container OS nodes. Can you specify the nodes you are using and give an example of your yaml file?

[denismaggior8]

Hi @devenes and thanks for woking this issue. My test has been conducted on Ubuntu (yet using GKE) because I thought that this was the only Linux distribution Gitpod supports. I have never tried to achieve the same on Google COS.

Please refer to the script below (can I open a PR to have this merged?):

---
apiVersion: v1
data:
  systest-ca-pem.crt: AA==
kind: Secret
metadata:
  name: ca-certs
  namespace: gitpod
type: Opaque
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-initializer
  namespace: gitpod
  labels:
    app: default-init
spec:
  selector:
    matchLabels:
      app: default-init
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: node-initializer
        app: default-init
    spec:
      imagePullSecrets:
        - name: artifact-registry
      volumes:
        - name: root-mount
          hostPath:
            path: /
        - name: entrypoint
          configMap:
            name: entrypoint
            defaultMode: 0744
        - name: ca-certs
          secret:
            secretName: ca-certs
            defaultMode: 420
      hostNetwork: true
      hostPID: true
      initContainers:
        - image: my-registry/ubuntu:18.04
          name: node-initializer
          command: ["/scripts/entrypoint.sh"]
          env:
            - name: ROOT_MOUNT_DIR
              value: /mnt
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
            privileged: true
          volumeMounts:
            - name: root-mount
              mountPath: /mnt
            - name: entrypoint
              mountPath: /scripts
            - name: ca-certs
              mountPath: /ca-certs
      containers:
        - image: "my-registry/pause:2.0"
          name: pause
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: entrypoint
  namespace: gitpod
  labels:
    app: default-init
data:
  entrypoint.sh: |
    #!/usr/bin/env bash

    set -euo pipefail

    DEBIAN_FRONTEND=noninteractive
    ROOT_MOUNT_DIR="${ROOT_MOUNT_DIR:-/root}"

    echo "Refresh SSL custom certs"
    ls -l $ROOT_MOUNT_DIR/usr/local/share/ca-certificates
    cp /ca-certs/systest-ca-pem.crt $ROOT_MOUNT_DIR/usr/local/share/ca-certificates
    nsenter --target 1 --mount update-ca-certificates
    nsenter --target 1 --mount systemctl restart containerd

[devenes]

@denismaggior8 Thanks for your great support, I will be working on it.

[denismaggior8]

Thanks to you! I'lol be happy to contribute with a PR!!

[devenes]

🚀 It sounds cool! I'm sure it will be very helpful for everyone. Looking forward your contribution 🐳

[denismaggior8]

@devenes I just submitted the PR #15565, I'm still working on it in order to have it validated and mergiable.