Fix issues with self-signed certificates

[Pothulapati]

Description

This PR includes temporary fixes to get Certificate option working with self-signed certificates.

Currently, Whenever the Certificate config option is set to a specific secret, we seem to assume by default
that they are publicly signed. This currently has the following issues when they are self-signed :

• Due to the usage of Certificate in internal facing services like registry-facade, Internal clients fail to communicate as the self-signed CA is not in their trust store. (Fix: Update registry-facade to use builtin-registry-facade-cert, generated by cert-manager)
• Due to the fact that internal registry communication also flows through the proxy, This communication fails as the proxy cert is self-signed as its not available in the trust store. (Fix: Update the registry-facade daemonset to also incldue crt of Certificate in its trust store).

This PR also fixes an issue with image-builder-mk3 when the customCACert option is set, by removing InternalVolumeCAMount as it makes the etc/ssl/certs empty and readOnly (which prevents from ounting the customCACert cert into etc/ssl/certs

Related Issue(s)

Part of #9074

How to test

Generate a CA, and self-signed SSL by following https://gist.github.com/Pothulapati/a7c68e073cdbf8573f12b64320711ce5 (or any other online tutorial for that matter)

# Create the relevant secrets before installing Gitpod
# myCa.pem is what you want to propogate across components to access things external to Gitpod
kubectl create secret generic ca-certificate --from-file=ca.crt=./certs/myCa.pem
# these are the ones created for the domain
kubectl create secret tls https-cert --cert=./certs/ssl.crt --key=./certs/ssl.key

# Set the relevant config options in config file
yq e -i ".certificate.name = \"https-cert\"" "${CONFIG_FILE}"
yq e -i ".certificate.kind = \"secret\"" "${CONFIG_FILE}"
yq e -i ".customCACert.kind = \"secret\"" "${CONFIG_FILE}"
yq e -i ".customCACert.name = \"ca-certificate\"" "${CONFIG_FILE}"

# Install GItpod
gitpod-installer \
        render \
        --config="${CONFIG_FILE}" > gitpod.yaml

# See https://github.com/gitpod-io/gitpod/tree/main/install/installer#error-validating-statefulsetstatus
yq eval-all --inplace \
        'del(select(.kind == "StatefulSet" and .metadata.name == "openvsx-proxy").status)' \
        gitpod.yaml

kubectl apply -f gitpod.yaml

Release Notes

Fix multiple issues with `self-signed` certs

Documentation

[Pothulapati]

/werft run

👍 started the job as gitpod-build-tar-etc-empty-mounts.7
(with .werft/ from main)

[mrsimonemms]

Works really well. It also works when using the ca-issuer-ca, which is the internal CA that's used to create all the internal certs.

[iQQBot]

This PR broken preview environment, because this self-signed cert is doesn't trust by k3s(contained), it cause the image pull failed

CC @Pothulapati

[Pothulapati]

@iQQBot This is weird, as we pass the same CA cert is loaded into kubelets by the same registry-facade daemonset! Wondering why that cert is not being used! 🤔

[iQQBot]

I haven't looked into why it wasn't loaded correctly into containerd, but it's easy to verify, just open a branch and build with with-vm annotation to verify

[iQQBot]

gitpod/install/installer/pkg/components/registry-facade/daemonset.go

Lines 189 to 194 in f5778fa

	corev1.EnvVar{
	// Install gitpod ca.crt in containerd to allow pulls from the host
	// https://github.com/containerd/containerd/blob/main/docs/hosts.md
	Name: "SETUP_SCRIPT",
	Value: fmt.Sprintf(`TARGETS="docker containerd";for TARGET in $TARGETS;do mkdir -p /mnt/dst/etc/$TARGET/certs.d/reg.%s:%v && echo "$GITPOD_CA_CERT" > /mnt/dst/etc/$TARGET/certs.d/reg.%s:%v/ca.crt && echo "OK";done`, ctx.Config.Domain, ServicePort, ctx.Config.Domain, ServicePort),
	},

maybe it is because here

In the preview environment, we will randomly specify the port of registry-facad, this replacement is processed in the post-process installer, most likely not processed to this (guess, not yet verified)

But by the way, in the harvester test environment, there is no need for random ports anymore