Allow DNS resolution of the runner pod for all k8s setup #886
Conversation
| {{- range include "tf-controller.runner.allowedNamespaces" . | fromJsonArray }} | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: tf-runner | ||
| namespace: {{ . }} | ||
| spec: | ||
| clusterIP: None | ||
| ports: | ||
| - name: grpc | ||
| port: 30000 | ||
| selector: | ||
| app.kubernetes.io/created-by: tf-controller | ||
| app.kubernetes.io/name: tf-runner | ||
| {{- end }} |
There was a problem hiding this comment.
Hi @syalioune
We want to rely only on Pod hostnames without introducing the Service layer.
Would it be possible to do so with your hacks?
There was a problem hiding this comment.
A service is mandatory in my proposal to have a consistent DNS entry generated regardless of the k8s cluster setup.
A service with pod hostnames/subdomains is the best compromise since it requires only one service per namespace instead of a service per runner pod.
controllers/tf_controller_runner.go
Outdated
| @@ -233,6 +234,8 @@ func (r *TerraformReconciler) runnerPodSpec(terraform infrav1.Terraform, tlsSecr | |||
| } | |||
|
|
|||
| return v1.PodSpec{ | |||
| Hostname: terraform.Name, | |||
| Subdomain: "tf-runner", | |||
There was a problem hiding this comment.
Maybe we introduce --use-pod-subdomain-resolution to guard here.
There was a problem hiding this comment.
Those two fields are now guarded by --use-pod-subdomain-resolution
mtls/rotator.go
Outdated
| cert, key, err := cr.createCertPEM(caArtifacts, hostname, time.Now().Add(-1*time.Hour), caArtifacts.validUntil) | ||
| hostnames := []string{ | ||
| fmt.Sprintf("*.%s.pod.%s", namespace, cr.ClusterDomain), | ||
| fmt.Sprintf("*.tf-runner.%s.svc.%s", namespace, cr.ClusterDomain), |
There was a problem hiding this comment.
Maybe we introduce --use-pod-subdomain-resolution to also guard here.
| selector: | ||
| app.kubernetes.io/created-by: tf-controller | ||
| app.kubernetes.io/name: tf-runner |
There was a problem hiding this comment.
| selector: | |
| app.kubernetes.io/created-by: tf-controller | |
| app.kubernetes.io/name: tf-runner | |
| selector: | |
| app.kubernetes.io/created-by: tf-controller | |
| app.kubernetes.io/name: tf-runner |
There was a problem hiding this comment.
Please introduce the new chart value called usePodSubdomainResolution to
- add
--use-pod-subdomain-resolutionflag to the controller (which requires you to create that flag for it too) - guard generation of
Serviceobjects here.
Please note that we need tf-runner Service by default for flux-system namespace too.
There was a problem hiding this comment.
Modifications performed 😄
|
Thank you for your contributions @syalioune Maybe we want to go with the subdomain resolution as you suggested. |
Signed-off-by: Alioune SY <sy_alioune@yahoo.fr>
Signed-off-by: syalioune <sy_alioune@yahoo.fr>
syalioune
force-pushed
the
feature/gke-dns-connectivity
branch
from
e1d4b00 to
574b05e
Compare
| app.kubernetes.io/created-by: tf-controller | ||
| app.kubernetes.io/name: tf-runner | ||
| {{- end }} | ||
| {{- end }} |
There was a problem hiding this comment.
the yaml file needs a new line I believe.
Co-authored-by: souleb <bah.soule@gmail.com>
Draft PR related to #843 (comment) which content I'll replicate here.
First of all, thanks for the great work on this controller.
We've tried deploying recently the
tf-controllerin a standardGKE clustersand hit a wall because of DNS resolution issues. The controller is not able to resolve DNS names like10-40-129-81.podinfo.pod.cluster.local.This issue is best described here #365 (comment) and here #462 (comment). GKE uses Cloud DNS which does not provide IP based DNS name resolution like
CoreDNS.It's still very rough but revolves around :
tf-runnerservice in each allowed namespaceshostname=terraform_crd_name, subdomain=tf-runnerfields to the runner pod so that anA record : terraform_crd_name.tf-runner.namespace.svc.cluster_domainis automatically createdSANinto the runner generated certificatePreliminary tests shows that it works.
Before going further, I'm looking for community/maintainer feedbacks 😄
Cheers