Reconciler error on ipv6 cluster

[dcshiman]

Hi, I am having a similar issue as issues:365

Have tried setting the coreDNS with no luck, i am suspecting it's cos my cluster is on IPV6. Looking at the log, it looks like the tf controller is able to detect the status of runner pod, but can't figure out why its failing to run.

Here is the log form the controller,

{"level":"info","ts":"2022-11-27T20:03:07.811Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2022-11-27T20:03:07.811Z","logger":"setup","msg":"Starting manager","version":"","sha":""}
{"level":"info","ts":"2022-11-27T20:03:07.812Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2022-11-27T20:03:07.812Z","msg":"Starting server","kind":"health probe","addr":"[::]:9440"}
I1127 20:03:07.913143       7 leaderelection.go:248] attempting to acquire leader lease flux-system/1953de50.contrib.fluxcd.io...
I1127 20:03:56.618185       7 leaderelection.go:258] successfully acquired lease flux-system/1953de50.contrib.fluxcd.io
{"level":"info","ts":"2022-11-27T20:03:56.618Z","msg":"Starting EventSource","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","source":"kind source: *v1alpha1.Terraform"}
{"level":"info","ts":"2022-11-27T20:03:56.618Z","msg":"Starting EventSource","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","source":"kind source: *v1beta2.GitRepository"}
{"level":"info","ts":"2022-11-27T20:03:56.618Z","msg":"Starting EventSource","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","source":"kind source: *v1beta2.Bucket"}
{"level":"info","ts":"2022-11-27T20:03:56.618Z","msg":"Starting EventSource","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","source":"kind source: *v1beta2.OCIRepository"}
{"level":"info","ts":"2022-11-27T20:03:56.619Z","msg":"Starting EventSource","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2022-11-27T20:03:56.619Z","msg":"Starting Controller","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform"}
{"level":"debug","ts":"2022-11-27T20:03:56.619Z","logger":"events","msg":"tf-controller-5c887786cb-2mzsr_7400f7f1-0336-4ae6-815c-bebd4a9038e7 became leader","type":"Normal","object":{"kind":"Lease","namespace":"flux-system","name":"1953de50.contrib.fluxcd.io","uid":"5fbfa3df-7a90-4784-ae2a-f33749a82784","apiVersion":"coordination.k8s.io/v1","resourceVersion":"19171701"},"reason":"LeaderElection"}
{"level":"info","ts":"2022-11-27T20:03:56.619Z","logger":"cert-rotation","msg":"starting cert rotator controller"}
{"level":"info","ts":"2022-11-27T20:03:57.320Z","msg":"Starting workers","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","worker count":24}
{"level":"info","ts":"2022-11-27T20:03:58.044Z","msg":">> Started Generation: 1","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","reconciliation-loop-id":"8a647f65-4f07-4cfa-9042-ad06ef45ecf4","start-time":"2022-11-27T20:03:57.320Z"}
{"level":"info","ts":"2022-11-27T20:03:58.044Z","msg":"getting source","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","reconciliation-loop-id":"8a647f65-4f07-4cfa-9042-ad06ef45ecf4","start-time":"2022-11-27T20:03:57.320Z"}
{"level":"info","ts":"2022-11-27T20:03:58.044Z","msg":"before lookup runner: checking ready condition","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","reconciliation-loop-id":"8a647f65-4f07-4cfa-9042-ad06ef45ecf4","start-time":"2022-11-27T20:03:57.320Z","ready":"&Condition{Type:Ready,Status:Unknown,ObservedGeneration:0,LastTransitionTime:2022-11-27 18:09:04 +0000 UTC,Reason:Progressing,Message:Reconciliation in progress,}"}
{"level":"info","ts":"2022-11-27T20:03:58.044Z","msg":"trigger namespace tls secret generation","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","reconciliation-loop-id":"8a647f65-4f07-4cfa-9042-ad06ef45ecf4","start-time":"2022-11-27T20:03:57.320Z"}
{"level":"info","ts":"2022-11-27T20:03:58.245Z","msg":"show runner pod state: ","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","reconciliation-loop-id":"8a647f65-4f07-4cfa-9042-ad06ef45ecf4","start-time":"2022-11-27T20:03:57.320Z","name":"terraform-db-config","state":"must-be-deleted"}
{"level":"info","ts":"2022-11-27T20:04:13.287Z","logger":"KubeAPIWarningLogger","msg":"would violate PodSecurity \"restricted:latest\": seccompProfile (pod or container \"tf-runner\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"}
{"level":"error","ts":"2022-11-27T20:04:58.288Z","msg":"Hit an error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","reconciliation-loop-id":"8a647f65-4f07-4cfa-9042-ad06ef45ecf4","start-time":"2022-11-27T20:03:57.320Z","function":"TerraformReconciler.lookupOrCreateRunner_000","error":"context deadline exceeded","stacktrace":"github.com/weaveworks/tf-controller/controllers.(*TerraformReconciler).LookupOrCreateRunner\n\t/workspace/controllers/tf_controller_runner.go:106\ngithub.com/weaveworks/tf-controller/controllers.(*TerraformReconciler).Reconcile\n\t/workspace/controllers/tf_controller.go:287\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":"2022-11-27T20:04:58.288Z","msg":"unable to lookup or create runner","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","reconciliation-loop-id":"8a647f65-4f07-4cfa-9042-ad06ef45ecf4","start-time":"2022-11-27T20:03:57.320Z","error":"context deadline exceeded","stacktrace":"github.com/weaveworks/tf-controller/controllers.(*TerraformReconciler).Reconcile\n\t/workspace/controllers/tf_controller.go:289\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":"2022-11-27T20:04:58.288Z","msg":"Reconciler error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"04736ed9-647c-4523-9a2b-897a80080726","error":"context deadline exceeded","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":"2022-11-27T20:04:58.294Z","msg":">> Started Generation: 1","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","reconciliation-loop-id":"504ff6b2-06b0-4f94-8ad1-57d3a611fb8d","start-time":"2022-11-27T20:04:58.293Z"}
{"level":"info","ts":"2022-11-27T20:04:58.294Z","msg":"getting source","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","reconciliation-loop-id":"504ff6b2-06b0-4f94-8ad1-57d3a611fb8d","start-time":"2022-11-27T20:04:58.293Z"}
{"level":"info","ts":"2022-11-27T20:04:58.294Z","msg":"before lookup runner: checking ready condition","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","reconciliation-loop-id":"504ff6b2-06b0-4f94-8ad1-57d3a611fb8d","start-time":"2022-11-27T20:04:58.293Z","ready":"&Condition{Type:Ready,Status:Unknown,ObservedGeneration:0,LastTransitionTime:2022-11-27 18:09:04 +0000 UTC,Reason:Progressing,Message:Reconciliation in progress,}"}
{"level":"info","ts":"2022-11-27T20:04:58.294Z","msg":"trigger namespace tls secret generation","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","reconciliation-loop-id":"504ff6b2-06b0-4f94-8ad1-57d3a611fb8d","start-time":"2022-11-27T20:04:58.293Z"}
{"level":"info","ts":"2022-11-27T20:04:58.294Z","logger":"cert-rotation","msg":"TLS already generated for ","namespace":"flux-system"}
{"level":"info","ts":"2022-11-27T20:04:58.294Z","msg":"show runner pod state: ","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","reconciliation-loop-id":"504ff6b2-06b0-4f94-8ad1-57d3a611fb8d","start-time":"2022-11-27T20:04:58.293Z","name":"terraform-db-config","state":"running"}
{"level":"error","ts":"2022-11-27T20:05:43.295Z","msg":"Hit an error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","reconciliation-loop-id":"504ff6b2-06b0-4f94-8ad1-57d3a611fb8d","start-time":"2022-11-27T20:04:58.293Z","function":"TerraformReconciler.lookupOrCreateRunner_000","error":"context deadline exceeded","stacktrace":"github.com/weaveworks/tf-controller/controllers.(*TerraformReconciler).LookupOrCreateRunner\n\t/workspace/controllers/tf_controller_runner.go:106\ngithub.com/weaveworks/tf-controller/controllers.(*TerraformReconciler).Reconcile\n\t/workspace/controllers/tf_controller.go:287\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":"2022-11-27T20:05:43.295Z","msg":"unable to lookup or create runner","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","reconciliation-loop-id":"504ff6b2-06b0-4f94-8ad1-57d3a611fb8d","start-time":"2022-11-27T20:04:58.293Z","error":"context deadline exceeded","stacktrace":"github.com/weaveworks/tf-controller/controllers.(*TerraformReconciler).Reconcile\n\t/workspace/controllers/tf_controller.go:289\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":"2022-11-27T20:05:43.295Z","msg":"Reconciler error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"05903d0e-160f-4eac-8aa1-4aadd4e3975f","error":"context deadline exceeded","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":"2022-11-27T20:05:43.305Z","msg":">> Started Generation: 1","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"f123ce04-8a06-47cf-89dc-8bfa295a0a12","reconciliation-loop-id":"7c8592fe-2195-4979-ad3f-b98fa1141d58","start-time":"2022-11-27T20:05:43.305Z"}
{"level":"info","ts":"2022-11-27T20:05:43.305Z","msg":"getting source","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"f123ce04-8a06-47cf-89dc-8bfa295a0a12","reconciliation-loop-id":"7c8592fe-2195-4979-ad3f-b98fa1141d58","start-time":"2022-11-27T20:05:43.305Z"}
{"level":"info","ts":"2022-11-27T20:05:43.305Z","msg":"before lookup runner: checking ready condition","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"f123ce04-8a06-47cf-89dc-8bfa295a0a12","reconciliation-loop-id":"7c8592fe-2195-4979-ad3f-b98fa1141d58","start-time":"2022-11-27T20:05:43.305Z","ready":"&Condition{Type:Ready,Status:Unknown,ObservedGeneration:0,LastTransitionTime:2022-11-27 18:09:04 +0000 UTC,Reason:Progressing,Message:Reconciliation in progress,}"}
{"level":"info","ts":"2022-11-27T20:05:43.305Z","msg":"trigger namespace tls secret generation","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"f123ce04-8a06-47cf-89dc-8bfa295a0a12","reconciliation-loop-id":"7c8592fe-2195-4979-ad3f-b98fa1141d58","start-time":"2022-11-27T20:05:43.305Z"}
{"level":"info","ts":"2022-11-27T20:05:43.305Z","logger":"cert-rotation","msg":"TLS already generated for ","namespace":"flux-system"}
{"level":"info","ts":"2022-11-27T20:05:43.306Z","msg":"show runner pod state: ","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"terraform-db-config","namespace":"flux-system"},"namespace":"flux-system","name":"terraform-db-config","reconcileID":"f123ce04-8a06-47cf-89dc-8bfa295a0a12","reconciliation-loop-id":"7c8592fe-2195-4979-ad3f-b98fa1141d58","start-time":"2022-11-27T20:05:43.305Z","name":"terraform-db-config","state":"running"}

What am i missing here ? Can direct me towards a solution ?

[chanwit]

Yes, the current behavior uses ipv4 to create mTLS certs. That's the main reason it's not working on your cluster.

We'll definitely need an enhancement for ipv6.

Thank you for reporting this @dcshiman!

[dcshiman]

I have tried making the connection via ip here.
https://github.com/weaveworks/tf-controller/blob/850c5aee1f6fcb6e07155db95743ca25b2204f0b/api/v1alpha1/terraform_types.go#L806-L809

func (in *Terraform) GetRunnerHostname(ip string) string {
	if strings.Count(ip, ":") < 2 {
		prefix := strings.ReplaceAll(ip, ".", "-")
		return fmt.Sprintf("%s.%s.pod.cluster.local", prefix, in.Namespace)
	} else {
		return fmt.Sprintf("[%s]", ip)
	}
}

But it dsn't work cos the certificate is singed using *.<namespace>.pod.cluster.local, I believe the best option will be to set a subdomain on the runner pod

[chanwit]

Thank you for pointing this out @dcshiman
According to the link, we would go setting the hostname and subdomain of the pod, and adding an extra SAN to the cert.

[lasomethingsomething]

Partially solved by recent implementation to the pod resolution technique contributed to the project. Next step would be to deprecate the subport of ipv4 or ipv6 but use subdomain resolution as primary mechanism to talk to the runner. Would require a minimal code change. How to look up or connect. Create headless service for port subdomain resolution. Currently it's being created manually via Helm chart, which is suboptimal.

[yitsushi]

If UsePodSubdomainResolution is enabled with --use-pod-subdomain-resolution or .Values.usePodSubdomainResolution (through Helm values), it uses cluster subdomain resolution and does not use IP addresses: https://github.com/weaveworks/tf-controller/blob/main/controllers/tf_controller_runner.go#L98-L102

This value is false default by default.