panic in capa after adding additionalSecurityGroup to running AWSMachine

[egernst]

/kind bug

What steps did you take and what happened:

capa-system panic, see below.

What did you expect to happen:

success.

Additional security group to be applied to the machines

Anything else you would like to add:

Initially I bring up the cluster and then control plane without any problems. I next add a security group assocaited with the cluster-api-created VPC, with rules specific to our CNI (cilium).
I then update the AWSMachineSet associated with our control plane to use this created group.

This is the yaml snippet:

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: AWSMachine
metadata:
  name: test-controlplane-0
spec:
  additionalSecurityGroups:
  - id: "sg-xxx"
  instanceType: t3.large
  # This IAM profile is part of the pre-requisites.
  iamInstanceProfile: "controllers.cluster-api-provider-aws.sigs.k8s.io"
  # Change this value to a valid SSH Key Pair present in your AWS Account.
  sshKeyName: default

When applying this change, the AWSMachine configuration is updated, but almost immediately capa-system panics:

E1003 18:25:38.167701       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/runtime/runtime.go:76
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/runtime/runtime.go:65
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/workspace/controllers/awsmachine_security_groups.go:77
/workspace/controllers/awsmachine_security_groups.go:51
/workspace/controllers/awsmachine_controller.go:308
/workspace/controllers/awsmachine_controller.go:148
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:216
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:192
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:171
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/wait/wait.go:152
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/wait/wait.go:153
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x157432e]

goroutine 342 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash 
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
sigs.k8s.io/cluster-api-provider-aws/controllers.(*AWSMachineReconciler).securityGroupsChanged 
        /workspace/controllers/awsmachine_security_groups.go:77 +0x12e
sigs.k8s.io/cluster-api-provider-aws/controllers.(*AWSMachineReconciler).ensureSecurityGroups 
        /workspace/controllers/awsmachine_security_groups.go:51 +0x112
sigs.k8s.io/cluster-api-provider-aws/controllers.(*AWSMachineReconciler).reconcileNormal 
        /workspace/controllers/awsmachine_controller.go:308 +0x863
sigs.k8s.io/cluster-api-provider-aws/controllers.(*AWSMachineReconciler).Reconcile 
        /workspace/controllers/awsmachine_controller.go:148 +0xb56
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler 
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:216 +0x146
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem 
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:192 +0xb5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker 
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:171 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/wait/wait.go:152 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil 
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190704094733-8f6ac2502e51/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:157 +0x311

Environment:

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[egernst]

/cc @mcastelino @krsna1729 @tgraf

[aaroniscode]

@egernst I was unable to reproduce this. I created a cluster with a single control plane node and no additionalSecurityGroups and then edited the AWSMachine and added a single security group id under additionalSecurityGroups. The security group was added:

I1024 03:17:47.319460       1 awsmachine_controller.go:260] controllers/AWSMachine "msg"="Reconciling AWSMachine" "awsCluster"="capa" "awsMachine"="capa-controlplane" "cluster"="capa" "machine"="capa-controlplane" "namespace"="default"
I1024 03:17:47.534939       1 awsmachine_controller.go:316] controllers/AWSMachine "msg"="Machine instance is running" "awsCluster"="capa" "awsMachine"="capa-controlplane" "cluster"="capa" "machine"="capa-controlplane" "namespace"="default" "instance-id"="i-09e4475f500fcb912"
I1024 03:17:48.057588       1 instances.go:625] controllers/AWSMachine "msg"="Updating security groups" "awsCluster"="capa" "awsMachine"="capa-controlplane" "cluster"="capa" "machine"="capa-controlplane" "namespace"="default" "groups"=["sg-0ab9ab1c804a32520","sg-0482a3e0372671bc8","sg-0d156fc41500db452","sg-06649a11de070a69f"]

This is using CAPA v0.4.3

[aaroniscode]

/assign @aaroniscode

[aaroniscode]

/priority awaiting-more-evidence

[ncdc]

@egernst is this still an issue for you?

[egernst]

Thanks @ncdc. I can't reproduce at this point; closing!