Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mysql provider crashes when creating user #74

Closed
sendmars opened this issue Mar 20, 2022 · 5 comments · Fixed by #75
Closed

Mysql provider crashes when creating user #74

sendmars opened this issue Mar 20, 2022 · 5 comments · Fixed by #75
Labels
bug Something isn't working

Comments

@sendmars
Copy link

sendmars commented Mar 20, 2022 •
edited
Loading

What happened?

I receive a SIGSEGV when I try to create a Mysql user.

E0320 16:39:56.005923       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1039 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x143f340, 0x23deea0})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000bde3e0})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/runtime/runtime.go:48 +0x75
panic({0x143f340, 0x23deea0})
        /opt/hostedtoolcache/go/1.17.7/x64/src/runtime/panic.go:1038 +0x215
github.com/crossplane-contrib/provider-sql/pkg/controller/mysql/user.upToDate(...)
        /home/runner/work/provider-sql/provider-sql/pkg/controller/mysql/user/reconciler.go:376
github.com/crossplane-contrib/provider-sql/pkg/controller/mysql/user.(*external).Observe(0xc000c24140, {0x17e2920, 0xc000c28540}, {0x182bcc8, 0xc0003f3800})
        /home/runner/work/provider-sql/provider-sql/pkg/controller/mysql/user/reconciler.go:235 +0x644
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile(0xc000a394a0, {0x17e2958, 0xc000c21da0}, {{{0x0, 0x0}, {0xc000ac6660, 0x13}}})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/github.com/crossplane/crossplane-runtime@v0.13.0/pkg/reconciler/managed/reconciler.go:577 +0xec7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000a39540, {0x17e28b0, 0xc000884000}, {0x148bfe0, 0xc000bde3e0})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:293 +0x303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000a39540, {0x17e28b0, 0xc000884000})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:248 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1({0x17e28b0, 0xc000884000})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:211 +0x46
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:185 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f8f5fd123c0)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x483abf, {0x17ba280, 0xc000c21d70}, 0x1, 0xc000784120)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0xe0, 0x0)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x17e28b0, 0xc000884000}, 0xc000c1d300, 0xc0000917a0, 0x0, 0x0)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:185 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x17e28b0, 0xc000884000}, 0xc0005fa768, 0xc0005fa778)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:99 +0x2b
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:208 +0x531
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1326144]

goroutine 1039 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000bde3e0})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x143f340, 0x23deea0})
        /opt/hostedtoolcache/go/1.17.7/x64/src/runtime/panic.go:1038 +0x215
github.com/crossplane-contrib/provider-sql/pkg/controller/mysql/user.upToDate(...)
        /home/runner/work/provider-sql/provider-sql/pkg/controller/mysql/user/reconciler.go:376
github.com/crossplane-contrib/provider-sql/pkg/controller/mysql/user.(*external).Observe(0xc000c24140, {0x17e2920, 0xc000c28540}, {0x182bcc8, 0xc0003f3800})
        /home/runner/work/provider-sql/provider-sql/pkg/controller/mysql/user/reconciler.go:235 +0x644
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile(0xc000a394a0, {0x17e2958, 0xc000c21da0}, {{{0x0, 0x0}, {0xc000ac6660, 0x13}}})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/github.com/crossplane/crossplane-runtime@v0.13.0/pkg/reconciler/managed/reconciler.go:577 +0xec7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000a39540, {0x17e28b0, 0xc000884000}, {0x148bfe0, 0xc000bde3e0})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:293 +0x303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000a39540, {0x17e28b0, 0xc000884000})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:248 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1({0x17e28b0, 0xc000884000})
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:211 +0x46
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:185 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f8f5fd123c0)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x483abf, {0x17ba280, 0xc000c21d70}, 0x1, 0xc000784120)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0xe0, 0x0)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x17e28b0, 0xc000884000}, 0xc000c1d300, 0xc0000917a0, 0x0, 0x0)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:185 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x17e28b0, 0xc000884000}, 0xc0005fa768, 0xc0005fa778)
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/k8s.io/apimachinery@v0.20.1/pkg/util/wait/wait.go:99 +0x2b
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
        /home/runner/work/provider-sql/provider-sql/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.0/pkg/internal/controller/controller.go:208 +0x531

The provider-sql-* pod in crossplane-system goes into CrashLoopBackOff cycle, which stops if I delete both user and providerconfig.

My config is quite the same as multiple examples and I haven't found a similar issue anywhere, so I recognize that I may have just missed some small thing.

Note: I have a similar setup with postgres role, works correctly. Not sharing the example here to not pollute the issue, let me know if it helps.

How can we reproduce it?

DB cluster is AWS aurora-mysql.

I created a connection secret similar to the one in README:

$ k create secret generic db-connection --from-literal=endpoint=mycluster.region.rds.amazonaws.com --from-literal=port=3306 --from-literal=username=admin --from-literal=password=x
secret/db-connection created

Digression: I thought that the issue was with the secret created by Crossplane that missed port. With that secret the connection to the cluster would not succeed with the timeout:

Warning   CannotObserveExternalResource   user/test-user  cannot select user: dial tcp 10.x.x.x:0: i/o timeout

Editing that secret to include port 3306 causes the same behavior as described here for the generic secret.
(end digression)

Then I created a providerconfig:

$ cat provider-mysql.yaml

apiVersion: mysql.sql.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: test-provider-config
  labels:
    name: test-provider-config
  annotations:
    crossplane.io/external-name: test-provider-config
spec:
  credentials:
    source: MySQLConnectionSecret
    connectionSecretRef:
      namespace: default
      name: db-connection

And finally, a user:

$ cat user-mysql.yaml
apiVersion: mysql.sql.crossplane.io/v1alpha1
kind: User
metadata:
  name: test-user
spec:
  providerConfigRef:
    name: test-provider-config
  forProvider:
    passwordSecretRef:
      namespace: default
      name: appuser-password          # created in advance
      key: password
  writeConnectionSecretToRef:
    name: cluster-appuser 
    namespace: default

What environment did it happen in?

Crossplane version: 1.6.1
provider-sql: 0.3.0
provider-aws: 0.23.0

Cloud provider: aws
Test done with mysql-aurora cluster, 1 writer: 8.0.mysql_aurora.3.01.0, instance db.r5.large
kubectl client: 1.21, server 1.20.
@sendmars sendmars added the bug Something isn't working label Mar 20, 2022
@Duologic
Copy link
Member

The attempted fix in #71 didn't solve it, working on a unit test now.

@Duologic Duologic mentioned this issue Mar 21, 2022
Merged
2 tasks
@jdotw jdotw closed this as completed in #75 Mar 21, 2022
@sendmars
Copy link
Author

Thanks for addressing this so fast @Duologic, I confirmed that the issue has been resolved.

@alexbowers
Copy link

alexbowers commented Apr 8, 2022 •
edited
Loading

What version is this fixed in? I'm on the latest tagged version (0.4.1) and it is still occurring for me.

@Duologic
Copy link
Member

0.4.1 should have fixed this, are you sure? Can you share some logs?

@alexbowers
Copy link

alexbowers commented Apr 13, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test(mysql/user): fail with panic, resourceOptions are optional crossplane-contrib/provider-sql
3 participants
@alexbowers @Duologic @sendmars