DBInstance update request without change (causes instance to restart in loop) #1348
Comments
|
@chlunde do you think it's time to upstream your version sparebank1utvikling@27b71a1 to get more details for diff ? |
|
Oh Oh... just got a status I didn't see before: |
|
Also worth noting... |
|
@haarchri yeah, I think that's a good idea, but I don't think I have much time this week. |
|
Hello there, we are facing a similar issue where the crossplane aws-provider v0.28.1 reconciles an RDS database over and over again. The requested change is to modify the Before v0.28.1 our database instances could not be synchronized at all because of wrong parameter combinations (e.g. backupRetention on DBInstance which is not allowed when using a DBCluster + DBInstance). After the update the synchronization is working again, but with the above problem. Below you can find some additional information like the CloudTrail event to modify the DBInstance and the logs from the aws-provider. Cloudtrail Event (ModifyDbInstance)aws-provider logsSee aws-provider.txt Kubernetes resources
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: DBInstance
metadata:
annotations:
crossplane.io/external-name: example-service-rds-instance-0
creationTimestamp: "2022-06-20T13:38:35Z"
finalizers:
- finalizer.managedresource.crossplane.io
generateName: example-service-rds-instance-0-dsfwv-
generation: 2
labels:
example.com/managed-by: example-service-rds-instance-0
crossplane.io/claim-name: example-service-rds-instance-0
crossplane.io/claim-namespace: dev
crossplane.io/composite: example-service-rds-instance-0-dsfwv
name: example-service-rds-instance-0-dsfwv
ownerReferences:
- apiVersion: example.com/v1
controller: true
kind: XRdsInstance
name: example-service-rds-instance-0-dsfwv
uid: 8d576847-52b9-4b0c-bc8d-dc84cd3d3875
resourceVersion: "194662643"
uid: 4228e5be-5fb2-4461-be40-de6a8e2757bc
spec:
deletionPolicy: Orphan
forProvider:
autoMinorVersionUpgrade: true
availabilityZone: eu-central-1c
dbClusterIdentifier: example-service-rds-cluster
dbInstanceClass: db.t3.small
dbName: example-service-dev
dbParameterGroupName: cluster-aurora-mysql-5-7
dbSubnetGroupName: test.example.com
enablePerformanceInsights: false
engine: aurora-mysql
engineVersion: 5.7.mysql_aurora.2.10.2
kmsKeyID: arn:aws:kms:eu-central-1:111111111111:key/abc
licenseModel: general-public-license
masterUsername: master
multiAZ: false
port: 3306
preferredMaintenanceWindow: tue:03:50-tue:04:20
promotionTier: 1
publiclyAccessible: false
region: eu-central-1
storageEncrypted: true
tags:
- key: claim-namespace
value: dev
- key: claim-name
value: example-service-rds-instance-0
- key: managed-resource-name
value: example-service-rds-instance-0-dsfwv
vpcSecurityGroupIDs:
- sg-1
providerConfigRef:
name: aws-provider-config
status:
atProvider:
caCertificateIdentifier: rds-ca-2019
customerOwnedIPEnabled: false
dbInstanceARN: arn:aws:rds:eu-central-1:111111111:db:example-service-rds-instance-0
dbInstanceIdentifier: example-service-rds-instance-0
dbInstancePort: 0
dbInstanceStatus: modifying
dbParameterGroups:
- dbParameterGroupName: cluster-aurora-mysql-5-7
parameterApplyStatus: applying
dbSubnetGroup:
dbSubnetGroupDescription: Database subnet group for test.example.com
dbSubnetGroupName: test.example.com
subnetGroupStatus: Complete
subnets:
- subnetAvailabilityZone:
name: eu-central-1c
subnetIdentifier: subnet-1
subnetOutpost: {}
subnetStatus: Active
- subnetAvailabilityZone:
name: eu-central-1b
subnetIdentifier: subnet-2
subnetOutpost: {}
subnetStatus: Active
- subnetAvailabilityZone:
name: eu-central-1a
subnetIdentifier: subnet-3
subnetOutpost: {}
subnetStatus: Active
vpcID: vpc-1
dbiResourceID: db-HXNHMYU4AQ4MT7N2L2KSS43L4Q
endpoint:
address: example-service-rds-instance-0.cj71ow7e0nmu.eu-central-1.rds.amazonaws.com
hostedZoneID: Z1RLNUO7B9Q6NB
port: 3306
iamDatabaseAuthenticationEnabled: false
instanceCreateTime: "2022-05-04T14:20:43Z"
optionGroupMemberships:
- optionGroupName: default:aurora-mysql-5-7
status: in-sync
pendingModifiedValues:
caCertificateIdentifier: rds-ca-2019
performanceInsightsEnabled: false
tagList:
- key: managed-resource-name
value: example-service-rds-instance-0-28tdw
- key: claim-namespace
value: dev
- key: claim-name
value: example-service-rds-instance-0
vpcSecurityGroups:
- status: active
vpcSecurityGroupID: sg-1
conditions:
- lastTransitionTime: "2022-06-23T07:13:18Z"
reason: ReconcileSuccess
status: "True"
type: Synced
- lastTransitionTime: "2022-06-20T13:38:35Z"
reason: Available
status: "True"
type: Ready
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: DBCluster
metadata:
annotations:
crossplane.io/external-name: example-service-rds-cluster
creationTimestamp: "2022-06-20T14:05:51Z"
finalizers:
- finalizer.managedresource.crossplane.io
generateName: example-service-rds-cluster-rtvqp-
generation: 2
labels:
example.com/managed-by: example-service-rds-cluster
crossplane.io/claim-name: example-service-rds-cluster
crossplane.io/claim-namespace: dev
crossplane.io/composite: example-service-rds-cluster-rtvqp
name: example-service-rds-cluster-rtvqp
ownerReferences:
- apiVersion: example.com/v1
controller: true
kind: XRdsCluster
name: example-service-rds-cluster-rtvqp
uid: 44843b1a-e386-49b7-9e24-a7daebf195f9
resourceVersion: "194674646"
uid: 1132a5e6-7b6b-4199-a2c2-4b63bf86075c
spec:
deletionPolicy: Orphan
forProvider:
applyImmediately: true
backupRetentionPeriod: 35
copyTagsToSnapshot: true
databaseName: example-service
dbClusterParameterGroupName: cluster-aurora-mysql-5-7
dbSubnetGroupName: test.cluster.example.com
deletionProtection: true
engine: aurora-mysql
engineMode: provisioned
finalDBSnapshotIdentifier: example-service-rds-final-snapshot
masterUserPasswordSecretRef:
key: password
name: example-service-rds-secret
namespace: crossplane-system
masterUsername: master
region: eu-central-1
skipFinalSnapshot: true
storageEncrypted: true
tags:
- key: claim-namespace
value: acp-dev
- key: claim-name
value: example-service-rds-cluster
- key: managed-resource-name
value: example-service-rds-cluster-rtvqp
- key: secret-name
value: example-service-rds-secret
- key: secret-namespace
value: crossplane-system
vpcSecurityGroupIDs:
- sg-1
providerConfigRef:
name: aws-provider-config
writeConnectionSecretToRef:
name: example-service-rds-cluster-rtvqp-rds-cluster-managed-resource
namespace: crossplane-system
status:
atProvider:
activityStreamStatus: stopped
allocatedStorage: 1
clusterCreateTime: "2022-05-04T14:13:29Z"
crossAccountClone: false
dbClusterARN: arn:aws:rds:eu-central-1:11111111:cluster:example-service-rds-cluster
dbClusterIdentifier: example-service-rds-cluster
dbClusterMembers:
- dbClusterParameterGroupStatus: in-sync
dbInstanceIdentifier: example-service-rds-instance-0
isClusterWriter: true
promotionTier: 1
dbClusterParameterGroup: cluster-aurora-mysql-5-7
dbClusterResourceID: cluster-FQTBKWPCU3S4R3ULR4IQZ2AZNI
dbSubnetGroup: test.cluster.example.com
earliestRestorableTime: "2022-06-22T00:58:37Z"
endpoint: example-service-rds-cluster.cluster-cj71ow7e0nmu.eu-central-1.rds.amazonaws.com
hostedZoneID: 1111111
httpEndpointEnabled: false
iamDatabaseAuthenticationEnabled: false
latestRestorableTime: "2022-06-23T08:23:11Z"
multiAZ: false
readerEndpoint: example-service-rds-cluster.cluster-ro-cj71ow7e0nmu.eu-central-1.rds.amazonaws.com
status: available
tagList:
- key: managed-resource-name
value: example-service-rds-cluster-92lgq
- key: claim-namespace
value: acp-dev
- key: secret-namespace
value: crossplane-system
- key: claim-name
value: example-service-rds-cluster
- key: secret-name
value: example-service-rds-secret
vpcSecurityGroups:
- status: active
vpcSecurityGroupID: sg-1
conditions:
- lastTransitionTime: "2022-06-20T14:05:51Z"
reason: ReconcileSuccess
status: "True"
type: Synced
- lastTransitionTime: "2022-06-20T14:05:52Z"
reason: Available
status: "True"
type: Ready
Any help would be greatly appreciated :-) |
|
So I've modified the AWS provider and ran it locally to output the actual differences between the current and desired state for my RDS DBInstances: My DBInstances are updated because of the following diff: I am still looking into how this is happening, as my cloudtrail events clearly state that both the |
|
@fbuchmeier-abi Thanks for that! |
|
So in my instance, the issue happens when using a aurora-mysql cluster (DBCluster & DBInstances) and passing unsupported parameters (e.g. copyTagsToSnapshot) to the DBInstances ( I had the following DBInstance Composition in place: This created DBInstance resources with I then updated my composition, removed both fields, to this: Now any newly provisioned DBCluster & DBInstances synchronize correctly. However, all DBInstances created with the old composition revision ( |
|
ping, anything new? |
|
okay we see today the same in our instances - reboot loop @chlunde did you running other version for dbinstance in your setup ? and we intoduced 2 new fields: |
|
We are having the exact same issue also! Any news on this? |
|
We had an issue with the KMS Key ID / ARN - Check for latest pr reference |
|
Yes |
|
We have just migrated to crossplane-provider-aws 0.32.0 which contains #1481 but we still see the same problem for docdb resources. Considering the above mentioned PR fixed the issue for |
|
@pandrez can you open an new issue for docd? |
|
@nicolasbelanger is your issue fixed ? |
@haarchri Sorry for the late reply. Thanks to the logs, we can see this now: &v1alpha1.DBInstanceParameters{
... // 4 ignored and 22 identical fields
LicenseModel: nil,
MasterUsername: nil,
- MaxAllocatedStorage: nil,
+ MaxAllocatedStorage: &200,
MonitoringInterval: nil,
MonitoringRoleARN: nil,
... // 3 ignored and 15 identical fields
}That's an issue in itself, but we're concerned by the Anything wrong in this? apiVersion: rds.aws.crossplane.io/v1alpha1
kind: DBParameterGroup
metadata:
annotations:
argocd.argoproj.io/sync-wave: '-1'
labels:
app.kubernetes.io/instance: _redacted_
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: _redacted_-_redacted_-_redacted_-test
app.kubernetes.io/part-of: _redacted_-rds-test-el
argocd.argoproj.io/instance: _redacted_-rds-test-el
release: _redacted_-rds-test-el
name: _redacted_-_redacted_-_redacted_-test
spec:
deletionPolicy: Delete
forProvider:
dbParameterGroupFamilySelector:
engine: mysql
engineVersion: 8.0.28
description: >-
DBParameterGroup for _redacted_-_redacted_-_redacted_-test RDS Instance via Crossplane
region: us-east-1
tags:
- key: Name
value: _redacted_-_redacted_-_redacted_-test
- key: env
value: development
- key: appName
value: _redacted_
- key: businessUnit
value: _redacted_
- key: contact
value: _redacted_
- key: product
value: _redacted_
- key: role
value: database
- key: service
value: _redacted_
providerConfigRef:
name: aws-provider |
|
I think this issue should be reopened as the core issue still exists... |
|
@nicolasbelanger please open an fresh issue |
What happened?
DBInstancealong withDBParameterGroup,DBSubnetGroupandSecurityGroupDBParameterGroup,DBSubnetGroupandSecurityGroup(External resource is up to date)DBInstance(Successfully requested update of external resource) forcing an apply and restart of the instance (it goes on forever untilprovider-aws-controlleris shut down)How can we reproduce it?
Subsequent request as seen in CloudTrail:
What environment did it happen in?
Crossplane version: 1.5.1
Provider-aws-controller: v0.27.0 (also reproduced with v0.26.1 and master at v0.28.0-rc.0.37.g9e1ce166)
kubectl version): 1.20.11Extra Notes:
Looks closely related to #960... Seems like not all the possible bools have been fixed...
The text was updated successfully, but these errors were encountered: