Upgrading from 2.8.3 to 2.11.0 triggers changes in listener rules.

[kashya2]

I am trying to update aws-load-balancer-controller from v2.8.3 to v2.11.0, upgrade is smooth but as soon as pods with new version is deployed, I see all the listener attributes for ALB changes with below messages, but confirmed there is nothing that actually changes, but there are entries in cloudtrail as well, mentioning there was ModifyRule event happend.

{"level":"info","ts":"<timestamp>","logger":"controllers.ingress","msg":"modified listener rule","stackID":"stackId","resourceID":"443:28","arn":"<ListenerARN>"}

• Can someone confirm on what are these changes is it related to listenerAttributes support added in v2.10.1 or something else?

[shraddhabang]

I don't think updating the listener attributes should result in modifying the listener rules as long as there is no changes done to the rules on ingress. I think the reason could be something else. Did you update rules by any chance?

[kashya2]

Actually there were no rule changes and we just upgraded from version v2.8.3 to v2.11.0 and as soon as new version pod is deployed it triggers rules changes but again there are no actual changes in rules, but somehow in logs it shows "Modified Listener rule"

[kashya2]

We tried this in another environment and its same behaviour there as well, this time we exported the listener rules before the upgrade and did the upgrade (modified listener rules was triggered) and compared with the latest listener rules and it was same.
Also thought of configuring AWS Config to confirm the changes but listener rules are not supported as of now.

[s-marinkovic]

Hi @shraddhabang ,

I have the same issue as @kashya2 described. Updated aws-load-balancer-controller from v2.9.1 to v2.11.0 and in the logs i see the ModifyRule events with following error:
api error ValidationError: A rule can only have '5' condition values

My questions are:

• Why there is ModifyRule event
• Why there is error even I've checked all our rules have 5 or less condition values

Can you please check this, because we won't rollout update to our other environments until this issue is present.

Update:

We had in different configuration update of conditions and it had indeed more than 5 values in total. So there is no issue with 2.11.0 .

[kashya2]

Any updates on this, tested again and got same results.

[usamaahmadkhan]

Faced same issue upgrading from 2.8.z to 2.13.1.

Looks like it doesn't check if the resource needs modification or not and print's the modifying listener rule|target group bindings every-time it reconciles. This gives a wrong impression that a resource will actually be modified on AWS.

We need a refactor of logs to avoid misdirection.

[shraddhabang]

@usamaahmadkhan We do check for any drifts for example before we modify any resource in controller. Would you provide logs to see check if there are any drifts?

[usamaahmadkhan]

@shraddhabang This is the test I ran; Upgraded the deployment:

1. First reconcile prints the following log:

{
  "level": "info",
  "ts": "2025-05-15T13:15:17Z",
  "logger": "controllers.ingress",
  "msg": "modifying listener rule",    <--- Prints this although no change occured. Only reconcile is happening
  "stackID": "kong-ingress",
  "resourceID": "443:1",
  "arn": "arn:aws:elasticloadbalancing:eu-west-1:AAAAAAAAAA:listener-rule/app/k8s-kongingress-<REDACTED>"
}

2. Updated Annotation (Shifted weights through annotation) to simulate change on LB
3. Log printed after the change:

{
  "level": "info",
  "ts": "2025-05-15T13:18:38Z",
  "logger": "controllers.ingress",
  "msg": "modifying listener rule",   <--- Prints this correctly as weights were actually modified
  "stackID": "kong-ingress",
  "resourceID": "443:1",
  "arn": "arn:aws:elasticloadbalancing:eu-west-1:AAAAAAAAAA:listener-rule/app/k8s-kongingress-<REDACTED>"
}

Looks like the check doesn't exist for listener rule and targetgroupbindings. Can u verify?

[shraddhabang]

Hey @usamaahmadkhan , Thank you for your patience.
I have found the problem. Its a small bug in our code but does not really affect anyone. We basically do a drift check before modifying any resource. For rules, we check if the actions are modified or not but we dont check the order of those actions. It seems you are using weighted tgs on rules. this means you are forwarding to two target groups with weight 0 and 100. When the reconcile happens, we build an object. This object for actions may have these two forward tgs in different order than the SDK rules object. The code mistakenly assumes that something is changed and calls modify rules. But in reality the order of actions does not matter, its same config only. Hence you dont see anything changed even after the modification calls. We will fix this bug on our side. But to address your concern, this does not cause any problem to current configuration.