ArgoCD out of sync because of unpredictable list order in networkacl and routetable objects

[adriananeci]

Describe the bug
There are few fields in networkacl and routetable objects spec that are of type list (e.g tags, entries, etc).
I've noticed that the tags and entries values are getting changed by the ack controller once created which means the order of the items from these lists are not set in a predictable order.

In the context of ArgoCD, because of the unpredictable order of the elements from a list, an application that creates/reconciles such objects is constantly out of sync

Steps to reproduce

Create an ArgoCD Application that is deploying a networkacl object which contains multiple tags and multiple entries in its spec.

Expected outcome
ArgoCD Application that is deploying a networkacl object should be in a synced state

Environment

• Kubernetes version: 1.27
• Using EKS (yes/no), if so version? : 1.27
• AWS service targeted (S3, RDS, etc.): EC2

cc @a-hilaly

[a-hilaly]

Thank you for reporting this @adriananeci - this is indeed a bug, and the controller shouldn't mess with the order of tags or any other field. More reasons to stop sorting/using samberlo

[gecube]

In the context of ArgoCD, because of the unpredictable order of the elements from a list, an application that creates/reconciles such objects is constantly out of sync

ArgoCD bugs. It should not react on diff when list elements were in changed order. I could recommed only to remove these particular fields from diff monitoring. The documentation is here: https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/

@a-hilaly it's what we already discussed in context of FluxCD usages. like here #1898 or #1880

[gecube]

ArgoCD Application that is deploying a networkacl object should be in a synced state

it's another thing. It is related to kstatus support I believe #1899 The very same issue is with Crossplane project.

[gecube]

Also proof that ArgoCD still has an issue with the ordering of the lists:

argoproj/argo-cd#12033

[adriananeci]

I could recommed only to remove these particular fields from diff monitoring. The documentation is here: https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/

That won't work because then any changes added to these lists won't be synced by argocd

[gecube]

@adriananeci thanks, your remark makes sense.

[adriananeci]

Wasn't able to reproduce it in the last EC2 ACK version so I guess we can close it. Will open a new issue if there are similar sync issues.
/close

[ack-prow]

@adriananeci: Closing this issue.

In response to this:

Wasn't able to reproduce it in the last EC2 ACK version so I guess we can close it. Will open a new issue if there are similar sync issues.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[adriananeci]

/reopen
Jantzen was able to reproduce it and will post more details

[ack-prow]

@adriananeci: Reopened this issue.

In response to this:

/reopen
Jantzen was able to reproduce it and will post more details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jantzenallphin]

I've already met with @a-hilaly on this. RouteTable objects seems to be fixed but NACLs are still showing out of sync due to list ordering. I showed this over our call together.

[jantzenallphin]

Fixed by aws-controllers-k8s/ec2-controller#208 in v1.2.17