argocd-server panics with concurrent map read and write #19913
Labels
bug
Something isn't working
component:server
component:settings
RBAC issues/enhancements
version:2.12
Latest confirmed affected version is 2.12
Comments
|
This may be fixed in later versions of k8s.io/apimachinery. |
|
Try upgrading to 2.13 or 2.12 and see if it fixes the issue. |
|
I am on 2.12 now but 2.10 is still a supported version. |
|
2.10.18 uses |
|
I've checked and 2.10 is not supported anymore per https://github.com/argoproj/argo-cd/security/policy#supported-versions. |
|
Though latest 2.11 has the same library version https://github.com/argoproj/argo-cd/blob/v2.11.12/go.mod#L314. |
|
Does 2.12 have the same issue? |
andrii-korotkov-verkada
added
the
more-information-needed
|
I will check through our logs today to see if it happened.
“We currently support the last 3 minor versions of Argo CD with security and bug fixes.” Isn’t 2.13 the latest version which would mean 2.10 is within 3 minor versions? |
|
It's 2.11, 2.12 and 2.13 |
|
Still happening in |
andrii-korotkov-verkada
added
version:2.12
|
What I think is happening is the informers to get or list secrets / config maps return a read only pointer and either we or the informer are editing the underlying struct. i will try to create a unit test reproducing and then submit a PR. |
|
I am seeing this happen in two places.
argocd-server-11-2.txt |
14 tasks
|
This seems to be also be the case with the |
All the thread dumps have the running thread accessing secrets. |
ishitasequeira
pushed a commit
that referenced
this issue
Nov 21, 2024
…rmers #19913 (#20805) * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * e2e Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> --------- Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com>
revitalbarletz
pushed a commit
to revitalbarletz/argo-cd
that referenced
this issue
Nov 21, 2024
…rmers argoproj#19913 (argoproj#20805) * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * e2e Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> --------- Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com>
adriananeci
pushed a commit
to adriananeci/argo-cd
that referenced
this issue
Dec 4, 2024
…rmers argoproj#19913 (argoproj#20805) * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * e2e Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> --------- Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> Signed-off-by: Adrian Aneci <aneci@adobe.com>
agaudreault
pushed a commit
to agaudreault/argo-cd
that referenced
this issue
Dec 17, 2024
…rmers argoproj#19913 (argoproj#20805) * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * fix(server): make a copy of secret objects when listing from the informers Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> * e2e Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> --------- Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com> Signed-off-by: Alexandre Gaudreault <alexandre_gaudreault@intuit.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist:
argocd version.Describe the bug
Server is panicking from reading and writing to a map concurrently. I believe it is happening from
github.com/argoproj/argo-cd/v2/util/settings.(*SettingsManager).GetSettingsTo Reproduce
Expected behavior
Screenshots
Version
{ "Version": "v2.10.6+d504d2b", "BuildDate": "2024-04-05T00:27:47Z", "GitCommit": "d504d2b1d92f0cf831a124a5fd1a96ee29fa7679", "GitTreeState": "clean", "GoVersion": "go1.21.3", "Compiler": "gc", "Platform": "linux/amd64", }Logs
argocd-server.log
The text was updated successfully, but these errors were encountered: