Skip to content

Race condition in xds package #8474

New issue
New issue
Closed
#8483
Closed
Race condition in xds package#8474
#8483
Assignees
eshitachandwani
Labels
P1Type: Bug
@tritone

Description

@tritone
tritone
opened on Jul 25, 2025

What version of gRPC are you using?

v1.74.2

What version of Go are you using (go version)?

v1.24.0

What operating system (Linux, Windows, …) and version?

Ubuntu

What did you do?

I'm a maintainer in google-cloud-go and saw a failure in a GCS integration test run with the -race flag. The stack trace showed only goroutines in the grpc/xds package so it seems to be internal to that. I wasn't able to reproduce the issue myself; it showed up as a one-time failure in a CI run. Issue details are at googleapis/google-cloud-go#12606.

Logs from the race detector:

==================
WARNING: DATA RACE
Read at 0x00c001670ff0 by goroutine 7570:
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/xdsclient/clientimpl_loadreport.go:35 +0x56
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x33

Previous write at 0x00c001670ff0 by goroutine 7569:
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/xdsclient/clientimpl_loadreport.go:44 +0x271
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x33

Goroutine 7570 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33

Goroutine 7569 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33
==================
==================
WARNING: DATA RACE
Write at 0x00c00656fea0 by goroutine 7570:
  sync/atomic.CompareAndSwapInt32()
      /usr/local/go/src/runtime/race_amd64.s:361 +0xb
  sync/atomic.CompareAndSwapInt32()
      :1 +0x18
  sync.(*Mutex).Lock()
      /usr/local/go/src/sync/mutex.go:46 +0x28
  google.golang.org/grpc/xds/internal/clients/lrsclient.(*LRSClient).getOrCreateLRSStream()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/clients/lrsclient/lrsclient.go:100 +0x84
  google.golang.org/grpc/xds/internal/clients/lrsclient.(*LRSClient).ReportLoad()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/clients/lrsclient/lrsclient.go:88 +0x3c4
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/xdsclient/clientimpl_loadreport.go:47 +0x331
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x33

Previous write at 0x00c00656fea0 by goroutine 7569:
  google.golang.org/grpc/xds/internal/clients/lrsclient.New()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/clients/lrsclient/lrsclient.go:71 +0x7a
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/xdsclient/clientimpl_loadreport.go:36 +0x137
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x33

Goroutine 7570 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33

Goroutine 7569 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/grpc@v1.74.2/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33
==================
    testing.go:1490: race detected during execution of test

Metadata

Metadata

Assignees

Labels

P1Type: Bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions