Cherry-pick #6856 to v1.60.x release branch

[easwars]

RELEASE NOTES:

• server: fix two bugs that could lead to panics at shutdown when using NumStreamWorkers (experimental feature).

[codecov]

Codecov Report

Merging #6864 (abfad18) into v1.60.x (6430548) will increase coverage by 0.08%.
The diff coverage is 100.00%.

Additional details and impacted files

@@             Coverage Diff             @@
##           v1.60.x    #6864      +/-   ##
===========================================
+ Coverage    83.67%   83.76%   +0.08%     
===========================================
  Files          286      286              
  Lines        30792    30797       +5     
===========================================
+ Hits         25765    25796      +31     
+ Misses        3974     3948      -26     
  Partials      1053     1053              

Files	Coverage Δ
server.go	81.45% <100.00%> (+2.45%)	⬆️

... and 23 files with indirect coverage changes

[zasweq]

LGTM.