Pubsub: Fix subscriber.stopAsync().awaitTerminated() does not destroy respective threads.

[pmakani]

Fixes #5227, #4211

[codecov]

Codecov Report

Merging #5495 into master will decrease coverage by <.01%.
The diff coverage is 60%.

@@             Coverage Diff              @@
##             master    #5495      +/-   ##
============================================
- Coverage     50.82%   50.82%   -0.01%     
+ Complexity    24189    24175      -14     
============================================
  Files          2271     2271              
  Lines        229912   229918       +6     
  Branches      25002    25007       +5     
============================================
+ Hits         116845   116848       +3     
- Misses       104435   104438       +3     
  Partials       8632     8632

Impacted Files	Coverage Δ	Complexity Δ
...in/java/com/google/cloud/pubsub/v1/Subscriber.java	77.77% <100%> (+0.24%)	17 <0> (ø)	⬇️
.../com/google/cloud/pubsub/v1/MessageDispatcher.java	84.89% <50%> (-0.97%)	23 <0> (ø)
...cloud/pubsub/v1/StreamingSubscriberConnection.java	64% <50%> (-0.76%)	9 <2> (ø)
...oogle/cloud/pubsub/v1/stub/GrpcSubscriberStub.java	94.48% <0%> (+0.25%)	23% <0%> (+1%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a560bb5...579bc89. Read the comment docs.

[sduskis]

@pmakani, thanks for the PR. I spoke at length with @ajaaym about this. I think we will have to defer to @kamalaboulhosn and team about how to proceed with this. There are a number of things to consider as part of the shutdown process.

[sduskis]

@pmakani, the Pub/Sub team will work on this.

[pmakani]

@hannahrogers-google PTAL.

[hannahrogers-google]

@pmakani, thanks for creating this PR. I have discussed this issue with other members of the Pub/Sub team to determine the best solution.

We would like to close the subStub, as you do in this PR. However, as part of the shutdown process, we would like make sure we process all outstanding acks/nacks. We would only like to close the stub once all related rpcs have completed.