[🐛 BUG]: grpc client is closed to early after graceful shutdown

[kkosowski-p]

No duplicates 🥲.

• I have searched for a similar issue in our bug tracker and didn't find any solutions.

What happened?

RR closes to early grpc connection so some running process are not able to complete correctly.

Version (rr --version)

2024.1.5

How to reproduce the issue?

(with temporal)

1. start activity which takes some time (for instance - makes a HTTP request)
2. send sigterm signal before end of the activity

In such scenario activity finishes but is not able to send result to temporal with grpc client.

Relevant log output

2024-08-12T06:59:36+02:00	{"level":"info","ts":1723438776025103636,"logger":"temporal","msg":"Task processing failed with error","Namespace":"orders-service","TaskQueue":"default","WorkerID":"default:159998ee-1ff5-4ab9-bfc6-ad74e2924210","WorkerType":"ActivityWorker","Error":"grpc: the client connection is closing"}
2024-08-12T06:59:36+02:00	{"message":"API Request finished"}
2024-08-12T06:59:35+02:00	{"level":"info","ts":1723438775062632589,"logger":"server","msg":"destroy signal received","timeout":60}
2024-08-12T06:59:35+02:00	{"level":"info","ts":1723438775062567608,"logger":"server","msg":"destroy signal received","timeout":60}
2024-08-12T06:59:35+02:00	{"level":"info","ts":1723438775062591477,"logger":"server","msg":"destroy signal received","timeout":30}
2024-08-12T06:59:35+02:00	{"level":"info","ts":1723438775062558877,"logger":"temporal","msg":"Stopped Worker","Namespace":"orders-service","TaskQueue":"default","WorkerID":"default:159998ee-1ff5-4ab9-bfc6-ad74e2924210"}
2024-08-12T06:59:35+02:00	stop signal received, grace timeout is: 30 seconds
2024-08-12T06:59:34+02:00	{"message":"API Request started"}

[kkosowski-p]

Example how to reproduce: https://github.com/pfy-oleksii-storozhylov/samples-php/pull/1/files

[rustatian]

@rustatian (note for myself):

1. Server plugin -> update Stop mechanism, remove central pool stop mechanism.
2. Update all plugins using pool to directly manage their pools of workers.
3. Temporal stop order (?): stop T workers -> stop PHP workers -> close T client connection.