Investigate the ordering of Init request and Metadata request in multi-worker scenarios

[surgupta-msft]

Ideally, it is expected that worker will receive WorkerInitRequest first and then FunctionMetadataRequest (in case of Stein) or FunctionLoadRequest (legacy scenario). However, by random chance, it seems the second worker may receive a function load request or a metadata request as its first request.

Need to investigate this. Ideally, these messages should be in predictable order (where WorkerInitRequest will be the first one).

Case A

Worker 1 gets an init request and then metadata request (as expected). However, worker 2 receives a function load request before an init request.

[2022-10-06T02:42:39.129Z] Worker 71780151-7b5a-4152-af87-95168dc0aacf received WorkerInitRequest
[2022-10-06T02:42:39.464Z] Worker 71780151-7b5a-4152-af87-95168dc0aacf received FunctionsMetadataRequest
[2022-10-06T02:42:39.514Z] Worker 71780151-7b5a-4152-af87-95168dc0aacf received FunctionLoadRequest
[2022-10-06T02:42:39.878Z] Worker bffcf551-f64f-48c7-808d-fba0659b37a1 received FunctionLoadRequest
[2022-10-06T02:42:39.886Z] Worker bffcf551-f64f-48c7-808d-fba0659b37a1 received WorkerInitRequest

Case B

Worker 1 is initialized, but worker 2 receives the metadata request before it is initialized.

[2022-10-06T02:36:12.135Z] Worker 554e325b-3cef-4db0-8626-6a32a4a67e7e received WorkerInitRequest
[2022-10-06T02:36:12.737Z] Worker d43a5b45-cb3e-45a8-ba89-a53548271186 received FunctionsMetadataRequest
[2022-10-06T02:36:12.741Z] Worker d43a5b45-cb3e-45a8-ba89-a53548271186 received WorkerInitRequest

[fabiocav]

@soninaren has completed the investigation on this and the underlying issue has been addressed with the selective loading changes tracked in #8900