Ensure extension RPC endpoints ready before processing gRPC messages #10255
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jviau
force-pushed
the
jviau/grpc/fix-startup-race
branch
from
6f6d1ab
to
a55bf13
Compare
fabiocav
approved these changes
Jun 28, 2024
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
brettsam
approved these changes
Jul 10, 2024
jviau
added a commit
that referenced
this pull request
Jul 11, 2024
…10255) * Ensure extension RPC endpoints ready before processing gRPC messages * Add timeout and tests to waiting on RPC extensions.
v-imohammad
added a commit
that referenced
this pull request
Jul 16, 2024
* JIT Trace - 4.35.0 using in-proc as base (#10281) * Adding a timeout to `IFunctionProvider.GetFunctionMetadataAsync` (#10249) (#10273) * Adding a timeout to IFunctionProvider.GetFunctionMetadataAsync so that if a provider does not return, it will not cause a deadlock state. * Adding release notes. * Added a new internal property `MetadataProviderTimeoutInSeconds` which can be set to a different value than default, from the tests. * Added comment about property being settable. * PR feedback fixes. Throws an error when the function metadata provider method throws or when the operation timeseout. * [in-proc port] Ensure extension RPC endpoints ready before processing gRPC messages (#10282) * Ensure extension RPC endpoints ready before processing gRPC messages (#10255) * Ensure extension RPC endpoints ready before processing gRPC messages * Add timeout and tests to waiting on RPC extensions. * update release_notes.md * [in-proc] Update Microsoft.Azure.WebJobs to 3.0.41 and Microsoft.Azure.WebJobs.Host.Storage to 5.0.1 (#10288) * Upgraded the following package versions: - Microsoft.Azure.WebJobs updated to 3.0.41 - Microsoft.Azure.WebJobs.Host.Storage updated to 5.0.1 - Microsoft.Extensions.Azure updated to 1.7.1 - Azure.Storage.Blobs updated to 12.19.1 * Updating deps.json for Tests. --------- Co-authored-by: Shyju Krishnankutty <connectshyju@gmail.com> Co-authored-by: Jacob Viau <javia@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue describing the changes in this PR
resolves #10251
Pull request checklist
IMPORTANT: Currently, changes must be backported to the
in-procbranch to be included in Core Tools and non-Flex deployments.in-procbranch is not requiredrelease_notes.mdAdditional information
It is believed there is a race condition on startup between the first gRPC message coming in and extension RPC endpoints being registered. While
EndpointDataSourcehas a change token system, leading us to believe that we can update endpoints post startup, there is one of two possibilities occurring:UseRoutingmiddleware is first encountered. In which case this host instance would not have extension RPC endpoints for its lifetime.This fix should address both of those possible scenarios, as routing middleware is only initialized on first call. By ensuring we collect extension endpoints before the first routing occurs we avoid the race condition.
RISK: there is a small risk this could introduce a deadlock. If there is some dependency existing, or later introduced, which requires RPC communication between host and worker before all extensions are loaded host-side, then this could cause a circular dependency and deadlock. I do not believe this is the case today, as testing has shown endpoints to be available immediately on startup before the first worker RPC message comes in.