Out of proc Support for Cancellation Tokens

Out-of-proc Support for Cancellation Tokens

In C# functions you can take a CancellationToken argument to allow your function to detect that a host shutdown is occurring and abort gracefully. This work item tracks providing an equivalent mechanism for languages that execute out-of-proc.

Important Note: This is not a new feature. As mentioned, the host already exposes cancellation tokens for in-proc C# function applications. We will be providing the ability for users to react to a cancellation signal. For example, users can bind to a cancellation token which is provided by the host, and observe the token. If a cancellation is signalled, users can react by implementing logic to handle a cancellation scenario.

• GitHub Issue
• In-proc C# CancellationToken Documentation

1 Requirements

1.1 Goals

1. Expose the existing cancellation feature to out-of-proc language workers, enabling them to handle a cancellation signal from the host

1.2 Non-goals

1. Cancellation design for each individual language worker
2. We will not be implementing a new feature for cancellations

1.3 Use Cases

This is a benefit to end-users as it will give them the ability to handle cancellations per invocation. Users would be able to cancel downstream calls to other APIs, and cleanup resources. It may be more efficient for users to handle termination, rather than having an abrupt termination of operations.

Timeouts

For in-proc C# application, we currently signal CancellationTokens for timeout scenarios. This enables users to clean up resources and operations when the function has timed out, allowing them to avoid an abrupt end.

Timer Triggers

Sometimes an issue can occur where we'll end up running two copies of a timer trigger at once. If we detect that, we'll cancel one of them but for that to play nicely with user code, it requires the cancellation token pattern. By enabling cancellation tokens here, users can avoid potential duplications downstream.

Customer code to avoid duplication on retries

Users sometimes write code that tries to avoid duplicates in the case of retries for messaging triggers. For example, lets say an EventHub triggered function is running and it loses the lease on the partition. This means another worker is going to run the same function execution using the same data. The cancellation token gets fired if there is one, and the user can react to that by aborting the step they are doing, which might involve updating a database or writing another message somewhere.

Durable Functions

This is out-of-scope for this work item as it requires a new feature to be scoped out. The goal of this work item is to expose an existing feature to out-of-proc language workers, there should be nothing new.

Durable users want the ability to cancel in-flight Activity invocations when its parent orchestration is cancelled. This would require each language to have some way that the user can write code that checks for cancellation, which in turn requires some host support.

• GitHub Issue

This issue appears to propose that the Durable Functions (DF) Extension fire a cancellation token to the Function host to signal cancellation of an in-flight Activity function.

2 Partner Dependencies

• Azure Functions out-of-proc language owners
  • We will need to work with language owners to define the cancellation contract between the host and out-of-proc workers

3 Functional Description

3.1 InvocationCancel gRPC Message

1. Extend the gRPC contract to include an operation that the host can call on a worker to indicate that a given execution is cancelled.

2. Each language worker would then figure out how to map that to the user code.

   For example, in .NET isolated it would mean that the isolated worker calls Cancel on the CancellationTokenSource instance for the associated invocation. In Javascript, it might be something like a property on the context object that user code can check e.g. if (context.isExecutionCancelled)

3.2 Host Implementation

This feature will be built behind a worker capability flag. If a worker does not have a cancellation capability, we will not send the InvocationCancel message.

Prior to invoking a Function, the host would subscribe to a CancellationToken and associate it to the invocation result. First we would need to request the cancellation token (CT) be passed in the context parameters from WebJobs. We can then flow the CT to the InvokeAsync method and eventually to SendInvocationRequest.

// WorkerFunctionDescriptorProvider.cs

// GetFunctionParametersAsync() - Add the cancellation token from WebJobs to the invocation parameters
parameters.Add(new ParameterDescriptor(ScriptConstants.SystemCancellationTokenParameterName, typeof(CancellationToken)));

// WorkerFunctionInvoker.cs

// InvokeCore() - Update the invocation context to pass through the cancellation token
CancellationToken = (CancellationToken)parameters[cancellationTokenParameterIndex],

Within SendInvocationRequest, we can register a callback for the CT that we can use to send the InvocationCancel message. We will also need to unregister the callback if we get an InvocationResponse from the worker.

// GrpcWorkerChannel.cs

// SendInvocationRequest()
context.CancellationToken.Register(() => SendInvocationCancel(invocationRequest.InvocationId));

// SendInvocationCancel()
- Check if the worker has the `HandlesInvocationCancelMessage` capability
- Send a `InvocationCancel` message with the InvocationId

Worker Implementation

For workers to enable this feature, they will need to:

1. Provide the HandlesInvocationCancelMessage capability in the WorkerInitResponse message
2. Implement a handler for the InvocationCancel proto message
   • It is up to workers how they handle the cancellation message

4 Interfaces

4.1 Protobuf

The current protobuf has a definition for InvocationCancel message that we can use and potentially extend. The idea is that the host sends a cancel message to attempt to cancel an invocation. If an invocation is cancelled, the host will receive an invocation response with status cancelled. StatusResult already supports a "cancelled" state.

// Host requests worker to cancel invocation
message InvocationCancel {
  // Unique id for invocation
  string invocation_id = 2;

  // We will not use this property as we cannot reliability provide a grace period
  // As removing this may be a breaking change, we can set it to always be a negative
  // value (i.e. `-1`) to indicate infinite/ not applicable
  google.protobuf.Duration grace_period = 1;
}

InvocationCancel proto message

5 How to use the cancellation token

5.1 Dotnet Isolated

5.2 Node

5.3 Java

5.4 Python

5.5 Powershell