Issue with input serialization in Durable Function with Orchestrator

[fcu423]

Description

Using .NET 8 and azure functions in isolated mode, I have an EventGridTrigger function that starts up an orchestrator while providing a typed input as parameter. The parameter's type is part of a class hierarchy. The orchestrator is receiving the input with the type of the base class and I am pattern checking this input against the type of the two subclasses to call the durable entity in one way or another.

The issue is that the serializer is not respecting the System.Text.Json attributes to include the type hints so the input received in the orchestrator doesn't have any polymorphism information/data about the types that originated it.

Expected behavior

I would expect to be able to check the input of the orchestrator against the type of any of the children classes.

Actual behavior

Pattern checking the base class against any of the children classes is always false.

Relevant source code snippets

The class hierarchy

[JsonDerivedType(typeof(IntegrationEventBase), typeDiscriminator: "IntegrationEventBase")]
[JsonDerivedType(typeof(WorkItemIntegrationEventBase), typeDiscriminator: "WorkItemIntegrationEventBase")]
public class IntegrationEventBase
{
    public string? Traceparent { get; set; }
}

[JsonDerivedType(typeof(WorkItemCreatedIntegrationEvent), typeDiscriminator: "WorkItemCreatedIntegrationEvent")]
[JsonDerivedType(typeof(WorkItemProgressedIntegrationEvent), typeDiscriminator: "WorkItemProgressedIntegrationEvent")]
public class WorkItemIntegrationEventBase : IntegrationEventBase
{
    public string? WorkItemId { get; set; }

    public ushort? Quantity { get; set; }
}

public class WorkItemCreatedIntegrationEvent : WorkItemIntegrationEventBase
{
}

public class WorkItemProgressedIntegrationEvent : WorkItemIntegrationEventBase
{
    public WorkItemStatus Status { get; set; }
}

The event grid function

[Function("WorkItemProgressedIntegrationEventHandler")]
public async Task WorkItemProgressedIntegrationEventHandler([EventGridTrigger] EventGridEvent eventGridEvent,
    [DurableClient] DurableTaskClient client,
    FunctionContext executionContex)
{
    var @event = _serializer.Deserialize<WorkItemProgressedIntegrationEvent>(eventGridEvent.Data.ToString());

    await client.ScheduleNewOrchestrationInstanceAsync(
        Constants.WORKFLOW_ORCHESTRATOR, @event);
}

The orchestrator

[Function("Orchestrator")]
public static async Task RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context, WorkItemIntegrationEventBase @event)
{
    var entityId = BuildEntityIds(@event);

    if (@event is WorkItemCreatedIntegrationEvent createdEvent) // This is false
        await context.Entities.SignalEntityAsync(workflowEntityId, nameof(IDurableWorkflow.EnqueueWork),
            WorkItemInput.FromWorkItemIntegrationEvent(createdEvent));
    else if (@event is WorkItemProgressedIntegrationEvent progressedEvent) // This is also false
        await context.Entities.SignalEntityAsync(workflowEntityId, nameof(IDurableWorkflow.ProgressWork),
            WorkItemInput.FromWorkItemIntegrationEvent(progressedEvent));
}

Known workarounds

As a workaround I:

1. Removed the input as parameter from the orchestrator
2. Included an Operation enum in the base class
3. In the orchestrator I am manually getting the input by calling context.GetInput<WorkItemCreatedIntegrationEvent>(); or context.GetInput<WorkItemProgressedIntegrationEvent>(); based on the value of the Operation enum

App Details

• Microsoft.Azure.Functions.Workter.Extensions.DurableTask v1.1.1:
• Microsoft.Azure.Functions.Worker v1.21.0
• Azure Functions runtime version 2.0 isolated mode:
• C# - .NET 8:

[lasyan3]

Guess it's related to this closed issue : #2577
In my case I try to pass a class as input, the class is initialized but the values are not retrieved.

[cgillum]

@jviau are you aware of issues related to polymorphic (de)serialization with STJ?

[jviau]

I think I have ran into this before!

In dotnet isolated, durable extension uses the worker-wide configured converter from WorkerOptions - which is an Azure.Core.ObjectSerializer. I think their JSON implementation uses a set of APIs from STJ that for some reason don't run polymorphic serialization.

@fcu423 you can see if this repros for you locally: try serializing/deserializing via System.Text.Json.JsonSerializer and then via Azure.Core.Serialization.JsonObjectSerializer. If I remember correctly, the first will respect the polymorphic attributes, the second will not.

We will need to consider how to let a customer specify a separate convert for just durable. It might be possible today via adding your own PostConfigure call:
https://github.com/Azure/azure-functions-durable-extension/blob/dev/src/Worker.Extensions.DurableTask/DurableTaskExtensionStartup.cs#L34

[amalea]

Hello @jviau,

I have the same issue while using:

• Microsoft.Azure.Functions.Worker.Extensions.DurableTask v1.1.2
• Microsoft.Azure.Functions.Worker.Sdk v1.15.0
• Azure Functions runtime isolated mode:
• C# - .NET 8:

Calling ScheduleNewOrchestrationInstanceAsync with input param of type Message, both fields are assigned the correct values:

await client.ScheduleNewOrchestrationInstanceAsync(
       nameof(MyOrchestrator),
       new Message
       {
           Id = calloutData.Result.Id,
           EntityId = AppEntityId.FromString(calloutData.Result.EntityId.ToString()),
       },
       new StartOrchestrationOptions { InstanceId = rowKey });

public class Message
{
    public Guid Id { get; set; }
    public AppEntityId EntityId { get; set; }
   
}

But, when trying to retrieve ctx.GetInput(); --> Id has the correct Guid value and EntityId is an empty Guid (even if I have checked that it is assigned before the call).

Could you please suggest me how can I handle this? Thanks.

[jviau]

@amalea what is AppEntityId? Can you verify serializing and deserializing this in a unit test via Azure.Core.Serialization.JsonObjectSerializer?

[amalea]

Hello @jviau,

AppEntityId is a struct, like this:

 public struct AppEntityId : IEquatable<AppEntityId >, IComparable<AppEntityId >, IComparable
 {
     public Guid Value { get; }

     public AppEntityId (Guid value) => Value = value;

     public static AppEntityId FromString(string value) => new AppEntityId (Guid.Parse(value));
}

I am not sure where exactly should I use serialization/deserialization in the above code.

[jviau]

@amalea, this doesn't look like it is related to this issue as you are not using polymorphic serialization. Yours looks like a general serialization issue external to durable. I recommend you write unit tests to validate you can serialize/deserialize your payload with System.Text.Json outside of durable.

[amalea]

Hello @jviau,

I have tried to isolate the problem and it seems that using struct type does not work with durable functions isolated worker / or some other lib has a wrong impact.

  public class Message
    {
         public InputModel Id { get; set; }
    }

   public struct InputModel
    {
         public Guid Id { get; }
         public InputModel(Guid value) => Id = value;
         public static InputModel FromString(string value) => new InputModel(Guid.Parse(value));
    }
          
     await client.ScheduleNewOrchestrationInstanceAsync(
         nameof(MyOrchestrator),
         new Message
           {
              Id = InputModel.FromString(data)
           },
         new StartOrchestrationOptions { InstanceId = rowKey });
            
    //after this call, Id is an empty Guid, even if the value is passed correctly, but that TaskOrchestrationContext GetInput modifies something and I can't find what/where        
    var message = ctx.GetInput<Message>();

In Program.cs, I have this line also:
services.Configure<JsonSerializerOptions>(o => o.IncludeFields = true);