Custom runtime defined Enums

[Bodekaer]

Hi @StephenHodgson ,
I find the new strict + enum feature very helpful, by forcing GPT to pick one of the available Enums.
The downside is that they need to be strictly defined in C# code.

This doesn't seem to a restriction for OpenAI though, and especially with Tools being possible to set/defined at each Run, it would be very useful to be able to provide a non-defined enum as possible string value options for GPT to choose from.

E.g. instead of:

enum PossibleEnumValues { a, b, c }
PossibleEnumValues myEnum;

Do you see a way for us to define it as something like:

string[] possibleEnumValues = { "a", "b", "c" };
string myEnum;

... magic here ... :)

Alternatively, are you aware of other ways to restrict the possible values of a field/string?

[StephenHodgson]

You could pass json string directly.

var mathSchema = new JsonSchema("math_response", @"
{
  ""type"": ""object"",
  ""properties"": {
    ""steps"": {
      ""type"": ""array"",
      ""items"": {
        ""type"": ""object"",
        ""properties"": {
          ""explanation"": {
            ""type"": ""string""
          },
          ""output"": {
            ""type"": ""string""
          }
        },
        ""required"": [
          ""explanation"",
          ""output""
        ],
        ""additionalProperties"": false
      }
    },
    ""final_answer"": {
      ""type"": ""string""
    }
  },
  ""required"": [
    ""steps"",
    ""final_answer""
  ],
  ""additionalProperties"": false
}");
var assistant = await OpenAIClient.AssistantsEndpoint.CreateAssistantAsync(
    new CreateAssistantRequest(
        name: "Math Tutor",
        instructions: "You are a helpful math tutor. Guide the user through the solution step by step.",
        model: "gpt-4o-2024-08-06",
        jsonSchema: mathSchema));
ThreadResponse thread = null;

try
{
    var run = await assistant.CreateThreadAndRunAsync("how can I solve 8x + 7 = -23",
        async @event =>
        {
            Debug.Log(@event.ToJsonString());
            await Task.CompletedTask;
        });
    thread = await run.GetThreadAsync();
    run = await run.WaitForStatusChangeAsync();
    Debug.Log($"Created thread and run: {run.ThreadId} -> {run.Id} -> {run.CreatedAt}");
    var messages = await thread.ListMessagesAsync();

    foreach (var response in messages.Items)
    {
        Debug.Log($"{response.Role}: {response.PrintContent()}");
    }
}
finally
{
    await assistant.DeleteAsync(deleteToolResources: thread == null);

    if (thread != null)
    {
        var isDeleted = await thread.DeleteAsync(deleteToolResources: true);
        Assert.IsTrue(isDeleted);
    }
}

[Bodekaer]

In the Weather example, the Enum is hardcoded, so I understand how this works.

But what if you want to change the Enum options at runtime?

As I understand it, the Schema generated in the Weather is example is something like:

{
    "type": "string",
    "enum": ["Celsius", "Fahrenheit"]
}

I would like to be able to change the enum options only, at runtime.
I can still only think of manually editing the JSONSchema after it has been generated for a string variable, and then inserting the enum: [ ... runtime_options_here ... ]

Another way to imagine this, is if you had to define the Weather example unit as a string as well. How would you limit the OpenAI API to use specific Enum options, without defining it as an Enum? :)

public static async Task<string> GetCurrentWeatherAsync(string location, string unit)

[StephenHodgson]

Current schema generation produces the following:

  {
    "type": "function",
    "function": {
      "name": "OpenAI_Tests_Weather_WeatherService_GetCurrentWeatherAsync",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The location the user is currently in."
          },
          "unit": {
            "type": "string",
            "enum": [
              "celsius",
              "fahrenheit"
            ],
            "description": "The units the user has requested temperature in. Typically this is based on the users location."
          }
        },
        "required": [
          "location",
          "unit"
        ],
        "additionalProperties": false
      },
      "strict": true
    }
  }

[StephenHodgson]

I don't see why you couldn't modify the enums as you see fit, but c# is rather picky about knowing types ahead of time (AOT).

Enums are typically "known" values. The way you are describing the usage, seems more like an arbitrary list of strings, and it seems more prudent to use an array.

[Bodekaer]

Ahh. I wanted to change the Enum, but a runtime. It's clear I can simply change them at compile time, but it would be very useful to be able to adjust them at runtime as well, as I don't always have all Enum options available at compile time.
In this specific example, I use Unity FlowCanvas to allow the game designers to modify the available options in visual scripting.
In this case, I cannot define the Enum at compile time.

[StephenHodgson]

I mean functionally you could still implement it as an array, and use a switch statement on the values.

It's just not very useful because c# isn't a functional programming language like JavaScript or Python

[Bodekaer]

@StephenHodgson I found a cumbersome way to make this possible. I'm not really happy with the complicated code, and I wonder if you can think of a cleaner/simpler solution?

I basically just add this RuntimeEnum extension to TypeExtensions.cs:

    public static class TypeExtensions
    {

        public class RuntimeEnum<T>
        {
            private static Dictionary<Type, string[]> _possibleValues = new Dictionary<Type, string[]>();

            public string Value { get; set; }

            public static string[] PossibleValues 
            { 
                get => _possibleValues[typeof(T)];
                set => _possibleValues[typeof(T)] = value;
            }

            public static implicit operator RuntimeEnum<T>(string x)
            {  
                return new RuntimeEnum<T>() { Value = x }; 
            }
        }


        public static JObject GenerateJsonSchema(this Type type, JObject rootSchema, JsonSerializer serializer = null)
        {
            ...
            else if (type.IsGenericType == true && type.GetGenericTypeDefinition() == typeof(RuntimeEnum<>)) 
            { 
                schema["type"] = "string";
                schema["enum"] = new JArray();

                var possibleValuesProperty = type.GetProperty("PossibleValues", BindingFlags.Public | BindingFlags.Static);
                var possibleValues = possibleValuesProperty.GetValue(null) as string[];

                foreach (var value in possibleValues ?? new string[0])
                {
                    ((JArray)schema["enum"]).Add(JToken.FromObject(value, serializer));
                }
            }
            else
            {
                ...
            }
        }
}

And then I can at runtime define custom enum values:

  public class SomeClass 
  {
    RuntimeEnum<AnyClass> MyEnum = new RuntimeEnum<AnyClass>();

    public SomeClass()
    {
       RuntimeEnum<AnyClass>.PossibleValues = new string[] { "Value 1", "Value 2", "Value 3" };
    }
  }

It all works fine. Just feels like there should be a simpler solution :)
Let me know what you think :)

[StephenHodgson]

Yeah part of the problem you're running into is being able to serialize the dictionary.

Does seem rather complicated and convoluted, plus a wrapper class just to hold it.

[StephenHodgson]

There might be an even easier way to do it with a custom json converter.

[Bodekaer]

I tweaked it a bit more and works now without the wrapper class.
I still need a "object/type class" to be able to distinguish the Runtime Enums possible values (AnyClass in example above), but it's a bit cleaner now. Updated above.

Good enough for now, but ping me if you get any other brilliant ideas for a cleaner approach :)

[StephenHodgson]

I think the best thing to do is probably add attribute for this and then have it check in TypeExtensions similar to what we do for functions.

Most likely you can use the same attributes.

[Bodekaer]

Yes that would be much cleaner. Good idea. Can you share example of how to change a method parameter attribute at runtime? That's the only thing I'm not sure how to do.

…

On Sat, Sep 14 2024 at 16:35, Stephen Hodgson < ***@***.*** > wrote: I think the best thing to do is probably add attribute for this and then have it check in TypeExtensions similar to what we do for functions. Most likely you can use the same attributes. — Reply to this email directly, view it on GitHub ( #289 (reply in thread) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AACJE4GUECOJLMARQ5BXDCDZWTB4HAVCNFSM6AAAAABNEPBGNKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRUHEZTOOA ). You are receiving this because you modified the open/close state. Message ID: <RageAgainstThePixel/com. openai. unity/repo-discussions/289/comments/10649378 @ github. com>

[Bodekaer]

Ahh. Maybe I'm missing something.
I'm looking for ways to set the schema["enum"] JSON Schema variable, but still only see it under "if (is enum) ...".
So my quick sense is that this won't enable changing an enum at runtime?

Something like the following would be interesting if you think it's possible:

public void SomeMethod(
    [FunctionParameter(Enum: ["Option_1", "Option_2"] )] string someVariable )
{
...
}

And then somehow the ability to change the parameter at runtime (pseudo code):

var method = GetMethod("SomeMethod");
var functionParameter = GetFunctionParameter(method, "someVariable");
functionParameter.Enum.Add("Option_3");

[StephenHodgson]

In your example you wouldn't be able to use attributes for runtime generation anyway.

I think you might be better off building json doc on the fly and passing that in either as string or JObject.

You could build an object up and serialize it as needed

[Bodekaer]

Can you clarify what you mean by "building a json doc and passing that in".
How would I pass it into the GenerateJsonSchema method?

[StephenHodgson]

Just build up an anon object or a defined object and then just serialize it to jsonSchema manually.

var jsonSchema = JsonSerializer.Serialize(myObject);

[Bodekaer]

I'm still not sure how that makes the Assistant use this Schema when submitting a CreateRun request :)
In any case, it sounds like the class RuntimeEnum<T> I made above is good enough for now, and that there isn't a much simpler/cleaner approach for now, so I can just use that for now. It works very well already as is, and now without the wrapper class needed.

[Bodekaer]

Hi @StephenHodgson,
We're updating the codebase to use the TypeExtensions.cs support for enum PossibleValues.
I've noticed though that the json schema isn't re-generated if I provide tools in a subsequent Run.
Since we change the possible enum values inbetween Runs, is there a simple way to tell the TypeExtensions to "reset"/recreate the definition for a specific object?

[StephenHodgson]

I've noticed though that the json schema isn't re-generated if I provide tools in a subsequent Run.

This is by design, because it was adding extra metadata and polluting requests.