Tool calling fails on NVIDIA GPUs

[natke]

Foundry Local version: 0.8.117 (RC)

Error:

Debug: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:17.12 0 Tokenizer.ApplyChatTemplate -> prompt:<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"name": "get_horoscope", "description": "Get today's horoscope for an astrological sign.", "parameters": {"sign": {"type": "str", "description": "An astrological sign like Taurus or Aquarius"}}}
{"name": "get_apps", "description": "Get a list of available apps installed on the system.", "parameters": {}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>assistant

Debug: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:17.12 0 Tokenizer.Encode -> input_ids:Microsoft.ML.OnnxRuntimeGenAI.Sequences
Debug: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:17.12 0 Guidance.Type -> lark_grammar
Debug: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:17.12 0 Guidance.Data -> start: toolcall
toolcall: <tool_call> functioncall </tool_call>
functioncall: %json {
  "x-guidance": {
    "whitespace_flexible": false,
    "key_separator": ": ",
    "item_separator": ", "
  },
  "type": "array",
  "items": {
    "anyOf": [
      {
        "description": "Get today's horoscope for an astrological sign.",
        "type": "object",
        "properties": {
          "name": {
            "const": "get_horoscope"
          },
          "parameters": {
            "additionalProperties": false,
            "type": "object",
            "properties": {
              "sign": {
                "type": "string",
                "description": "An astrological sign like Taurus or Aquarius"
              }
            },
            "required": [
              "sign"
            ]
          }
        },
        "required": [
          "name",
          "parameters"
        ],
        "additionalProperties": false
      },
      {
        "description": "Get a list of available apps installed on the system.",
        "type": "object",
        "properties": {
          "name": {
            "const": "get_apps"
          },
          "parameters": {
            "additionalProperties": false,
            "type": "object",
            "properties": {},
            "required": []
          }
        },
        "required": [
          "name",
          "parameters"
        ],
        "additionalProperties": false
      }
    ]
  },
  "minItems": 1
}
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:18.51 0 AppendTokenSequences -> numTokensAppended:196
Debug: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:18.93 0 Replacing cached generator. RefCount of cached:1
Debug: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:18.96 0 RefCountedChatGenerator Dispose called. 0 remaining references.
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:18.96 0 Using new generator
Debug: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx[0]
   02:35:18.96 0 RefCountedChatGenerator Dispose called. 1 remaining references.
Error: Microsoft.Neutron.OpenAI.Delegates.OpenAIApi[0]
   02:35:18.96 0 Failed to handle openAI completion message:Error committing tokens: Parser Error: token "<" doesn't satisfy the grammar; forced bytes: got 'ÿ'; applying '<'
<state>
Tokens: ⟦<⟧
1 tokens, 0 bytes; grm_prefix: ""
Flags:
Parser: {
  "compute_time_us": 0,
  "rows": 6,
  "cached_rows": 0,
  "all_items": 13,
  "lexer_cost": 3806,
  "slices_applied": 0,
  "trie_nodes_walked": 0,
  "definitive_bytes": 24,
  "lexer_ops": 0,
  "num_lex_errors": 0,
  "num_lexemes": 0
}
Stop: ParserTooComplex
Error: Parser Error: token "<" doesn't satisfy the grammar; forced bytes: got 'ÿ'; applying '<'
</state><grammar>

</grammar> stack:   at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x47
   at Microsoft.ML.OnnxRuntimeGenAI.Generator.GenerateNextToken() + 0x1b
   at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase`1.<HandleChatCompletionRequestAsync>d__42.MoveNext() + 0x88c
--- End of stack trace from previous location ---
   at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase`1.<HandleChatCompletionRequestAsync>d__42.MoveNext() + 0xb7f
--- End of stack trace from previous location ---
   at Microsoft.Neutron.OpenAI.Delegates.OpenAIApi.<ChatCompletionsAsync>d__0.MoveNext() + 0xbf1

Device specs: