Planners vs auto completion strategy

[guru98]

Hi, with connectors we should have an abstraction vs the llm utilized for answer but obtain a real abstraction is not sample as different llm have very different behaviours. More specific my question is: if i would use plugins to answer a question and i want compare different llms is better use planner or realy on autocompletion features. I make some test using gpt-4 and mistral-large and using the same system prompt and plugin gpt-4 work exaclty i aspected while the plugin is never call in mistral. I use a different prompt more simple and the plugin is called and works. My feeling is that if the use of the plugin is not implicit in the user prompt but is a requirement in the system prompt and should be called after the llm build a complex answer it doesn't work. E.g. if I ask for an internal company procedure and i have a plugin that resolve the name of the persons in charge for the roles citated in the answer the plugin is correclty called in gpt-4 and not in mistral. If i ask to mistral about who is in charge for a role it call the plugin. So to mitigate it better use planners and in general use planners is a better choiche in this scenario?

[sophialagerkranspandey]

The planners are build on top of the Auto function calling so you will likely not see additional benefit by moving to them. We have noticed models are better at function calling, you may have to use something like OpenAI vs. MistralAI for this scenario.