[BFCL] Ambigious Instructions in Java/Javascript Test Cases

[tanliboy]

Description

The instructions in Java/Javascript test cases often request explanations or descriptions of function calls rather than directly requesting the construction of function calls. This leads to failures in the Function Calling (FC) model to generate the appropriate function calls.

Data Point Reference

1. Relevant Data Points / Model Handler Permalink:
   • Java Test Cases
   • JavaScript Test Cases

Identified Issue

In these test cases, the user prompt asks "How can I initiate" rather than directly requesting "Help me initiate." This subtle difference in wording leads to the model generating an explanation or an example of how the function can be called, rather than constructing the function call directly.

Example

Prompt:

{
  "role": "user",
  "content": "How can I initialize the GIS geometry presentation in a user interface, providing a specific result set controller mapController and a composite UI element mapArea to display the GIS data?"
}

Model's Answer:

{
  "role": "assistant",
  "content": "To initialize the GIS geometry presentation in a user interface, you can use the GeometryPresentation.createPresentation function. Here's an example of how to call this function with the provided result set controller mapController and composite UI element mapArea:\n\nString controller = \"mapController\";\nString parent = \"mapArea\";\n\nString result = GeometryPresentation.createPresentation(controller, parent);\n\nIn this example, the controller and parent parameters are passed as strings. The result variable will contain the output of the createPresentation function."
}

Analysis:

In this case, the model's response, which describes how to construct the function call, is more fitting than directly generating the function call. However, the original prompt's phrasing could be improved to align better with the intended function-calling task.

Proposed Solution

For all Java and JavaScript test cases, adjust the user prompts to explicitly request the action rather than an explanation. Using the previous example:

Original:

{
  "role": "user",
  "content": "How can I initialize the GIS geometry presentation in a user interface, providing a specific result set controller mapController and a composite UI element mapArea to display the GIS data?"
}

Revised:

{
  "role": "user",
  "content": "Help me initialize the GIS geometry presentation in a user interface, providing a specific result set controller mapController and a composite UI element mapArea to display the GIS data."
}

[HuanzhiMao]

Thanks for the issue @tanliboy
We will address those soon.