Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Genkitx-openai generate response candidate not properly generated when streamingCallback is used #137

Open
1 of 7 tasks
pedi opened this issue Sep 3, 2024 · 0 comments · May be fixed by #138
Open
1 of 7 tasks
Labels
bug Something isn't working

Comments

@pedi
Copy link

pedi commented Sep 3, 2024

Affected plugin(s)

  • Anthropic
  • Azure OpenAI
  • Cohere
  • Convex
  • Groq
  • Mistral
  • OpenAI

Describe the bug
Using the sample flow generated by Genkit

const prompt =
      `Suggest an item for the menu of a ${subject} themed restaurant`;
    const llmResponse = await generate({
      model: gpt4o,
      prompt: prompt,
      config: {
        temperature: 1,
      },
      streamingCallback: streamingCallback,
    });

    console.log('history: ', llmResponse.toHistory());

The output of the history will be

history:  [
  { role: 'user', content: [ [Object] ] },
  { role: 'model', content: [] }
]

The user's original prompt is recorded but the model's response is empty

The issue is due to the following line
https://github.com/TheFireCo/genkit-plugins/blob/main/plugins/openai/src/gpt.ts#L379

When calling

response = await stream.finalChatCompletion();

The response looks like below

response {
  role: 'assistant',
  content: 'Item: "The Exam Platter"\n' +
    '\n' +
    'Description: A sampler platter that offers a bit of everything to test your taste buds. It includes:\n' +
    '\n' +
    '- Pop Quiz Poppers: Bite-sized jalapeño poppers with a surprise spicy kick.\n' +
    '- Study Group Sliders: Mini beef, chicken, and veggie sliders, each with unique toppings to represent different study preferences.\n' +
    '- Multiple Choice Mozzarella Sticks: Classic mozzarella sticks served with a variety of dipping sauces - marinara, ranch, and spicy buffalo.\n' +
    '- Flashcard Fries: A generous portion of seasoned curly fries, waffle fries, and sweet potato fries for a mix-and-match experience.\n' +
    '- True/False Tenders: Chicken tenders served with an assortment of savory dips like honey mustard, BBQ, and buffalo.\n' +
    '\n' +
    'Perfect for those who love a bit of variety and want to ace the culinary experience!',
  refusal: null,
  parsed: null,
  tool_calls: []
}

Which contains an empty array of tool_calls.

The current code only check for truthiness of toolRequestParts, but we should also check its length instead.

@pedi pedi added the bug Something isn't working label Sep 3, 2024
@pedi pedi linked a pull request Sep 3, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant