Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scoreThreshold no longer works in 4.17.0 and later #4568

Closed
dmvtech opened this issue Nov 14, 2023 · 3 comments · Fixed by #4575
Closed

scoreThreshold no longer works in 4.17.0 and later #4568

dmvtech opened this issue Nov 14, 2023 · 3 comments · Fixed by #4575
Assignees
Labels
Bot Services Required for internal Azure reporting. Do not delete. Do not change color. bug Indicates an unexpected problem or an unintended behavior. customer-replied-to Indicates that the team has replied to the issue reported by the customer. Do not delete. customer-reported Issue is created by anyone that is not a collaborator in the repository.

Comments

@dmvtech
Copy link

dmvtech commented Nov 14, 2023

Describe the bug

When supplying scoreThreshold, it is not honored and will return the answer even though the score is much lower.

Versions

4.17.0 and later
Works in 4.16.0

Describe the bug

When supplying scoreThreshold, it is not honored and will return the answer even though the score is much lower.

const qnaOptions = {scoreThreshold:0.65};
const qnaResults = await this.qnaMaker.getAnswers(context,qnaOptions);

To Reproduce

Steps to reproduce the behavior:

  1. Use sample such as: https://github.com/microsoft/BotBuilder-Samples/tree/releases/4.16/samples/javascript_nodejs/11.qnamaker
  2. Add options to use the scoreThreshold
    const qnaOptions = {scoreThreshold:0.65};
    const qnaResults = await this.qnaMaker.getAnswers(context,qnaOptions);
  3. Test with a question that has a score that will not meet the threshold
  4. See that you will get a valid answer anyway

Expected behavior

Should give No QnA Maker answers were found. or similar error as the answer is not above the threshold.

Additional context

  • .NET SDK does not seem to have the issue.
  • Note that we (at least in the JS SDK, do not send a valid scoreThreahold to QnA endpoint. QnA endpoint will use 0-100 for the threshold, and the SDK uses a float 0.0-1 and therefore will almost always hit the QnA threshold. We then filter/compare after the results returned and determine if it is a valid answer.

Somewhat related issue:
#1808

@dmvtech dmvtech added bug Indicates an unexpected problem or an unintended behavior. needs-triage The issue has just been created and it has not been reviewed by the team. customer-reported Issue is created by anyone that is not a collaborator in the repository. Bot Services Required for internal Azure reporting. Do not delete. Do not change color. customer-replied-to Indicates that the team has replied to the issue reported by the customer. Do not delete. labels Nov 14, 2023
@dmvtech
Copy link
Author

dmvtech commented Nov 14, 2023

@ceciliaavila Let me know if there are any questions. I can reproduce and will try and add some screenshots shortly.

@tracyboehrer tracyboehrer removed the needs-triage The issue has just been created and it has not been reviewed by the team. label Nov 14, 2023
@dmvtech
Copy link
Author

dmvtech commented Nov 15, 2023

In my example;

  • The question What are pickles? has a score of 100
  • The question Do you know those cucumbers in vinegar? has a score of 46.31
  • The scoreThreshold in the bot is set to 0.65

Working as expected in 4.16.0

image

Failing in 4.17.0

image

@tracyboehrer
Copy link
Member

Possibly related to this: #4208

tracyboehrer pushed a commit that referenced this issue Nov 30, 2023
* Restore score threshold filter in QnAMaker results

* Add missing dot in comment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bot Services Required for internal Azure reporting. Do not delete. Do not change color. bug Indicates an unexpected problem or an unintended behavior. customer-replied-to Indicates that the team has replied to the issue reported by the customer. Do not delete. customer-reported Issue is created by anyone that is not a collaborator in the repository.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants