Botbuilder AI QnAMaker scoreThreshold should be between 0 and 100

[araver]

Versions

What package version of the SDK are you using. 4.7.2
What nodejs version are you using: 10
What browser version are you using: NA
What os are you using: windows

Describe the bug

The sdk throws an exception when the threshold is greater than 1. But QNA Maker's API uses scoring from 0 to 100.

To Reproduce

Use a scoreThreshold greater than 1

Expected behavior

SDK shouldn't throw an exception when scoreThreshold is greater than 1

[bug]

[stevengum]

@araver is this where the exception is being thrown from?

JS:

botbuilder-js/libraries/botbuilder-ai/src/qnamaker-utils/generateAnswerUtils.ts

Lines 157 to 163 in aa2723a

	private validateScoreThreshold(scoreThreshold: number): void {
	if (typeof scoreThreshold !== 'number' || !(scoreThreshold > 0 && scoreThreshold < 1)) {
	throw new TypeError(
	`"${ scoreThreshold }" is an invalid scoreThreshold. QnAMakerOptions.scoreThreshold must have a value between 0 and 1.`
	);
	}
	}

Link to C# code

[Zerryth]

@araver To confirm, it's throwing the error that tells you explicitly to make score thresholds between 0 and 1, right? (not some other thing that's breaking)

While you're right that the QnA Maker API does return its results from 0 to 100, the bot framework QnA Maker handles the score threshold from 0 to 1, using float values, as you've discovered. (Upon receiving the results from QnA Maker service, the framework divides scores by 100.)
--I'm not quite sure how long it's been that way or who made that decision originally.
And changing the behavior now to require input to be 0 - 100 range would break bots that currently implement a threshold.

I mean I guess we could convert with a "if ScoreThreshold is less than 1, multiply by 100" condition--but that doesn't seem graceful. What if someone originally put a ScoreThreshold of 1 in, intending it to be 100% score, and not 1%? We have no way of differentiating the author's intention...

---

Anyways, while conceding that it would be less confusing if both BF and QnA service itself were identical, if we agree that there's no way to gracefully change the input behavior of score threshold in bot framework to align with the scores that the qna service puts out, then this is probably something that's resolved by just a method documenting fix.

It's more apparent in the C# code in QnAMakerOptions.ScoreThreshold that the ScoreThreshold should be 0.0 - 1.0 range and type is float to help further hint. Not so much in the JS QnAMakerOptions docs, but it does mention it defaults to 0.3, which I guess is just hinting and not explicit.

And the errors thrown also inform users that ScoreThresholds should be 0 to 1, but running into errors shouldn't be the way people discover how to use our framework properly.

[simer99]

Thanks @araver for reporting and @Zerryth for responding. May be the task would be to make it explicit in the documentation.

Thanks.

[Zerryth]

reopening issue, for tracking
made quick PR updating for clarification

[araver]

@Zerryth there must have been a change in the api, because it is expecting a value from 0 to 100.

[araver]

Also, in my opinion, the SDK should be less opinionated about that sort of thing. If the API has a problem with the developer using a value that is not within its valid range, it should respond with an error. The SDK should document the understood values, but not prevent the developer from using an invalid value.

[Zerryth]

ah, I see @araver
I put in .9 as scoreThreshold, you would expect it to convert it to 90 before querying the QnA API

As you see, it does not.

It queries QnA with .9 scoreThreshold, which is by far lower than the 90% threshold I intended.
However, you actually DO get the correctly filtered out results as expected (using 0.0 to 1.0 range), once it reaches back to the SDK, because it then goes on to formatQnaResult() and divides the scores of each answer received back by 100. So if the score was 80, that becomes .8, which would not meet the minimum scoreThreshold of .9, therefore no result.

^ above the answer came back from QnA Service that has only a score of .81, even though our threshold is .9

But, once it sortsAnswersWithinThreshold, it filters against the scoreThreshold specified.

Therefore ultimately the answer of score 81% never makes it through, as intended.

---

So the TL;DR is, the SDK scoreThreshold option does work, it just does so inefficiently, as it always queries QnA API with a lower threshold than intended, which leads to more answers being return than intended, before the SDK then goes on to filter the results properly and return the correct results to the bot.

We can't change the input value behavior, as not introducing breaking changes to existing customers is a policy we uphold, so that customers have stability in our framework--could you imagine the frustration of your bot working one day, but then not the next, without you making changes? It's unfortunate that in this case the the scoreThreshold value input is different in the SDK than the API, but we can't change that now

I'll ask the team what we want to do about the inefficient behavior.