-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Model chat: allow multiple final ratings #4276
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a cool additional feature! Happy to approve.
From an API design perspective, I wonder about the choice to |
-separate your questions and output vs providing both as an array. I suppose the issue here is backwards-compatibility?
let all_ratings_filled = true; | ||
let rating = ""; | ||
for (let i = 0; i < ratings.length; i++) { | ||
if (ratings[i] === "") { | ||
all_ratings_filled = false; | ||
} | ||
if (i === 0) { | ||
rating += ratings[i]; | ||
} else { | ||
rating += "|" + ratings[i]; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could be:
let all_ratings_filled = ratings.every((r) => r !== "")
let rating = ratings.join('|')
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ha, great - thanks, changed
Yeah, exactly - a bunch of projects seem to currently be relying on model chat, making breaking changes costly, and I'm not convinced of how often we'll want to use multiple final questions anyway. I'm definitely open to changing to the array syntax, which would indeed be more ideal, down the line |
Patch description
In the model chat crowdsourcing task, ask for and save multiple 1-to-5 ratings at the end of the HIT. Questions can be specified through
mephisto.blueprint.final_rating_question
, with multiple questions separated by a pipe character. Multiple ratings will be saved in the format'2|4|3'
.Testing steps
With 1 final question:
With multiple final questions: