-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bucket sampling for preference experiments #586
Comments
Our approach for this has changed since the bug was initially filed, after talking with Sunnah and Gregg. Now, preference experiments specify a count of how many buckets out of 100,000 that they want to sample, and branches specify the ratio of matched users that they should match. Thus, the filtering for this simplifies to doing a normal, independent bucket sample of users based on that count. We can probably accomplish this within the preference experiment action too, which removes the need to refactor the backend to support this for all actions. Yay! |
…re enrolling users.
Per the discussion in #650, this is being moved to a non-blocker since we can write a filter expression that accomplishes this easily and has the same effect as the PR I put up. We still should address what UI we want to making this kind of filtering accessible to the audience we eventually expect will be using Normandy to launch preference experiments, but for an MVP it's not necessary. |
I think the solution to this lies in #316 and related efforts to simplify sampling controls. This issue seems redundant now. |
This is related to #584 in that we require per-action server logic. In this case, we need to make the server compute a custom filter expression for recipes that are a certain type of action, and append it to the other filters on the recipe already.
preference-experiment
actions will use this to filter by sampling buckets. Users will choose the number of buckets they want, and the server will compute a filter expression that:The text was updated successfully, but these errors were encountered: