Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bucket sampling for preference experiments #586

Closed
Osmose opened this issue Mar 7, 2017 · 3 comments
Closed

Bucket sampling for preference experiments #586

Osmose opened this issue Mar 7, 2017 · 3 comments

Comments

@Osmose
Copy link
Contributor

Osmose commented Mar 7, 2017

This is related to #584 in that we require per-action server logic. In this case, we need to make the server compute a custom filter expression for recipes that are a certain type of action, and append it to the other filters on the recipe already.

preference-experiment actions will use this to filter by sampling buckets. Users will choose the number of buckets they want, and the server will compute a filter expression that:

  • Splits the userbase into 100,000 buckets, and reserve even chunks of that space for each branch.
  • Uses the user's unique ID and the experiment slug for the hash
  • At each of those start points, matches users for X buckets, where X is defined in the recipe's arguments per-branch.
@Osmose Osmose changed the title Per-action computer filter expressions Per-action computed filter expressions Mar 7, 2017
@Osmose Osmose self-assigned this Mar 9, 2017
@Osmose Osmose removed their assignment Mar 16, 2017
@Osmose Osmose self-assigned this Mar 27, 2017
@Osmose Osmose added this to the Sprint 8 - March 20th to April 1st milestone Mar 27, 2017
@Osmose
Copy link
Contributor Author

Osmose commented Mar 29, 2017

Our approach for this has changed since the bug was initially filed, after talking with Sunnah and Gregg. Now, preference experiments specify a count of how many buckets out of 100,000 that they want to sample, and branches specify the ratio of matched users that they should match. Thus, the filtering for this simplifies to doing a normal, independent bucket sample of users based on that count.

We can probably accomplish this within the preference experiment action too, which removes the need to refactor the backend to support this for all actions. Yay!

@Osmose Osmose changed the title Per-action computed filter expressions Bucket sampling for preference experiments Mar 29, 2017
Osmose pushed a commit to Osmose/normandy that referenced this issue Mar 29, 2017
@Osmose
Copy link
Contributor Author

Osmose commented Mar 29, 2017

Per the discussion in #650, this is being moved to a non-blocker since we can write a filter expression that accomplishes this easily and has the same effect as the PR I put up.

We still should address what UI we want to making this kind of filtering accessible to the audience we eventually expect will be using Normandy to launch preference experiments, but for an MVP it's not necessary.

@Osmose Osmose removed this from the Sprint 8 - March 20th to April 1st milestone Mar 30, 2017
@Osmose Osmose removed their assignment Apr 12, 2017
@Osmose
Copy link
Contributor Author

Osmose commented Jun 16, 2017

I think the solution to this lies in #316 and related efforts to simplify sampling controls. This issue seems redundant now.

@Osmose Osmose closed this as completed Jun 16, 2017
@mythmon mythmon added this to the preference-experiments milestone Jan 30, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants