Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add A/A testing #6793

Closed
luabud opened this issue Jul 31, 2019 · 3 comments
Closed

Add A/A testing #6793

luabud opened this issue Jul 31, 2019 · 3 comments
Assignees
Labels
feature-request Request for new features or functionality

Comments

@luabud
Copy link
Member

luabud commented Jul 31, 2019

A/A testing is used to validate metrics for A/B testing and/or the sampling logic itself.
Considering we have X experiments with Y% users in each group, we should take the remaining Z=100%-(X*Y)% (i.e. people who are not in any of the experiment groups) and sample them into control and treatment groups, with Y% users in each one.

@luabud luabud added feature-request Request for new features or functionality triage-needed Needs assignment to the proper sub-team labels Jul 31, 2019
@ghost ghost removed the triage-needed Needs assignment to the proper sub-team label Jul 31, 2019
@luabud
Copy link
Member Author

luabud commented Aug 13, 2019

Just to be clear, we want to separate users who are not in any A/B experiments between experiment and control groups so we can see if our sampling method is being fair. We want to make sure that there's no significant change in the behavior in each group

/cc @qubitron

@DonJayamanne
Copy link

Size - 10%

@luabud
Copy link
Member Author

luabud commented Oct 15, 2019

Validated.

@luabud luabud closed this as completed Oct 15, 2019
@ghost ghost removed the needs PR label Oct 15, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Oct 22, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature-request Request for new features or functionality
Projects
None yet
Development

No branches or pull requests

6 participants