# MABFDR
Code for the MAB-FDR framework introduced in "A framework for Multi-A(rmed)/B(andit) Testing with Online FDR Control", NIPS 2017  arXiv preprint available at https://arxiv.org/abs/1706.05378

--------------- For reconstructing the plots ------------------------

0. pip install sortedcontainers
If you use a remote, uncomment ‘agg’

1. Run experiments
For Gaussian:
python main_onlocal.py --dist-style 1
For Bernoulli:
python main_onlocal.py --dist-style 0 
For Bernoulli New Yorker:
python main_onlocal.py --ny-data 1

2. Generate power plots
For Gaussians:
python plot_main.py --dist-style 1
Plots in the paper are found in plots/BDRvsTT_D1.... and plots/SPSvsNA_D1...

For Bernoulli:
python plot_main.py --dist-style 0
Plots in the paper are found in plots/BDRvsTT_D0_MS2.... and plots/SPSvsNA_D0_MS2...

For Bernoulli from New Yorker:
python plot_main.py --dist-style 0 --ny-data 1
Plots in the paper are found in plots/BDRvsTT_D0_MS0.... and plots/SPSvsNA_D0_MS0...

3. Generate FDP and mFDR plots for Gaussians
python main_onlocal.py --dist-style 1 --power-plot 0
python plot_main.py --dist-style 1 --power-plot 0

Plots in the paper are found in plots/FDPvsHYP... and plots/mFDRvsPi...


----------------- Brief overview over code structure -------------------- 

********** High level scripts *********************************
main_onlocal.py:	runs a for loop which calls single experiments using run_single() in exp_new.py and saves 
plot_main.py:		produces and saves power plots

********** Functional scripts that are used above *************
generate_mu.py:	 	generates experimental settings (means and which hypotheses are true nulls and alternatives)
parse_mu.py:	 	loads mu vectors from file or calls generate_mu.py if the settings do not exist yet
parse_new_yorker.py:	generates experiment for New Yorker experiment with realistic distribution of means

exp_new.py:		Runs a sequence of experiments using online FDR
rowexp_new.py:		class for one experiment using MAB
plot_results.py:	Produces power and sample plots

exp_new_punif.py:	Runs a sequence of experiments using online FDR, drawing uniform p-value for null values
plot_results_punif.py:	Produces FDP and mFDR plots

Online FDR procedures can be found in the folder onlineFDR_proc

********** Parameters which are used by all of the modules **********
dist_style:
Distribution type
0: Bernoulli
1: Gaussian

mu_style:
Different ways you can choose your means to be distributed
1: one peak, rest same low (with sigma noise)
2: one high, uniform down
3: some same around highest (epsilon), rest same low.
4: some same around highest, rest uniform down

hyp_style:
Different ways you can simulate how true nulls and alternatives are distributed
0: uniform across num_hyp
1: many alt at beginning - lin prob
2: many alt at end - lin prob
3: many alt at beginning - step down
4: many alt at beginning - step up

FDRrange (online FDR procedure)
0: LORD++ ( we use the ++ version, described in the paper "Online control of the false discovery rate with decaying memory" available at https://arxiv.org/abs/1710.00499, for the LORD procedure described in our paper )
1: LORD
3: no FDR, plain alpha
4: alpha invest (Foster & Stine '07 )
5: Bonferroni

proc_list (MAB combined with different online multiple testing procedures)
0: MAB-LORD (corrected with LORD++)
3: MAB-IND (uncorrected)
5: MAB-Bonf 

alg_list ( sampling procedure per experiment )
0: LUCB best-arm MAB algorithm ( Kalyanakrishnan et al. '12 )
1: Uniform sampling

------------------------- Some notes ---------------------------
- If you run the code on your local computer, you may comment out the lines in import section of the plotting related code containing 'agg'
- Although AlphaInvest was implemented, it is not plotted in the graphs
- Epsilon-modified LUCB is not currently implemented
- The number of runs that are averaged over are aggregated. Thus, if you run the exp multiple times, you'll average over more samples
- For parallelization (of the e.g. New Yorker experiments) you may write a wrapper and use ipyparallel