This repository contains code for reproducing results in the papers:
- Kamalika Chaudhuri*, Chuan Guo*, Mike Rabbat. Privacy-Aware Compression for Federated Data Analysis.
- Chuan Guo, Kamalika Chaudhuri, Pierre Stock, Mike Rabbat. Privacy-Aware Compression for Federated Learning Through Numerical Mechanism Design.
Dependencies: numpy, scipy, cvxpy, pytorch, opacus, kymatio, Handcrafted-DP, private_prediction, fastwht.
After cloning repo and installing dependencies (see requirements.txt), download submodules and run the install script to apply some patches.
git submodule update --init
python install.py
cd fastwht/python
./setup.sh
for epsilon in 1 3 5; do
python optimize_mvu.py --input_bits 3 --budget 3 --epsilon $epsilon --dp_constraint strict --method tr
python mean_estimation_single.py --epsilon $epsilon
done
For L1-sensitivity setting, first optimize the MVU mechanisms:
for epsilon in 1 2 3 4 5 6 7 8 9 10; do
python optimize_mvu.py --input_bits 9 --budget 3 --epsilon $epsilon --dp_constraint metric-l1 --method penalized
done
Then run the DME experiment and plot result:
for epsilon in 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5; do
python mean_estimation_multi.py --norm_type l1 --epsilon $epsilon --skellam_budget 16 --skellam_s 100 --mvu_input_bits 9 --mvu_budget 3
done
python plot_dme_l1.py
For L2-sensitivity setting, first optimize the MVU mechanisms and compute Renyi divergence curve for both the pure and approximate DP variants:
for epsilon in 2 4 6 8 10 12 14 16 18 20; do
python optimize_mvu.py --input_bits 5 --budget 3 --epsilon $epsilon --dp_constraint metric-l2 --method penalized
done
for epsilon in 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5; do
python optimize_mvu.py --input_bits 5 --budget 3 --epsilon $epsilon --dp_constraint metric-l1 --method penalized
python compute_renyi_div.py --input_bits 5 --budget 3 --epsilon $epsilon --dp_constraint metric-l1
done
Then run the DME experiment and plot result:
for epsilon in 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5; do
python mean_estimation_multi.py --norm_type l2 --epsilon $epsilon --skellam_budget 16 --skellam_s 15 --mvu_input_bits 5 --mvu_budget 3
done
python plot_dme_l2.py
Note: These experiments will take a few hours to run.
To run the DP-SGD training experiment, first optimize the MVU mechanism:
python optimize_mvu.py --input_bits 9 --budget 1 --epsilon <epsilon> --dp_constraint metric-l1 --method penalized
python optimize_mvu.py --input_bits 1 --budget 1 --epsilon <epsilon> --dp_constraint metric-l1 --method penalized
Then run DP-SGD training with Gaussian mechanism, signSGD, Skellam, MVU and I-MVU:
python train.py --save-model --dataset mnist --model <convnet/linear> --mechanism gaussian --quantization 0 --epochs <epochs> --scale <sigma> --lr <lr> --norm-clip <norm_clip>
python train.py --save-model --dataset mnist --model <convnet/linear> --mechanism gaussian --quantization 1 --epochs <epochs> --scale <sigma> --lr <lr> --norm-clip <norm_clip>
python train.py --save-model --dataset mnist --model <convnet/linear> --mechanism skellam --quantization 16 --epochs <epochs> --scale <sigma> --lr <lr> --norm-clip <norm_clip>
python train.py --save-model --dataset mnist --model <convnet/linear> --mechanism mvu --input-bits 9 --quantization 1 --beta 1 --epochs <epochs> --epsilon <epsilon> --lr <lr> --norm-clip <norm_clip>
python train.py --save-model --dataset mnist --model <convnet/linear> --mechanism mvu_l2 --input-bits 1 --quantization 1 --beta 1 --epochs <epochs> --epsilon <epsilon> --lr <lr> --norm-clip <norm_clip>
To train on CIFAR-10, simply replace --dataset mnist by --dataset cifar10. See appendix in our paper for the full grid of hyperparameter values.
The majority of Privacy-Aware Compression is licensed under CC-BY-NC, however portions of the project are available under separate license terms: CVXPY and Opacus are licensed under the Apache 2.0 license; Kymatio is licensed under the BSD license; and Handcrafted-DP is licensed under the MIT license.