Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmarks #342

Merged
merged 5 commits into from
Jul 5, 2021
Merged

Benchmarks #342

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c50ea11
code benchmarks
JustineBoulant Apr 29, 2021
5517183
notebook_benchmarks
JustineBoulant May 4, 2021
bab7c29
delete benchmarks.py
JustineBoulant May 4, 2021
a869701
modify pcts
JustineBoulant Jun 15, 2021
b7ec1ce
whitespaces
JustineBoulant Jun 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 282 additions & 0 deletions notebooks/reference_scenarios/benchmarks.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,282 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reference Scenarios"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from mplc.scenario import Scenario\n",
"from mplc.experiment import Experiment\n",
"from mplc.corruption import Randomize\n",
"import pathlib"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Parameters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"NB_EPOCH=40\n",
"NB_MINIBATCH=20\n",
"NB_GRAD_UPDATE=8"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Scenarios"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### #1 | A duo: The Hero and its Sidekick [MNIST]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Stratified split of data among 2 partners (70% / 30%)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sc1 = Scenario(\n",
" scenario_id=1,\n",
" partners_count=2,\n",
" dataset='mnist',\n",
" amounts_per_partner=[0.7, 0.3],\n",
" samples_split_option='stratified',\n",
" multi_partner_learning_approach='fedavg', #default\n",
" aggregation='data-volume', #default\n",
" contributivity_methods=[\"Independent scores\",\"Shapley values\"],\n",
" epoch_count=NB_EPOCH,\n",
" minibatch_count=NB_MINIBATCH,\n",
" gradient_updates_per_pass_count=NB_GRAD_UPDATE,\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### #2 | A different duo: Good cop / Bad cop [CIFAR10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Random split of data among 2 partners (70% / 30%). Half of the 2nd partner’s data is corrupted."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sc2 = Scenario(\n",
" scenario_id=2,\n",
" partners_count=2,\n",
" dataset='cifar10',\n",
" amounts_per_partner=[0.7, 0.3],\n",
" samples_split_option='random', #default\n",
" corruption_parameters=['not-corrupted',Randomize(proportion=0.5)],\n",
" multi_partner_learning_approach='fedavg', #default\n",
" aggregation='data-volume', #default\n",
" contributivity_methods=[\"Independent scores\",\"Shapley values\"],\n",
" epoch_count=NB_EPOCH,\n",
" minibatch_count=NB_MINIBATCH,\n",
" gradient_updates_per_pass_count=NB_GRAD_UPDATE,\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### #3 | A trio: Two cool guys and their challenger [CIFAR10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Mixed split of data among 3 partners (42%, 42%, 16%). Samples of 3 classes are randomly split, cool guys get 3 specific classes each, challenger only 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sc3 = Scenario(\n",
" scenario_id=3,\n",
" partners_count=3,\n",
" dataset='cifar10',\n",
" amounts_per_partner=[0.42, 0.42, 0.16],\n",
" samples_split_option='flexible',\n",
" samples_split_configuration=[[0.42, 0.42, 0.42, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0], \n",
" [0.42, 0.42, 0.42, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0],\n",
" [0.16, 0.16, 0.16, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0],\n",
" ],\n",
" multi_partner_learning_approach='fedavg', #default\n",
" aggregation='data-volume', #default\n",
" contributivity_methods=[\"Independent scores\",\"Shapley values\"],\n",
" epoch_count=NB_EPOCH,\n",
" minibatch_count=NB_MINIBATCH,\n",
" gradient_updates_per_pass_count=NB_GRAD_UPDATE,\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### #4 | A quintet: The basketball team with an injured player [CIFAR10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Random split of data among 5 partners (25%, 20% for 3 partners, 15%). A partner holds only corrupted samples."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sc4 = Scenario(\n",
" scenario_id=4,\n",
" partners_count=5,\n",
" dataset='cifar10',\n",
" amounts_per_partner=[0.25, 0.2, 0.2, 0.2, 0.15],\n",
" samples_split_option='random', # default\n",
" corruption_parameters=['not-corrupted', 'random', 'not-corrupted', 'not-corrupted', 'not-corrupted'],\n",
" multi_partner_learning_approach='fedavg', # default\n",
" aggregation='data-volume', # default\n",
" contributivity_methods=[\"Independent scores\"], # \"Shapley values\" too long?\n",
" epoch_count=NB_EPOCH,\n",
" minibatch_count=NB_MINIBATCH,\n",
" gradient_updates_per_pass_count=NB_GRAD_UPDATE,\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### #5 | A band: The soccer team with 2 substitutes and an injured player [MNIST]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Mixed split of data among 11 partners. 9 partners share 8 classes, the 2 other each hold 1 specific class. 1 of the 9 partners sharing classes holds corrupted samples."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sc5 = Scenario(\n",
" scenario_id=5,\n",
" partners_count=11,\n",
" dataset='mnist',\n",
" amounts_per_partner=[0.8/9.0]*9 + [0.1]*2,\n",
" samples_split_option='advanced',\n",
" samples_split_configuration=[[8, 'shared']]*9 + [[1, 'specific']]*2,\n",
" corruption_parameters=['not-corrupted']*5 + ['random'] + ['not-corrupted']*5,\n",
" multi_partner_learning_approach='fedavg', # default\n",
" aggregation='data-volume', # default\n",
" contributivity_methods=[\"Independent scores\"], # \"Shapley values\" too long?\n",
" epoch_count=NB_EPOCH,\n",
" minibatch_count=NB_MINIBATCH,\n",
" gradient_updates_per_pass_count=NB_GRAD_UPDATE,\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experiment"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"exp = Experiment(\n",
" experiment_name='benchmarks', \n",
" nb_repeats=10, \n",
" scenarios_list=[sc1,sc2,sc3,sc4,sc5],\n",
" is_save=True,\n",
" experiment_path=pathlib.Path('saved_benchmarks')\n",
" )\n",
"\n",
"exp.run()"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "1 _INTRO_MNIST.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 1
}