Merge pull request #360 from LabeliaLabs/fix_config_file

Fix errors in config files required to launch the script via the main.py approach

README.md

@@ -150,7 +150,7 @@ Alternatively, you can also use the `main.py` provided in the repository togethe
 experiment_name: my_custom_experiment
 n_repeats: 5
 scenario_params_list:
-  - dataset_name:
+  - dataset:
     'mnist':
     - 'random_initialization'
     'cifar10':
@@ -166,10 +166,10 @@ scenario_params_list:
       - ['advanced', [[7, 'shared'], [6, 'shared'], [2, 'specific']]]
     multi_partner_learning_approach:
       - 'fedavg'
-    aggregation_weighting:
-      - 'data_volume'
+    aggregation:
+      - 'data-volume'
       - 'uniform'
-    methods:
+    contributivity_methods:
       - ["Shapley values", "Independent scores", "TMCS"]
     epoch_count:
       - 20
@@ -179,7 +179,7 @@ scenario_params_list:
       - 8
 ```
 
-Under `scenario_params_list`, enter a list of sets of scenario(s). Each set starts with `- dataset_name:` and must have only one `partners_count` value. The length of `amount_per_partners`, `corrupted_datasets` (and `samples_split_option` when the advanced definition is used) must match the `partner_counts` value. If for a given parameter multiple values are specified, e.g. like for `agregation_weighting` in the example scenario above, all possible combinations of parameters will be assembled as separate scenarios and run.
+Under `scenario_params_list`, enter a list of sets of scenario(s). Each set starts with `- dataset:` and must have only one `partners_count` value. The length of `amount_per_partners`, `corrupted_datasets` (and `samples_split_option` when the advanced definition is used) must match the `partner_counts` value. If for a given parameter multiple values are specified, e.g. like for `aggregation` in the example scenario above, all possible combinations of parameters will be assembled as separate scenarios and run.
 
 2. Then execute `main.py -f config.yml`. Add the `-v` argument if you want a more verbose output.
 

config.yml

@@ -1,23 +1,23 @@
 experiment_name: my_example
 n_repeats: 5
 scenario_params_list:
-  - dataset:
-      - 'mnist'
-      - 'cifar10'
-    partners_count:
-      - 3
-    amounts_per_partner:
-      - [ 0.4, 0.3, 0.3 ]
-    samples_split_option:
-      - [ 'basic', 'random' ]
-      - [ 'basic', 'stratified' ]
-      - [ 'advanced', [ [ 4, 'shared' ], [ 6, 'shared' ], [ 4, 'specific' ] ] ]
+ - dataset:
+     - 'mnist'
+     - 'cifar10'
+   partners_count:
+     - 3
+   amounts_per_partner:
+     - [ 0.4, 0.3, 0.3 ]
+   samples_split_option:
+     - 'random' 
+     - 'stratified' 
+     - [ 'advanced', [ [ 4, 'shared' ], [ 6, 'shared' ], [ 4, 'specific' ] ] ]
    multi_partner_learning_approach:
      - 'fedavg'
      - 'seq-pure'
      - 'seq-with-final-agg'
      - 'seqavg'
-   aggregation_weighting:
+   aggregation:
      - 'data-volume'
      - 'uniform'
    epoch_count: 
@@ -30,22 +30,22 @@ scenario_params_list:
      - 8
    dataset_proportion:
      - 1
- - dataset_name:
+ - dataset:
      - 'mnist'
    partners_count:
      - 2
    amounts_per_partner: 
      - [0.5, 0.5]
    samples_split_option:
-     - ['basic', 'stratified']
+     - 'stratified'
    multi_partner_learning_approach:
      - 'fedavg'
-   aggregation_weighting: 
-     - 'data_volume' 
+   aggregation: 
+     - 'data-volume' 
      - 'uniform'
    epoch_count: 
      - 50
-   methods:
+   contributivity_methods:
      - ["Shapley values", "SMCS", "IS_lin_S", "IS_reg_S"]
    minibatch_count: 
      - 10

config_quick_debug.yml

@@ -12,10 +12,9 @@ scenario_params_list:
      - 'random'
      - 'stratified'
      - [ 'advanced', [ [ 4, 'shared' ], [ 6, 'shared' ], [ 4, 'specific' ] ] ]
-
    multi_partner_learning_approach:
      - 'fedavg'
-   aggregation_weighting:
+   aggregation:
      - 'uniform'
    contributivity_methods:
      - [ "Federated SBS constant", "Federated SBS linear", "Federated SBS quadratic", "Shapley values", "Independent scores", "TMCS" ]

mplc/doc/documentation.md

@@ -351,14 +351,14 @@ There are several parameters influencing how the collaborative and distributed l
 
   Example: `multi_partner_learning_approach='seqavg'`
 
-- `aggregation_weighting`: `'uniform'` (default), `'data_volume'` or `'local_score'`  
+- `aggregation`: `'uniform'` (default), `'data-volume'` or `'local_score'`  
   After a training iteration over a given mini-batch, how individual models of each partner are aggregated:
 
   - `'uniform'`: simple average (non-weighted)
-  - `'data_volume'`: average weighted with per the amounts of data of partners (number of data samples)
+  - `'data-volume'`: average weighted with per the amounts of data of partners (number of data samples)
   - `'local_score'`: average weighted with the performance (on a central validation set) of the individual models
 
-  Example: `aggregation_weighting='data_volume'`
+  Example: `aggregation='data-volume'`
 
 - `epoch_count`: `int` (default: `40`)  
   Number of epochs, i.e. of passes over the entire datasets. Superseded when `is_early_stopping` is set to `true`.  
@@ -380,7 +380,7 @@ There are several parameters influencing how the collaborative and distributed l
 
 ### Configuration of contributivity measurement methods to be tested
 
-- `methods`:  
+- `contributivity_methods`:  
   A declarative list `[]` of the contributivity measurement methods to be executed.
   All methods available are:
 

mplc/utils.py

@@ -88,7 +88,7 @@ def get_scenario_params_list(config):
             scenario = dict(zip(params_name, el))
             if scenario['partners_count'] != len(scenario['amounts_per_partner']):
                 raise Exception("Length of amounts_per_partner does not match number of partners.")
-            if scenario['samples_split_option'][0] == 'advanced':
+            if scenario['samples_split_option'][0] in ['advanced', 'flexible']:
                 if scenario['partners_count'] != len(scenario['samples_split_option'][1]):
                     raise Exception("Length of samples_split_option does not match number of partners.")
                 else: