The performance on SYN on FPL

[AtsuMiyai]

Thank you for sharing the code for this very interesting work.

I tried to reproduce your FPL.
It generally worked, but I could not reproduce only the SYN dataset for Digits. The results in Table 2 of the paper report a score of 61.20, but in my multiple replication experiments I only got about 44.1. (43.8 in FedAVG.)

I have a few questions.

Is the data set for SYN this url, right?
Also, when I increased the number of data in the dataset from 1% to 1.5%, the results were comparable to the paper's score. Is there any other solution besides increasing the data?

We apologize for any inconvenience caused. Thank you for your cooperation.

[WenkeHuang]

Do the counterparts work consistently?
I just check the results:
30 FedProx para1 98.11 90.24 77.01 56.66 80.505
44 FPL InfoT 0.02 98.31 92.71 80.27 61.2 83.1225

[AtsuMiyai]

Thanks for quick response!

My results on office+caltech seem correct.
My results on Digits seems different from original ones:
FPL: 96.74, 92.97, 87.2, 44.1

For datasets, I download SYN via this url and I use other datasets (e.g., MNIST) automatically downloaded via your code.

I change the argument in main.py as follows:
--dataset fl_digits --rand_dataset=False --parti_num 20
The clients are
Counter({'usps': 7, 'svhn': 6, 'syn': 4, 'mnist': 3})

[WenkeHuang]

The SVHN dataset link is： http://ufldl.stanford.edu/housenumbers/
Format 2: Cropped Digits
I will add the datasets in the onedrive.

[AtsuMiyai]

The SVHN dataset link is： http://ufldl.stanford.edu/housenumbers/

Yes. I use the same ones named train_32x32.mat and test_32x32.mat.

I will add the datasets in the onedrive.

Thanks. I will try with these datasets!

[WenkeHuang]

Thanks for your patience.
https://drive.google.com/drive/folders/1SSv9dqQPBGyHS3rSwoFKmpBIeF4GX-i6?usp=sharing

[WenkeHuang]

--rand_dataset=True, We made a typing error in the uploading code. The code is based on the seed = 0 and {'mnist': 6, 'syn': 7, 'usps': 4, 'svhn': 3}.

[AtsuMiyai]

Thanks for the modification. But, I still have difficulty reproducing the results on digits...

I may be doing something wrong, so I'll try some times later.

[YutoShibata07]

@AtsuMiyai
Could you tell me how you trained your model?
I'm now trying to reproduce the results like you, but it seems a local update part is not working currently.

At utils/training.py
def loc_update(self, priloader_list): pass

Thanks in advance.

[zekunshi]

@YutoShibata07
You may use the fuction in sub-class(like fedavg.py) but not superclass.

[ennnjoy123]

--rand_dataset=True, We made a typing error in the uploading code. The code is based on the seed = 0 and {'mnist': 6, 'syn': 7, 'usps': 4, 'svhn': 3}.

Thank you for your code, So it means that the results for the Digits task reported in your paper are based on: MNIST: 6, USPS: 7, SVHN: 4 and SYN: 3 setting, Not MNIST: 3, USPS: 7, SVHN: 6 and SYN: 4?

thank you