-
Notifications
You must be signed in to change notification settings - Fork 86
Provide more details about the experiments? #1
Comments
Hi @ddayzzz, glad to hear your interest! To answer your questions:
Please feel free to report any issues and to create PRs. We will be happy to help and work with you. |
Hi @hwang595. I'm looking into your repo/paper and I saw that you guys did report mnist dataset using lenet on the paper that was published at ICLR. I'm trying to reproduce by change some lines at the code but some tricky problems are happening. How hard is it to reproduce the same result showed at matched averaging? |
Hi @joaolcaas, thanks a lot for your interests in our work! Can you please provide more details on the issue you encountered for the MNIST experiments? Based on our tests, PFNM already provides a good accuracy on MNIST in both homogeneous and heterogeneous settings. Thus, this repo focuses more on the CIFAR-10 and Shakespeare experiments. But I'm happy to help with resolving the issues in the MNIST experiment. Please also feel free to take a look at the PFNM github repo. Thanks! |
Thanks, @hwang595. As you requested, more details below. First I tried to run the experiment using this command to check how the algorithm behaves: I saw that block_patching does not support lenet architecture, so I add That was as far as I could get. |
@hwang595 Hi Hongyi, I met the same issue. I think in the released code, the shape estimator is not defined for LeNet. |
Hi @joaolcaas @wildphoton, thanks for providing the detailed error messages. I can replicate your issues. As I mentioned since this repository focuses more on the CIFAR-10 and Shakespeare experiments, when I made the first commit, I didn't realize the MNIST+LeNet part of code is not up-to-date. Sorry about that! I made the fixes, it should run without problem now. But please keep in mind that we still don't support the multi-round version of MNIST+LeNet since one round of FedMA already gives >97% accuracy and matches the accuracy we can expect by the ensemble method. But I'm happy to make further improvement to support multi-round FedMA if you are interested. I'm happy to help more on your experiments and provide more detailed information! |
That was quicker than I expected, huge thanks @hwang595. I'm testing for now, but you are saying that if I use |
Hi @joaolcaas, use |
@hwang595 yeah, it would be really amazing if you can do that! |
Hi @hwang595, I understand that the experiments focus on well-known models and small datasets. What if we try large datasets and models with different architectures? I failed to make modifications like changing the model to, e.g., DenseNet and MobileNet, and adding another dataset with input shape greater than 32x32. I've been trying the following scenario: Use
Obs.: This dataset has only 10 examples per class in a total of 5 classes. This setting is used just to test the training pipeline. But the training process takes too long and also it stucks in that part bellow. Was that expected? Maybe I'm doing something wrong.. I've been concerned about the relationship of input dimensions to the general complexity that the Matching Average adds during communications. So, my questions are:
|
@hwang595 Thanks for your quick response! I tried the updated code, but it gave me the following error by running
|
yep, the same error here, unfortunately |
Hi @wildphoton @joaolcaas, thanks a lot for trying it out! Yes, that error is as expected since you already entered the But even before |
So --comm_round=0 actually means round 1? If I understand it correctly, PFNM is basically one round FedMA? |
Good job! I am very interested in this work and I tried run the experiments mentioned in the paper. My questions are:
Thank you.
The text was updated successfully, but these errors were encountered: