Add niid-Bench baseline

[aashishkolluri]

Issue

Description

Originally, the task has been to reproduce the results from SCAFFOLD paper (Karimireddy el al. 2019). The results in that paper are difficult to reproduce due to missing details on the experimental setup. However, SCAFFOLD has been benchmarked in another work (https://arxiv.org/pdf/2102.02079.pdf) along with other FL protocols FedAvg, FedProx, and FedNova. Therefore, in the best interests of demonstrating SCAFFOLD on Flower, I have implemented the other FL protocols too and reproduced results from this work instead.

Related issues/PRs

Proposal

Explanation

Checklist

• Implement proposed change
• Write tests
• Update documentation
• Update changelog
• Make CI checks pass
• Ping maintainers on Slack (channel #contributions)

Any other comments?

[jafermarq]

Hi @aashishkolluri ,

Thanks for including the changes. I have highlighted a few minor ones. I also run all the experiments. It looks good.

But it seems the ./dev/test-baseline.sh ... command still shows quite a few errors. For those in the form too-many- you can disable them by placing a line like #pylint: disable=too-many-<...> next or above the line triggering the error. But please don't use the disable option for other errors. If there is some error you are having a hard time to fix, let me know and I can probably help.

Once that test passes, we can merge the baseline.

[jafermarq]

Hi @aashishkolluri ,

Thanks for applying those many changes since my last review. Your code and results look good so i'm happy to merge this baseline. But before we do this just one small thing: part of the tests check for the existence of client.py, server.py, strategy.py (which you didn't have). I added them as pretty much empty files. But i used that opportunity to add your custom strategies into strategy.py and I made the necessary changes elsewhere in the code (including the configs) so everything should work as before.

➡️ Could you please double check the changes i did in my last commit?

Also, if there is anything you'd like to add or modify to your readme, now it's the time.

Let me know when you have a couple of minutes to double check the above. Maybe running a couple of experiments to see everything remains the same won't hurt.

All test in the GitHub CI pass

[aashishkolluri]

Hi @aashishkolluri ,

Thanks for applying those many changes since my last review. Your code and results look good so i'm happy to merge this baseline. But before we do this just one small thing: part of the tests check for the existence of client.py, server.py, strategy.py (which you didn't have). I added them as pretty much empty files. But i used that opportunity to add your custom strategies into strategy.py and I made the necessary changes elsewhere in the code (including the configs) so everything should work as before.

➡️ Could you please double check the changes i did in my last commit?

Also, if there is anything you'd like to add or modify to your readme, now it's the time.

Let me know when you have a couple of minutes to double check the above. Maybe running a couple of experiments to see everything remains the same won't hurt.

All test in the GitHub CI pass

Hi @jafermarq ,

I have checked your changes in the last commit and everything seems fine to me. I have also run a few commands from the Readme again to check the results and they are as expected.

[jafermarq]

@aashishkolluri , thanks for confirming. This baseline is ready to merge then! amazing work 🚀