Add DepthFL baseline

[Peterpan828]

Related issues/PRs

Implementation for #2038

Checklist

• Implement proposed change
• Write tests
• Update documentation
• Update changelog
• Make CI checks pass
• Ping maintainers on Slack (channel #contributions)

Any other comments?

[jafermarq]

Hi @Peterpan828, I took an initial pass through your DepthFL code following the instructions in your readme. I was able to run it so that's great! I have added some small comments below, mostly making suggestions to the pyproject.toml, the readme and one question regarding our previous discussion for stateful clients (I believe none of the prev_grad is needed in the strategies?)

[jafermarq]

Hi @Peterpan828 ,

Thanks for incorporating the changes in the previous pass i did. I have left a few more (very minor).

Also, could you run the formatting script? it will detect quite a few issues. You can do so by: (1) sourcing your environment from your baseline directory, (2) cd .., (3) ./dev/format-baseline.sh depthfl

After that you can run the automatic tests with ./dev/test-baseline.sh depthfl

[jafermarq]

Hi @Peterpan828,

Thanks for applying those suggestions. I have a few more. Mostly cosmetic.

Let me know if you need some advice on how to deal with the formatting errors/warning that appear when you run the formatting script. They are generally easy to address.

[jafermarq]

Hi @Peterpan828 , hope things are going well. Please let me know if there is anything you'd like discuss about your baseline. From our last discussion it seems everything is in place result-wise. But i saw you added some extra commits regarding heterofl. Happy to help if some parts need a second pair of eyes or if you are struggling making the ./dev/test-baseline.sh script pass. Just let me know~

[Peterpan828]

Hi @jafermarq,

While modifying the code to pass the test, I figured out that the heterofl-related code was not operating properly, so I modified it. As of now, all code seems to be working as intended and the script test has also passed.

[jafermarq]

Hi @jafermarq,

While modifying the code to pass the test, I figured out that the heterofl-related code was not operating properly, so I modified it. As of now, all code seems to be working as intended and the script test has also passed.

By "heterofl-related code was not operating properly" do you mean there was a bug or it was just an issue with tests? If the former, probably some experiments need to be re-run?

You'll notice I pushed some changes to your branch (just some formatting and added a bit of text).

Could you do a final pass through the code ? I think now it is the time to ensure everything is documented.

[Peterpan828]

While modifying the code to pass the test, a problem arose, so I fixed it, and actually re-run most of experiments to confirm that there were no more problems.

[jafermarq]

Hi @Peterpan828 ,

Thanks for clarifying the above. Just one final suggestion to edit how the Expected Results section is presented. I think because you have a lot of results (this is great!) there are many commands included in the readme and some are applicable for more than one table. It would be best if we re-arrange the Expected Results section adding a sub-section for each table showing first the command to generate it entirely, then present the table. Then repeat this for all three tables.

What do you think?

[Peterpan828]

I modified the expected results section as suggested.

[jafermarq]

Hi @Peterpan828, thanks for making those changes.

I have started running the experiment on my side as part of the reviewing process and managed to reproduce Table2. Great!

I have just some questions on the commands for Table 3,4. Please take a look. I'm not entirely sure if those are all people will need to generate those tables.

You'll notice i have also added a commit including reflecting your baseline in the changelog of the repo.

[jafermarq]

@Peterpan828 , this looks great! 💯 We'll merge your baseline shortly