Skip to content

Commit

Permalink
Update benchmark.html
Browse files Browse the repository at this point in the history
  • Loading branch information
@TsingZ0
TsingZ0 committed Dec 12, 2024
1 parent b991a6f commit 10a672b
Showing 1 changed file with 142 additions and 1 deletion.
143 changes: 142 additions & 1 deletion docs/benchmark.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,12 @@
text-decoration: none;
color: #6DA945;
}
table {
margin-left: auto;
margin-right: auto;
text-align: center;
font-size: 0.8em;
}
@media (max-width: 768px) {
.container {
flex-direction: column;
Expand Down Expand Up @@ -179,7 +185,7 @@ <h2>Benchmark & Evaluation Platform</h2>
<p>To integrate all the algorithms, datasets, and scenarios, we standardize the experimental settings and create a <strong>unified evaluation platform</strong> for a fair comparison of these algorithms. Here, we present the benchmark results of <strong>20 algorithms</strong> across two widely-used <em><strong>label skew</strong></em> scenarios. This is just one example. You can obtain different results by resetting all the configurations in <code>main.py</code> in our PFLlib.</p>
<h3>Leaderboard</h3>
<h4 style="width:100%; text-align:center;">The test accuracy (%) on the CV and NLP tasks in <em>label skew</em> settings.</h4>
<table border="1" cellpadding="5" cellspacing="0" style="width:100%; text-align:center; font-size: 0.8em;">
<table border="1" cellpadding="5" cellspacing="0">
<thead>
<tr>
<th rowspan="2"><strong>Settings</strong></th>
Expand Down Expand Up @@ -420,6 +426,141 @@ <h4 style="width:100%; text-align:center;">The test accuracy (%) on the CV and N
</tr>
</tbody>
</table>

<h4 style="width:100%; text-align:center;">The time cost (in minutes) on Tiny-ImageNet using ResNet-18 in the practical <em>label skew</em> setting.</h4>
<table border="1" cellpadding="5" cellspacing="0">
<thead>
<tr>
<th>Items</th>
<th>Total time</th>
<th>Iterations</th>
<th>Average time</th>
</tr>
</thead>
<tbody>
<tr>
<td><b>FedAvg</b></td>
<td>365</td>
<td>230</td>
<td>1.59</td>
</tr>
<tr>
<td><b>FedProx</b></td>
<td>325</td>
<td>163</td>
<td>1.99</td>
</tr>
<tr>
<td><b>FedGen</b></td>
<td>259</td>
<td>50</td>
<td>5.17</td>
</tr>
<tr>
<td><b>Per-FedAvg</b></td>
<td>121</td>
<td>34</td>
<td>3.56</td>
</tr>
<tr>
<td><b>pFedMe</b></td>
<td>1157</td>
<td>113</td>
<td>10.24</td>
</tr>
<tr>
<td><b>Ditto</b></td>
<td>318</td>
<td>27</td>
<td>11.78</td>
</tr>
<tr>
<td><b>APFL</b></td>
<td>156</td>
<td>57</td>
<td>2.74</td>
</tr>
<tr>
<td><b>FedFomo</b></td>
<td>193</td>
<td>71</td>
<td>2.72</td>
</tr>
<tr>
<td><b>FedAMP</b></td>
<td>92</td>
<td>60</td>
<td>1.53</td>
</tr>
<tr>
<td><b>APPLE</b></td>
<td>132</td>
<td>45</td>
<td>2.93</td>
</tr>
<tr>
<td><b>FedALA</b></td>
<td>123</td>
<td>63</td>
<td>1.93</td>
</tr>
<tr>
<td><b>FedPer</b></td>
<td>83</td>
<td>43</td>
<td>1.92</td>
</tr>
<tr>
<td><b>FedRep</b></td>
<td>471</td>
<td>115</td>
<td>4.09</td>
</tr>
<tr>
<td><b>FedRoD</b></td>
<td>87</td>
<td>50</td>
<td>1.74</td>
</tr>
<tr>
<td><b>FedBABU</b></td>
<td>811</td>
<td>513</td>
<td>1.58</td>
</tr>
<tr>
<td><b>FedCP</b></td>
<td>204</td>
<td>74</td>
<td>2.75</td>
</tr>
<tr>
<td><b>GPFL</b></td>
<td>171</td>
<td>75</td>
<td>2.28</td>
</tr>
<tr>
<td><b>FedDBE</b></td>
<td>171</td>
<td>107</td>
<td>1.60</td>
</tr>
<tr>
<td><b>FedDistill</b></td>
<td>45</td>
<td>16</td>
<td>2.78</td>
</tr>
<tr>
<td><b>FedProto</b></td>
<td>138</td>
<td>25</td>
<td>5.52</td>
</tr>
</tbody>
</table>

<h3>Experimental Setup</h3>
<p>We set up the experiments following our pFL algorithm <a href="https://arxiv.org/pdf/2308.10279v3.pdf"><strong>GPFL</strong></a>, as it provides comprehensive evaluations. Here are the details:</p>
<h4>Datasets and Models</h4>
Expand Down

0 comments on commit 10a672b

Please sign in to comment.