Skip to content

Commit 34d38bc

Browse files
celrmcelrm
committed
icml25
1 parent bf39664 commit 34d38bc

File tree

10 files changed

+140
-36
lines changed

10 files changed

+140
-36
lines changed

‎_bibliography/references.bib

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
---
22
---
3+
@inproceedings{jacobs2025mirror,
4+
title={Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?},
5+
author={Tom Jacobs and Chao Zhou and Rebekka Burkholz},
6+
booktitle={Forty-second International Conference on Machine Learning},
7+
year={2025},
8+
url={https://arxiv.org/abs/2504.12883},
9+
pdf={https://arxiv.org/pdf/2504.12883},
10+
img={mirror-mirror.jpg},
11+
abstract={Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent overfitting. While both concepts have been studied separately, in practice, they often act in tandem. Understanding their interplay is key to controlling the shape and strength of implicit bias, as it can be modified by explicit regularization. To this end, we incorporate explicit regularization into the mirror flow framework and analyze its lasting effects on the geometry of the training dynamics, covering three distinct effects: positional bias, type of bias, and range shrinking. Our analytical approach encompasses a broad class of problems, including sparse coding, matrix sensing, single-layer attention, and LoRA, for which we demonstrate the utility of our insights. To exploit the lasting effect of regularization and highlight the potential benefit of dynamic weight decay schedules, we propose to switch off weight decay during training, which can improve generalization, as we demonstrate in experiments.},
12+
}
313

414
@inproceedings{
515
jacobs2025mask,

‎_data/alumni_members.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,16 @@
1515
end_date: Jul 24
1616
url: https://nelaturuharsha.github.io/
1717

18+
- role: Research engineers
19+
members:
20+
- name: Nikita (Nik) Vedeneev
21+
last_name: Vedeneev
22+
photo: c01mive.jpg
23+
start_date: Dec 24
24+
end_date: May 25
25+
email: mikita.vedzeneyeu@cispa.de
26+
description: "I am interesting in making modern AI models efficient. In particular, I work on discovering and exploiting structure in Neural Networks (sparsity, low-dimensional representations and similar) for efficient training, fine-tuning and inference. I am a former full-time core developer for [PyTorch](https://github.com/pytorch/pytorch) and [Lightning Thunder](https://github.com/Lightning-AI/lightning-thunder). Check my [GitHub](https://github.com/nikitaved) to see what I work on now."
27+
1828
- role: Visiting students
1929
members:
2030
- name: Otto Piramuthu

‎_data/news.yml

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,20 @@
1+
- date: 2. June 2025
2+
headline: "Rebekka and Celia are presenting at the International Network Science Conference ([NetSci](https://netsci2025.github.io/)) in Maastricht."
3+
4+
- date: 1. June 2025
5+
headline: Welcome Baraah!
6+
7+
- date: 1. May 2025
8+
headline: "Our paper on [implicit bias](https://arxiv.org/pdf/2504.12883) has been accepted at ICML 2025."
9+
110
- date: 24. March 2025
211
headline: "Rebekka is at [CPAL](https://cpal.cc/spotlight_track/) in Stanford presenting three of our [papers](/publications) as recent spotlights."
312

413
- date: 22. January 2025
514
headline: "Two papers
615
[(1)](https://openreview.net/forum?id=g6v09VxgFw)
716
[(2)](https://openreview.net/forum?id=U47ymTS3ut)
8-
have been accepted at ICLR 2025 (see [publications](/publications))."
17+
have been accepted at ICLR 2025."
918

1019
- date: 1. December 2024
1120
headline: "Welcome to Gowtham and Nik!"
@@ -21,7 +30,7 @@
2130
headline: "Welcome to Chao, Rahul, and Dong!"
2231

2332
- date: 14. June 2024
24-
headline: "Celia, Advait and Adarsh are presenting at the Helmholtz AI Conference: AI for Science ([HAICON](https://eventclass.it/haic2024/scientific/external-program/session?s=S-05a))."
33+
headline: "Celia, Advait and Adarsh are presenting at the Helmholtz AI Conference: AI for Science ([HAICON](https://eventclass.it/haic2024/scientific/external-program/session?s=S-05a)) in Düsseldorf."
2534

2635
- date: 1. May 2024
2736
headline: "Our paper on [improving GATs](https://openreview.net/forum?id=Sjv5RcqfuH) has been accepted at ICML 2024."

‎_data/team_members.yml

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -74,12 +74,9 @@
7474
url: https://cispa.de/en/people/c01dosu
7575
description: "My current research focuses on theoretically elucidating the superior performance of Mixture of Experts models, with an emphasis on their generalization performance, sample complexity, training dynamics, and robustness to adversarial noises."
7676

77-
- role: Research engineers
78-
members:
79-
- name: Nikita (Nik) Vedeneev
80-
last_name: Vedeneev
81-
photo: c01mive.jpg
82-
start_date: Dec 2024
83-
email: mikita.vedzeneyeu@cispa.de
84-
url: https://cispa.de/en/people/c01mive
85-
description: "I am interesting in making modern AI models efficient. In particular, I work on discovering and exploiting structure in Neural Networks (sparsity, low-dimensional representations and similar) for efficient training, fine-tuning and inference. I am a former full-time core developer for [PyTorch](https://github.com/pytorch/pytorch) and [Lightning Thunder](https://github.com/Lightning-AI/lightning-thunder). Check my [GitHub](https://github.com/nikitaved) to see what I work on now."
77+
- name: Baraah Sidahmed
78+
last_name: Sidahmed
79+
photo:
80+
start_date: Jun 2025
81+
email: baraah.adil@cispa.de
82+
url: https://cispa.de/en/people/c01baad

‎_site/images/publications/mirror-mirror.jpg

32.1 KB
Loading

‎_site/index.html

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -120,28 +120,28 @@ <h1 id="relational-ml-lab">Relational ML Lab</h1>
120120
<h3>News</h3>
121121
<div class="well">
122122

123-
<b>24 Mar 2025</b>
124-
<p>Rebekka is at <a href="https://cpal.cc/spotlight_track/">CPAL</a> in Stanford presenting three of our <a href="/publications">papers</a> as recent spotlights.</p>
123+
<b>02 Jun 2025</b>
124+
<p>Rebekka and Celia are presenting at the International Network Science Conference (<a href="https://netsci2025.github.io/">NetSci</a>) in Maastricht.</p>
125125

126126
<hr/>
127127

128-
<b>22 Jan 2025</b>
129-
<p>Two papers <a href="https://openreview.net/forum?id=g6v09VxgFw">(1)</a> <a href="https://openreview.net/forum?id=U47ymTS3ut">(2)</a> have been accepted at ICLR 2025 (see <a href="/publications">publications</a>).</p>
128+
<b>01 Jun 2025</b>
129+
<p>Welcome Baraah!</p>
130130

131131
<hr/>
132132

133-
<b>01 Dec 2024</b>
134-
<p>Welcome to Gowtham and Nik!</p>
133+
<b>01 May 2025</b>
134+
<p>Our paper on <a href="https://arxiv.org/pdf/2504.12883">implicit bias</a> has been accepted at ICML 2025.</p>
135135

136136
<hr/>
137137

138-
<b>25 Sep 2024</b>
139-
<p>Three papers <a href="https://openreview.net/forum?id=EMkrwJY2de">(1)</a> <a href="https://openreview.net/forum?id=IfZwSRpqHl">(2)</a> <a href="https://openreview.net/forum?id=FNtsZLwkGr">(3)</a> have been accepted at NeurIPS 2024.</p>
138+
<b>24 Mar 2025</b>
139+
<p>Rebekka is at <a href="https://cpal.cc/spotlight_track/">CPAL</a> in Stanford presenting three of our <a href="/publications">papers</a> as recent spotlights.</p>
140140

141141
<hr/>
142142

143-
<b>01 Jul 2024</b>
144-
<p>Welcome to Chao, Rahul, and Dong!</p>
143+
<b>22 Jan 2025</b>
144+
<p>Two papers <a href="https://openreview.net/forum?id=g6v09VxgFw">(1)</a> <a href="https://openreview.net/forum?id=U47ymTS3ut">(2)</a> have been accepted at ICLR 2025.</p>
145145

146146
<hr/>
147147

‎_site/news.html

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,13 +71,28 @@ <h1 id="news">News</h1>
7171

7272
<hr />
7373

74+
<h4>02 June 2025</h4>
75+
<p>Rebekka and Celia are presenting at the International Network Science Conference (<a href="https://netsci2025.github.io/">NetSci</a>) in Maastricht.</p>
76+
77+
<hr />
78+
79+
<h4>01 June 2025</h4>
80+
<p>Welcome Baraah!</p>
81+
82+
<hr />
83+
84+
<h4>01 May 2025</h4>
85+
<p>Our paper on <a href="https://arxiv.org/pdf/2504.12883">implicit bias</a> has been accepted at ICML 2025.</p>
86+
87+
<hr />
88+
7489
<h4>24 March 2025</h4>
7590
<p>Rebekka is at <a href="https://cpal.cc/spotlight_track/">CPAL</a> in Stanford presenting three of our <a href="/publications">papers</a> as recent spotlights.</p>
7691

7792
<hr />
7893

7994
<h4>22 January 2025</h4>
80-
<p>Two papers <a href="https://openreview.net/forum?id=g6v09VxgFw">(1)</a> <a href="https://openreview.net/forum?id=U47ymTS3ut">(2)</a> have been accepted at ICLR 2025 (see <a href="/publications">publications</a>).</p>
95+
<p>Two papers <a href="https://openreview.net/forum?id=g6v09VxgFw">(1)</a> <a href="https://openreview.net/forum?id=U47ymTS3ut">(2)</a> have been accepted at ICLR 2025.</p>
8196

8297
<hr />
8398

@@ -97,7 +112,7 @@ <h4>01 July 2024</h4>
97112
<hr />
98113

99114
<h4>14 June 2024</h4>
100-
<p>Celia, Advait and Adarsh are presenting at the Helmholtz AI Conference: AI for Science (<a href="https://eventclass.it/haic2024/scientific/external-program/session?s=S-05a">HAICON</a>).</p>
115+
<p>Celia, Advait and Adarsh are presenting at the Helmholtz AI Conference: AI for Science (<a href="https://eventclass.it/haic2024/scientific/external-program/session?s=S-05a">HAICON</a>) in Düsseldorf.</p>
101116

102117
<hr />
103118

‎_site/publications/index.html

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,12 +80,66 @@ <h1 id="publications">Publications</h1>
8080
<option value="Jacobs">Tom Jacobs</option>
8181
<option value="Nittala">Rahul Nittala</option>
8282
<option value="Sun">Dong Sun</option>
83-
<option value="Vedeneev">Nikita (Nik) Vedeneev</option>
83+
<option value="Sidahmed">Baraah Sidahmed</option>
8484
</select></p>
8585

8686
<ol class="bibliography"><li><!-- _layouts/bib.html -->
8787

8888

89+
<div class="row" style="margin-top: 30px;" id="bib-jacobs2025mirror" authors="JacobsZhouBurkholz">
90+
<div class="preview"><img class="preview z-depth-1 rounded" src="/images/publications/mirror-mirror.jpg" />
91+
<!-- include figure.html
92+
path=entry_path
93+
class="preview z-depth-1 rounded"
94+
zoomable=false
95+
alt=entry.img --></div>
96+
<!-- Entry bib key -->
97+
<div id="jacobs2025mirror" class="col-sm-10">
98+
<!-- Title -->
99+
100+
<div class="title"><a href="https://arxiv.org/abs/2504.12883"><b>Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?</b></a></div>
101+
102+
<!-- Author -->
103+
<div class="author">
104+
105+
106+
Tom Jacobs,&nbsp;Chao Zhou,&nbsp;and&nbsp;Rebekka Burkholz</div>
107+
108+
<!-- Journal/Book title and date -->
109+
110+
111+
<div class="periodical">
112+
<em>In Forty-second International Conference on Machine Learning</em>, 2025
113+
</div>
114+
<div class="periodical">
115+
116+
</div>
117+
118+
<!-- Links/Buttons -->
119+
<div class="links"><a class="conf btn btn-sm z-depth-0">ICML</a><a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a>
120+
<a class="abstract btn btn-sm z-depth-0" role="button">Abs</a>
121+
<!-- <a href="https://arxiv.org/pdf/2504.12883" class="btn btn-sm z-depth-0" role="button">PDF</a> -->
122+
</div>
123+
124+
125+
<!-- Hidden abstract block -->
126+
<div class="abstract hidden">
127+
<p>Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent overfitting. While both concepts have been studied separately, in practice, they often act in tandem. Understanding their interplay is key to controlling the shape and strength of implicit bias, as it can be modified by explicit regularization. To this end, we incorporate explicit regularization into the mirror flow framework and analyze its lasting effects on the geometry of the training dynamics, covering three distinct effects: positional bias, type of bias, and range shrinking. Our analytical approach encompasses a broad class of problems, including sparse coding, matrix sensing, single-layer attention, and LoRA, for which we demonstrate the utility of our insights. To exploit the lasting effect of regularization and highlight the potential benefit of dynamic weight decay schedules, we propose to switch off weight decay during training, which can improve generalization, as we demonstrate in experiments.</p>
128+
</div><!-- Hidden bibtex block -->
129+
<div class="bibtex hidden">
130+
<figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">jacobs2025mirror</span><span class="p">,</span>
131+
<span class="na">title</span> <span class="p">=</span> <span class="s">{Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?}</span><span class="p">,</span>
132+
<span class="na">author</span> <span class="p">=</span> <span class="s">{Jacobs, Tom and Zhou, Chao and Burkholz, Rebekka}</span><span class="p">,</span>
133+
<span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Forty-second International Conference on Machine Learning}</span><span class="p">,</span>
134+
<span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span>
135+
<span class="na">url</span> <span class="p">=</span> <span class="s">{https://arxiv.org/abs/2504.12883}</span><span class="p">,</span>
136+
<span class="p">}</span></code></pre></figure>
137+
</div>
138+
</div>
139+
</div></li>
140+
<li><!-- _layouts/bib.html -->
141+
142+
89143
<div class="row" style="margin-top: 30px;" id="bib-jacobs2025mask" authors="JacobsBurkholz">
90144
<div class="preview"><img class="preview z-depth-1 rounded" src="/images/publications/mask_in_the_mirror.png" />
91145
<!-- include figure.html

‎_site/team/index.html

Lines changed: 21 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -233,21 +233,15 @@ <h4>Dong Sun</h4>
233233

234234
<div class="row">
235235

236-
<h3 style="margin-left:16px;">Research engineers</h3>
237-
238236
<div class="col-sm-6 clearfix">
239237

240-
<p><img src="/images/team members/c01mive.jpg" class="img-responsive" width="20%" style="float: left; margin: 0px 22px 24px 0;" /></p>
241-
242-
<h4>Nikita (Nik) Vedeneev</h4>
243-
<p><a href="https://cispa.de/en/people/c01mive">Homepage</a>
244-
| <a href="mailto:mikita.vedzeneyeu@cispa.de">Email</a>
245-
| <a class="membercv" role="button">Description</a> <br />
246-
<i>since Dec 2024</i></p>
238+
<p><img src="/images/team members/blank.jpg" class="img-responsive" width="20%" style="float: left; margin: 0px 22px 24px 0;" /></p>
247239

248-
<div class="membercv hidden">
249-
<p>I am interesting in making modern AI models efficient. In particular, I work on discovering and exploiting structure in Neural Networks (sparsity, low-dimensional representations and similar) for efficient training, fine-tuning and inference. I am a former full-time core developer for <a href="https://github.com/pytorch/pytorch">PyTorch</a> and <a href="https://github.com/Lightning-AI/lightning-thunder">Lightning Thunder</a>. Check my <a href="https://github.com/nikitaved">GitHub</a> to see what I work on now.</p>
250-
</div>
240+
<h4>Baraah Sidahmed</h4>
241+
<p><a href="https://cispa.de/en/people/c01baad">Homepage</a>
242+
| <a href="mailto:baraah.adil@cispa.de">Email</a>
243+
<br />
244+
<i>since Jun 2025</i></p>
251245

252246
</div>
253247

@@ -272,6 +266,21 @@ <h3>Research assistants</h3>
272266
</ul>
273267
</div>
274268

269+
<div class="col-sm-6 clearfix">
270+
<h3>Research engineers</h3>
271+
<ul>
272+
273+
<li style="margin-left:-16px;">Nikita (Nik) Vedeneev:
274+
<i>Dec 24-May 25</i>.
275+
</li>
276+
277+
</ul>
278+
</div>
279+
280+
</div>
281+
282+
<div class="row">
283+
275284
<div class="col-sm-6 clearfix">
276285
<h3>Visiting students</h3>
277286
<ul>

‎images/publications/mirror-mirror.jpg

32.1 KB
Loading

0 commit comments

Comments
 (0)