Question about SGD update

[codlife]

Hi @zhangyuc
I find in your code, the weight update is as below:
Local mode: number of cores on the local machine

while(k < n) { values(k) *= - actual_stepsize / math.sqrt(s(k) + 1e-16f) k += 1 }
what troube me is why the update is this, which seem not the standard format:
such as w = w - stepsize*gradient.
thank you!

[zhangyuc]

It implements the AdaGrad algorithm (http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf), which chooses different stepsizes for different coordinates. The stepsize of coordinate k is controlled by the accumulated gradient squared-norm s(k).

[codlife]

got it,thank you!