KANs are MLPs? MLP Equivalent

[hypnopump]

This note shows equivalence of KAN to MLP in the piecewise linear approximation. I guess non-linearity of spline might help in some cases, but would be cool to have it as a baseline. Here's the reddit discussion

[minh-nguyenhoang]

Only if you are using b-splines of order 1

[Blealtan]

It's equivalent to activating the same hidden state with multiple activation functions and then use a wider linear transformation to shrink it back. Somewhat like Gated Linear Unit, but somewhat in reverse: linear transformation goes after the broadening activation.

[bilzard]

Hi. I also doubt fundamental difference between KANs and MLPs.

Suppose input dimension is equal to output dimension, KAN is mathematically equivalent to MLP with residual connections (see the rightmost picture in the below diagram).
Trainable non-linear activation function in KAN is different from fixed non-linear activation in MLP. However, I think they aren't fundamentally different.

Feel free to point out if I make some misunderstanding.

P.S.
@Blealtan Thank you for releasing implementation of efficient-KAN. The above insight is deeply inspired by your implementation.

[bilzard]

Sorry, the above statement is incorrect. f(x)+g(x) and g(f(x)+x)+f(x)+x are not generally equivalent.