ELU activation

[zhouxiao-coder]

Add ELU activation operator, resolve #4364 .
SELU activation is not added to the activation_op.{h cc cu} file, instead it can be trivially implemented with this ELU interface. Check #4364 for more details.

[tonyyang-svail]

it could be changed to AddOutput("Y", "Output of ELU operator").NotInGradient();, cause y is not used in ELUGradKernel.

[zhouxiao-coder]

@tonyyang-svail Thanks for pointing that out, you are right.

However, after reexamining the math, I find there is a better way to rewrite the gradient formulation which does require using y. I'm working on it, and this PR will be updated soon.

[zhouxiao-coder]

Done.
Now the gradient of negative part is computed as follow:
'''
dy * (y + alpha) * (x < static_cast(0)).template cast();
'''
So now value of y is used in the gradient calculation.

[QiJune]

There may be some non-differentiable point in elu. You can refer to #4120

[zhouxiao-coder]

Thanks for the reference, I'm a little bit confused by the line x[np.abs(x) < 0.005] = 0.02 before and #4120 explains the motivation.

However, ELU has a quite "smooth" negative part, so this modification may be unnecessary here. In fact, elu(x=-0.005) gives -0.0049875208073176802, which makes the relative numeric gradient error less than 2e-5 and is small enough to pass normal gradient check.
Here is the code I used to experiment.

import matplotlib.pyplot as plt
import numpy as np
def elu(x):
    return np.maximum(0, x) + np.minimum(0, 1. * (np.exp(x) - 1))
x = np.linspace(-0.5, 0.5, num=100)
y = elu(x)
plt.plot(x,y)

[zhouxiao-coder]

But on the other hand, I can make another PR if you think we should make this modification x[np.abs(x) < 0.005] = 0.02 a common practice in unit test.

[QiJune]

Yes, elu is actually very smooth. 0 is a non-differentiable point in elu. We'd better to filter it in out test data.
This line

x[np.abs(x) < 0.005] = 0.02

will filter these potential points and set them to 0.02.(0.02-0.005 = 0.015;0.02+0.005=0.025, both of 0.015 and 0.025 are larger than 0)

[zhouxiao-coder]

As long as we are talking about the standard elu, i.e. alpha=1, then it is differentiable at point 0 since both left derivate and right derivate at 0 equal to 1. The "piecewise" form of the function only makes it non-differentiable on the derivative function at 0, i.e. the second order derivative.

[zhouxiao-coder]

@QiJune I didn't add the filtering code to the new commit, since as we discussed it is not necessary. However, I add a comment to explain the reason.

[lcy-seso]

Please follow our comment style https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/name_convention.md#opprotomaker-names

I think fc_op has a good comment style.

[zhouxiao-coder]

I see. fc_op provides a nice reference indeed, and I'll try to follow that style.

[zhouxiao-coder]

Done. I didn't squash all my commit because they get interleaved with other's commit. Please check commit 4436ba0 for the new comments. Sorry for the inconvenience

[qingqing01]

LGTM.