ReLU Function

Introduced with AlexNet, the Rectifier Linear Unit function solves a lot of problems.

$$\text{ReLU}(x)=\max (0,x)$$

First of all it doesn’t kill the gradients for positive $x$s. It’s also very efficient and allows a faster convergence.

The two problems are that gradients are killed for negative $x$s, and the fact that is not zero-centered like sigmoid or tanh. Here we will introduce some ReLU variants that aim to solve some of these problems:

• $\text{Leaky ReLU}(x)=\max (0.1x,x)$: doesn’t kill the gradients by introducing a small coeficient.
• $\text{Parametric ReLU}(x)=\max (\alpha x,x)$: like the Leaky ReLU, but makes the coefficeint $\alpha$ learnable.
• $\text{ELU}(x)=\{\begin{array}{llc}x& & x\ge 0\\ \alpha (e^x-1)& & x<0\end{array}$: Exponential Linear Unit, which is less efficient because of the exponential computation.
• $\text{SELU}(x)$: Scaled ELU, which is better for deep networks since it has a self-normalizing properties and so it doesn’t need Batch Norm.