#machine-learning, eth/cil/theory

Activation Functions

Rectified Linear Unit (ReLU)

\begin{aligned} f (x) & = max (0, x) \\ f^{'} (x) & = {\begin{cases} 0 for x \leq 0 \\ 1 for x > 0 \end{cases} \end{aligned}

\begin{aligned} σ (x) & = \frac{1}{1 + e^{- x}} = \frac{e^{x}}{e^{x} + 1} \\ σ^{'} (x) & = σ (x) (1 - σ (x)) \end{aligned}

\begin{aligned} \tanh (x) & = \frac{e^{x} - e^{- x}}{e^{x} + e^{- x}} \\ \tanh^{'} (x) & = 1 - \tanh^{2} (x) \end{aligned}