ELU
ELU, short for Exponential Linear Unit, is an activation function designed to keep a neural network learning even when a unit receives negative inputs. It behaves like ReLU for positive values, but instead of switching negative values completely off, it follows a smooth curve below zero.
How ELU transforms a signal
For an input x, ELU is:
- x when x is positive
- α(ex − 1) when x is zero or negative
The parameter α, commonly set to 1, controls the negative curve’s lowest value: ELU approaches −α rather than falling without limit. On the positive side it retains ReLU’s simple, non-saturating gradient. On the negative side, it has a nonzero gradient for inputs near zero, so a unit is less likely to become permanently inactive.
Why this helps training
A plain ReLU outputs zero, and has zero gradient, for every negative input. If a neuron is repeatedly pushed into that region, gradient descent cannot move its incoming weights through that neuron: this is the dying ReLU problem. ELU gives the negative region a smooth escape route. Its negative outputs also pull average activations closer to zero, which can make optimization more stable by reducing shifts in the signal passed between layers.
Practical trade-offs
ELU is available as torch.nn.ELU in PyTorch. It can be useful in feed-forward or convolutional hidden layers when ReLU units die or loss progress stalls. Unlike ReLU, however, ELU requires evaluating an exponential for negative inputs, so it costs more compute. Its negative branch also saturates far below zero, where gradients become very small; it reduces dead units, but does not eliminate vanishing gradients in a very deep poorly designed network. Modern architectures frequently pair simpler activations with LayerNorm, residual connections, and Adam-style optimization instead.
ELU (Exponential Linear Unit) is an activation function that returns its input for positive values and a smooth negative exponential value for negative inputs. Unlike ReLU, it retains nonzero negative-side outputs and gradients, reducing permanently inactive (“dying”) units and shifting activations toward zero mean. This can improve gradient flow and training stability, though ELU is more computationally expensive than ReLU.
ELU, short for Exponential Linear Unit, is like a kinder on/off switch inside a neural network. A basic switch may shut completely when it receives an unhelpful signal, and then it can be hard for the network to learn from that part again. ELU avoids such a harsh shutdown.
For useful positive signals, it lets information pass through much like a simple open gate. For negative signals, instead of cutting them off entirely, it keeps a small, smooth response. This helps more parts of the network stay active and learn from examples, which can make training steadier and sometimes faster.