Tanh
Tanh—short for hyperbolic tangent—is a smooth activation function that turns any real-valued input into a number between −1 and +1. It gives a neural network a controlled, non-linear response: large negative inputs approach −1, large positive inputs approach +1, and inputs near zero retain fine detail.
How it transforms signals
Mathematically, tanh is written as tanh(x). Its S-shaped curve resembles the sigmoid function, but it is zero-centred: an input of zero produces zero. That matters because a layer’s activations can contain both positive and negative values rather than being shifted entirely above zero. Near zero, tanh behaves almost like the identity function, so small changes in input produce meaningful changes in output. Its derivative is 1 − tanh²(x), which reaches its maximum of 1 at zero.
Saturation and fading gradients
The same bounding behavior that makes tanh stable also creates its main weakness. When an input is strongly positive or negative, tanh is nearly flat at +1 or −1. In these saturated regions, its derivative is close to zero. During backpropagation, gradients are multiplied through each layer; many small derivatives can make the gradient reaching early layers vanish. A deep network can then appear to train while its first layers barely change.
- Poorly scaled inputs or overly large initial weights push activations into saturation.
- Xavier/Glorot initialization was designed in part to keep activations and gradients at workable scales for tanh-like functions.
- LayerNorm can keep pre-activation values closer to tanh’s responsive central region.
Where tanh still fits
Tanh appears in classic recurrent networks, where hidden states benefit from being bounded and signed; PyTorch exposes it as torch.nn.Tanh. It is also useful for an output that must lie in −1 to +1. For very deep hidden stacks, ReLU-family activations usually train more reliably because they avoid tanh’s positive-side saturation. Tanh trades easy gradient flow at depth for smooth, bounded outputs.
Tanh (hyperbolic tangent) is a smooth, zero-centered activation function that maps any input to the bounded range −1 to 1. Its derivative is largest near zero and approaches zero for large positive or negative inputs, causing saturation. Tanh provides nonlinear, signed activations, but deep networks using it can suffer vanishing gradients when units saturate, slowing or preventing effective training.
Imagine a volume knob that can turn smoothly from fully left to fully right, but never beyond either limit. Tanh does something similar inside a neural network: it takes any number and reshapes it into a value between -1 and 1.
Small inputs stay responsive, while very large positive or negative inputs get pushed close to the limits. This gives the network a balanced way to express “strongly yes,” “strongly no,” or anything in between. It was especially useful in older systems that processed sequences, such as text or speech. Its limits can also make learning slow when signals get stuck near -1 or 1, so newer alternatives are often preferred.