Activation Function

An activation function is the small operation that gives a neural network the ability to learn bends, boundaries, and decisions rather than just scaling inputs. A neuron first computes a weighted sum, z = Wx + b, then transforms it into an output a = f(z).

Why nonlinearity matters
Without f, every layer is just a linear transformation. Stacking ten such layers still collapses mathematically into one larger linear transformation, no matter how many parameters it contains. An activation function breaks that collapse: different inputs can be amplified, suppressed, or reshaped at each layer. During backpropagation, its derivative, f′(z), also controls how much error signal reaches earlier layers. If these derivatives repeatedly become tiny, gradients fade and early layers stop learning.

Common choices and their trade-offs
Hidden layers commonly use:

• ReLU, max(0, z): cheap, simple, and passes a full gradient for positive values. But a unit stuck on the negative side has zero output and zero gradient: a “dead ReLU.”
• Leaky ReLU: keeps a small negative-side slope, reducing dead units.
• GELU: smoothly scales values rather than applying a hard cutoff; it is used inside Transformer blocks.
• Sigmoid and tanh: useful when bounded outputs are needed, but their flat saturated regions produce tiny gradients in deep hidden stacks.

Training consequences
Activation choice works with initialization and normalization. A deep ReLU network is commonly paired with He initialization; poor scaling can push most units negative, leaving learning stalled. In a Transformer, LayerNorm helps keep the inputs to GELU in a usable range. A loss curve that plateaus early can signal saturated activations or dead units; exploding or erratic values can indicate poorly controlled activation scales. Activations are inexpensive individually, but their intermediate outputs must be stored for backpropagation, so very large networks also pay a memory cost.