Activation Functions

E

Activation function — Wikipedia (definitions, nonlinearity, comparison tables) Ref… 1 term

C

CS231n — Neural Networks Part 1: commonly used activation functions Art… 1 term

A

Maxout Networks (Goodfellow, Warde-Farley, Mirza, Courville, Bengio, ICML 2013) — arXiv full text Pap… 1 term

P

Maxout Networks — original paper by Goodfellow et al. Pap… 1 term

A

Mish: A Self Regularized Non-Monotonic Activation Function Pap… 1 term

O

ONNX HardSigmoid operator specification — y = max(0, min(1, alpha*x + beta)) Art… 1 term

A

Original ELU paper: Fast and Accurate Deep Network Learning by Exponential Linear Units Pap… 1 term

A

Original GELU paper: Gaussian Error Linear Units Pap… 1 term

A

Original PReLU paper: Delving Deep into Rectifiers Pap… 1 term

P

PyTorch documentation for the Sigmoid module Docs 1 term

P

PyTorch GELU documentation Docs 1 term

P

PyTorch Hardsigmoid documentation Docs 1 term

P

PyTorch LeakyReLU documentation Docs 1 term

P

PyTorch Mish documentation Docs 1 term

P

PyTorch PReLU documentation Docs 1 term

P

PyTorch ReLU documentation Docs 1 term

P

PyTorch SiLU documentation Docs 1 term

P

PyTorch Softplus documentation Docs 1 term

P

PyTorch Tanh documentation Docs 1 term

C

Rectified Linear Units Improve Restricted Boltzmann Machines (Nair & Hinton, ICML 2010) PDF 1 term

A

Rectifier Nonlinearities Improve Neural Network Acoustic Models (Maas, Hannun & Ng, ICML 2013) PDF 1 term

A

Searching for Activation Functions: the Swish paper Pap… 1 term

A

Self-Normalizing Neural Networks paper Pap… 1 term

E

Sigmoid function — Wikipedia (logistic sigmoid, range, saturation, use as activation) Ref… 1 term

E

Softmax function — Wikipedia (logits to probability distribution, numerical stability, attention) Ref… 1 term

T

tf.keras.activations.tanh — TensorFlow API documentation (formula, range [-1,1], zero-centred) Docs 1 term

T

tf.math.softplus — TensorFlow API documentation (softplus(x) = log(exp(x) + 1)) Docs 1 term

P

torch.nn.ELU — PyTorch documentation (ELU formula and alpha parameter) Docs 1 term

P

torch.nn.SELU — PyTorch documentation (scale/alpha constants, self-normalizing networks note) Docs 1 term

P

torch.nn.Softmax — PyTorch documentation (formula and dim argument) Docs 1 term