Deep Learning — Weblinks

A

Attention Is All You Need — the original Transformer paper Pap… 16 terms

+8 more

P

PyTorch Conv2d documentation Docs 5 terms

C

Stanford CS231n — Neural Networks Part 1: Modeling one neuron, activation functions, architecture Art… 5 terms

D

Deep Learning (Goodfellow, Bengio & Courville) — Ch. 6: Deep Feedforward Networks Art… 4 terms

D

Dive into Deep Learning 5.4 — Xavier initialization and variance stability across layers Art… 4 terms

P

PyTorch scaled dot-product attention documentation Docs 4 terms

C

Understanding LSTM Networks (Christopher Olah) Art… 4 terms

D

7.5.1. Maximum Pooling and Average Pooling — Dive into Deep Learning Art… 3 terms

C

CS231n guide to convolutional networks Art… 3 terms

C

CS231n — Backpropagation, Intuitions (computational graphs, chain rule, local and upstream gradients) Art… 3 terms

D

Dive into Deep Learning 5.3 — Forward Propagation, Backward Propagation, and Computational Graphs (intermediate pre-activation z) Art… 3 terms

A

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (Cho et al., 2014) — introduces the gated recurrent unit Pap… 3 terms

B

Long Short-Term Memory — Hochreiter & Schmidhuber, Neural Computation 9(8), 1997 PDF 3 terms

A

Original PReLU paper: Delving Deep into Rectifiers Pap… 3 terms

P

PyTorch MultiheadAttention documentation Docs 3 terms

P

torch.nn.GRU — PyTorch documentation Docs 3 terms

P

Understanding the difficulty of training deep feedforward neural networks (Xavier initialization) Pap… 3 terms

D

11.1. Queries, Keys, and Values — Dive into Deep Learning Art… 2 terms

D

7.2. Convolutions for Images — defines the feature map as a convolutional layer's output (Dive into Deep Learning) Art… 2 terms

D

7.4. Multiple Input and Multiple Output Channels — Dive into Deep Learning Art… 2 terms

D

9.7. Backpropagation Through Time — Dive into Deep Learning Art… 2 terms

A

A guide to convolution arithmetic for deep learning (Dumoulin & Visin) — how kernel, stride and padding set feature-map size Pap… 2 terms

D

Deep Learning (Goodfellow et al.) — Ch. 10 Sequence Modeling: Recurrent and Recursive Nets Art… 2 terms

D

Dive into Deep Learning 5.1 — Multilayer Perceptrons (feedforward architecture and hidden layers) Art… 2 terms

A

He et al., Deep Residual Learning for Image Recognition (ResNet paper) Pap… 2 terms

A

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Pap… 2 terms

A

Network In Network: the paper that introduced 1x1 convolutional channel mixing Pap… 2 terms

A

On the difficulty of training Recurrent Neural Networks Pap… 2 terms

P

PyTorch nn.Linear — learnable weight and bias parameters Docs 2 terms

A

Revisiting Small Batch Training for Deep Neural Networks — best performance at batch sizes 2–32 Pap… 2 terms

P

torch.nn.LSTM — PyTorch documentation (gate equations) Docs 2 terms

P

torch.nn.utils.clip_grad_norm_ — PyTorch documentation Docs 2 terms

D

10.1. Long Short-Term Memory (LSTM) — Dive into Deep Learning (forget gate and internal state) Art… 1 term

D

10.3. Deep Recurrent Neural Networks — Dive into Deep Learning (stacking recurrent layers) Art… 1 term

D

10.4. Bidirectional Recurrent Neural Networks — Dive into Deep Learning Art… 1 term

D

9.4. Recurrent Neural Networks — Dive into Deep Learning (RNNs with hidden states) Art… 1 term

M

A Gentle Introduction to Attention Masking in Transformer Models Art… 1 term

L

A survey on Image Data Augmentation for Deep Learning (Shorten & Khoshgoftaar, J. Big Data 2019) Pap… 1 term

A

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — the gradual learning-rate warmup scheme Pap… 1 term

E

Activation function — Wikipedia (definitions, nonlinearity, comparison tables) Ref… 1 term

A

ADADELTA: An Adaptive Learning Rate Method Pap… 1 term

A

Adam: A Method for Stochastic Optimization Pap… 1 term

N

AMSGrad Optimizer — annotated PyTorch implementation with the max-of-past-second-moments fix Art… 1 term

E

Artificial neuron — Wikipedia Ref… 1 term

P

Autograd mechanics — PyTorch documentation Docs 1 term

P

Automatic Mixed Precision package - torch.amp (PyTorch) Docs 1 term

C

Backpropagation Through Time: What It Does and How to Do It — Werbos, Proc. IEEE 78(10), 1990 (PDF) PDF 1 term

D

Bidirectional Recurrent Neural Networks — Schuster & Paliwal, IEEE Trans. Signal Processing 45(11), 1997 (PDF) PDF 1 term

D

Build the Neural Network — PyTorch tutorial (nn.Module and the forward() method) Docs 1 term

G

clovaai/CutMix-PyTorch — official PyTorch implementation of the CutMix regularizer Code 1 term

D

Computing Receptive Fields of Convolutional Neural Networks (Distill) Art… 1 term

C

Cornell CS4780 Lecture 3: The Perceptron Art… 1 term

C

CS231n: Neural Networks Part 3 — Gradient Checks Art… 1 term

A

CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features Pap… 1 term

D

Cybenko (1989), Approximation by Superpositions of a Sigmoidal Function — Mathematics of Control, Signals and Systems 2:303-314 Art… 1 term

T

Data augmentation — official TensorFlow Core tutorial (Keras preprocessing layers, tf.image) Docs 1 term

A

Decoupled Weight Decay Regularization (AdamW paper) Pap… 1 term

A

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs Pap… 1 term

M

Difference Between a Batch and an Epoch in a Neural Network Art… 1 term

D

Distill: Deconvolution and Checkerboard Artifacts Art… 1 term

D

Dive into Deep Learning 12.5: Minibatch Stochastic Gradient Descent Art… 1 term

D

Dive into Deep Learning 4.1 — Softmax Regression: the output layer and its activation Art… 1 term

D

Dive into Deep Learning 8.6 — Residual Networks (ResNet) and ResNeXt: residual blocks and skip connections Art… 1 term

L

Efficient BackProp (LeCun, Bottou, Orr & Müller, 1998) — the 1/√fan-in initialization rule Art… 1 term

A

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Pap… 1 term

G

facebookresearch/mixup-cifar10 — official mixup implementation from the paper's authors Code 1 term

A

Fixup Initialization: Residual Learning Without Normalization (Zhang, Dauphin & Ma, ICLR 2019) Pap… 1 term

K

GlobalAveragePooling2D layer — Keras API documentation Docs 1 term

D

Google Machine Learning Glossary — bias (math) / bias term Docs 1 term

D

Google ML Crash Course: Hyperparameters — learning rate, batch size, epochs Docs 1 term

A

Group Normalization research paper Pap… 1 term

A

How to Construct Deep Recurrent Neural Networks — Pascanu, Gulcehre, Cho & Bengio (ICLR 2014) Pap… 1 term

H

Hugging Face Transformers: Gradient accumulation (effective batch size, loss scaling) Docs 1 term

O

Incorporating Nesterov Momentum into Adam (Dozat, ICLR 2016 workshop) — the Nadam paper Pap… 1 term

A

Instance Normalization: The Missing Ingredient for Fast Stylization Pap… 1 term

K

Keras 3 API — Input object (input layer) documentation Docs 1 term

K

Keras Dense layer — API documentation (output = activation(dot(input, kernel) + bias)) Docs 1 term

K

Keras documentation: LecunNormal initializer Docs 1 term

K

Keras Lion optimizer API documentation Docs 1 term

K

Keras Model.summary() documentation (prints layers and parameter counts) Docs 1 term

K

Keras ReduceLROnPlateau callback documentation Docs 1 term

L

Label Smoothing — Lei Mao's Log Book (derivation against cross-entropy with soft targets) Art… 1 term

O

Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes Pap… 1 term

A

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (the LAMB paper, arXiv) Pap… 1 term

A

Layer Normalization original paper Pap… 1 term

A

Lookahead Optimizer: k steps forward, 1 step back Pap… 1 term

A

Maxout Networks (Goodfellow, Warde-Farley, Mirza, Courville, Bengio, ICML 2013) — arXiv full text Pap… 1 term

P

Maxout Networks — original paper by Goodfellow et al. Pap… 1 term

G

michaelrzhang/lookahead — reference Lookahead optimizer implementation by the paper's first author Code 1 term

A

Mish: A Self Regularized Non-Monotonic Activation Function Pap… 1 term

A

Mixed Precision Training (Micikevicius et al.) Pap… 1 term

A

Mixup: Beyond Empirical Risk Minimization Pap… 1 term

A

Multi-Scale Context Aggregation by Dilated Convolutions Pap… 1 term

A

Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho, Bengio) Pap… 1 term

D

Neural networks: Multi-class classification (softmax output layer) — Google Machine Learning Crash Course Docs 1 term

D

Neural networks: Nodes and hidden layers — Google Machine Learning Crash Course Docs 1 term

N

Nielsen, Neural Networks and Deep Learning — Ch. 4: A visual proof that neural nets can compute any function Art… 1 term

L

Number of Parameters and Tensor Sizes in a Convolutional Neural Network (LearnOpenCV) Art… 1 term

A

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (Keskar et al., 2016) Pap… 1 term

A

On Layer Normalization in the Transformer Architecture Pap… 1 term

A

On the Convergence of Adam and Beyond (the AMSGrad paper) Pap… 1 term

O

ONNX HardSigmoid operator specification — y = max(0, min(1, alpha*x + beta)) Art… 1 term

D

Optimizing Model Parameters — PyTorch official training-loop tutorial (train_loop / test_loop) Docs 1 term

J

Original Adagrad paper: Adaptive Subgradient Methods for Online Learning and Stochastic Optimization Pap… 1 term

A

Original Batch Normalization paper Pap… 1 term

A

Original ELU paper: Fast and Accurate Deep Network Learning by Exponential Linear Units Pap… 1 term

A

Original GELU paper: Gaussian Error Linear Units Pap… 1 term

A

Original Stochastic Depth paper Pap… 1 term

H

Performing gradient accumulation with Accelerate (Hugging Face) Docs 1 term

A

Pooling Methods in Deep Neural Networks, a Review (Gholamalinezhad & Khosravi, 2020) Pap… 1 term

A

Professor Forcing: A New Algorithm for Training Recurrent Networks — Lamb et al. (NeurIPS 2016) Pap… 1 term

P

PyTorch Adadelta optimizer documentation Docs 1 term

D

PyTorch Adagrad documentation Docs 1 term

P

PyTorch Adam optimizer documentation Docs 1 term

P

PyTorch AdamW documentation Docs 1 term

P

PyTorch AvgPool2d documentation Docs 1 term

P

PyTorch BatchNorm2d documentation Docs 1 term

P

PyTorch ConvTranspose2d documentation Docs 1 term

P

PyTorch CosineAnnealingLR documentation Docs 1 term

P

PyTorch documentation for Kaiming normal initialization Docs 1 term

P

PyTorch documentation for the Sigmoid module Docs 1 term

P

PyTorch Dropout module documentation Docs 1 term

P

PyTorch GELU documentation Docs 1 term

P

PyTorch GroupNorm documentation Docs 1 term

P

PyTorch Hardsigmoid documentation Docs 1 term

P

PyTorch LayerNorm documentation Docs 1 term

P

PyTorch LeakyReLU documentation Docs 1 term

P

PyTorch LinearLR scheduler documentation Docs 1 term

P

PyTorch MaxPool2d documentation Docs 1 term

P

PyTorch Mish documentation Docs 1 term

P

PyTorch OneCycleLR documentation Docs 1 term

P

PyTorch optimizer and parameter-update documentation Docs 1 term

P

PyTorch orthogonal_ initialization documentation Docs 1 term

P

PyTorch PReLU documentation Docs 1 term

P

PyTorch ReduceLROnPlateau documentation Docs 1 term

P

PyTorch ReLU documentation Docs 1 term

P

PyTorch RMSprop optimizer documentation Docs 1 term

P

PyTorch SiLU documentation Docs 1 term

P

PyTorch Softplus documentation Docs 1 term

P

PyTorch spectral_norm parametrization documentation Docs 1 term

P

PyTorch Tanh documentation Docs 1 term

D

PyTorch torch.nn.InstanceNorm2d documentation — per-sample, per-channel normalization Docs 1 term

D

PyTorch torch.optim.NAdam documentation Docs 1 term

P

PyTorch Transformer module documentation Docs 1 term

P

PyTorch TransformerDecoderLayer documentation Docs 1 term

P

PyTorch TransformerEncoderLayer documentation Docs 1 term

P

PyTorch weight_norm parametrization documentation Docs 1 term

C

Rectified Linear Units Improve Restricted Boltzmann Machines (Nair & Hinton, ICML 2010) PDF 1 term

A

Rectifier Nonlinearities Improve Neural Network Acoustic Models (Maas, Hannun & Ng, ICML 2013) PDF 1 term

A

ReZero is All You Need: Fast Convergence at Large Depth — zero-initialized residual gate = identity at init Pap… 1 term

A

RoFormer: Enhanced Transformer with Rotary Position Embedding Pap… 1 term

D

Rosenblatt (1958), The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain — Psychological Review 65(6):386-408 Art… 1 term

A

Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (Bengio et al., 2015) Pap… 1 term

A

Searching for Activation Functions: the Swish paper Pap… 1 term

A

Self-Normalizing Neural Networks paper Pap… 1 term

A

SGDR: Stochastic Gradient Descent with Warm Restarts Pap… 1 term

E

Sigmoid function — Wikipedia (logistic sigmoid, range, saturation, use as activation) Ref… 1 term

E

Softmax function — Wikipedia (logits to probability distribution, numerical stability, attention) Ref… 1 term

A

Spectral Normalization for Generative Adversarial Networks Pap… 1 term

C

Stanford CS231n — Setting up the data and the model: “Pitfall: all zero initialization” Art… 1 term

A

Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates (the one-cycle policy) Pap… 1 term

A

Symbolic Discovery of Optimization Algorithms (Lion paper) Pap… 1 term

A

Telgarsky (2016), Benefits of Depth in Neural Networks Pap… 1 term

T

TensorFlow guide to writing a training loop Docs 1 term

T

TensorFlow Keras RMSprop documentation Docs 1 term

T

TensorFlow tf.nn.convolution documentation (SAME / VALID padding) Docs 1 term

H

Text generation — autoregressive generation with LLMs (Hugging Face Transformers docs) Docs 1 term

T

tf.keras.activations.tanh — TensorFlow API documentation (formula, range [-1,1], zero-centred) Docs 1 term

T

tf.keras.Input — TensorFlow API docs (input tensor shape and dtype specification) Docs 1 term

T

tf.keras.mixed_precision.LossScaleOptimizer (TensorFlow) Docs 1 term

T

tf.math.softplus — TensorFlow API documentation (softplus(x) = log(exp(x) + 1)) Docs 1 term

T

tf.nn.depthwise_conv2d — TensorFlow API docs for per-channel depthwise convolution Docs 1 term

A

The Expressive Power of Neural Networks: A View from the Width (Lu et al., 2017) Pap… 1 term

J

The Illustrated GPT-2 — auto-regressive decoding, one token at a time Art… 1 term

J

The Illustrated Transformer Art… 1 term

P

The original DropConnect research paper Pap… 1 term

J

The original Dropout paper in the Journal of Machine Learning Research Pap… 1 term

K

The Unreasonable Effectiveness of Recurrent Neural Networks (Andrej Karpathy) Art… 1 term

D

torch.amp: autocast and gradient scaling (GradScaler) documentation Docs 1 term

D

torch.autograd.gradcheck.gradcheck — PyTorch documentation (analytical vs finite-difference gradients) Docs 1 term

P

torch.nn.ELU — PyTorch documentation (ELU formula and alpha parameter) Docs 1 term

P

torch.nn.RNN — PyTorch documentation (hidden state h_t, initial h_0 and final h_n) Docs 1 term

P

torch.nn.SELU — PyTorch documentation (scale/alpha constants, self-normalizing networks note) Docs 1 term

P

torch.nn.Softmax — PyTorch documentation (formula and dim argument) Docs 1 term

P

Torchvision stochastic_depth documentation Docs 1 term

P

Training with PyTorch (official training-loop tutorial) Docs 1 term

A

Transformer Feed-Forward Layers Are Key-Value Memories (Geva et al., 2021) Pap… 1 term

H

Transformer-based Encoder-Decoder Models — Hugging Face (cross-attention layer) Docs 1 term

A

Understanding the Difficulty of Training Transformers (Liu et al., 2020) — Post-LN instability and Admin Pap… 1 term

A

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks (Luo et al., NIPS 2016) Pap… 1 term

E

Vanishing gradient problem — Wikipedia Ref… 1 term

A

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks Pap… 1 term

A

When Does Label Smoothing Help? (Müller, Kornblith & Hinton, NeurIPS 2019) Pap… 1 term

A

Wide Residual Networks (Zagoruyko & Komodakis, 2016) — trading depth for width Pap… 1 term

A

Xception: Deep Learning with Depthwise Separable Convolutions (Chollet, 2017) Pap… 1 term