Optimizers — Weblinks

A

Revisiting Small Batch Training for Deep Neural Networks — best performance at batch sizes 2–32 Pap… 2 terms

A

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — the gradual learning-rate warmup scheme Pap… 1 term

A

ADADELTA: An Adaptive Learning Rate Method Pap… 1 term

A

Adam: A Method for Stochastic Optimization Pap… 1 term

N

AMSGrad Optimizer — annotated PyTorch implementation with the max-of-past-second-moments fix Art… 1 term

P

Automatic Mixed Precision package - torch.amp (PyTorch) Docs 1 term

A

Decoupled Weight Decay Regularization (AdamW paper) Pap… 1 term

M

Difference Between a Batch and an Epoch in a Neural Network Art… 1 term

D

Dive into Deep Learning 12.5: Minibatch Stochastic Gradient Descent Art… 1 term

D

Google ML Crash Course: Hyperparameters — learning rate, batch size, epochs Docs 1 term

H

Hugging Face Transformers: Gradient accumulation (effective batch size, loss scaling) Docs 1 term

O

Incorporating Nesterov Momentum into Adam (Dozat, ICLR 2016 workshop) — the Nadam paper Pap… 1 term

K

Keras Lion optimizer API documentation Docs 1 term

K

Keras ReduceLROnPlateau callback documentation Docs 1 term

O

Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes Pap… 1 term

A

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (the LAMB paper, arXiv) Pap… 1 term

A

Lookahead Optimizer: k steps forward, 1 step back Pap… 1 term

G

michaelrzhang/lookahead — reference Lookahead optimizer implementation by the paper's first author Code 1 term

A

Mixed Precision Training (Micikevicius et al.) Pap… 1 term

A

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (Keskar et al., 2016) Pap… 1 term

A

On the Convergence of Adam and Beyond (the AMSGrad paper) Pap… 1 term

A

On the difficulty of training Recurrent Neural Networks (Pascanu, Mikolov, Bengio) Pap… 1 term

D

Optimizing Model Parameters — PyTorch official training-loop tutorial (train_loop / test_loop) Docs 1 term

J

Original Adagrad paper: Adaptive Subgradient Methods for Online Learning and Stochastic Optimization Pap… 1 term

H

Performing gradient accumulation with Accelerate (Hugging Face) Docs 1 term

P

PyTorch Adadelta optimizer documentation Docs 1 term

D

PyTorch Adagrad documentation Docs 1 term

P

PyTorch Adam optimizer documentation Docs 1 term

P

PyTorch AdamW documentation Docs 1 term

P

PyTorch CosineAnnealingLR documentation Docs 1 term

P

PyTorch LinearLR scheduler documentation Docs 1 term

P

PyTorch OneCycleLR documentation Docs 1 term

P

PyTorch optimizer and parameter-update documentation Docs 1 term

P

PyTorch ReduceLROnPlateau documentation Docs 1 term

P

PyTorch RMSprop optimizer documentation Docs 1 term

P

PyTorch torch.nn.utils.clip_grad_norm_ documentation Docs 1 term

D

PyTorch torch.optim.NAdam documentation Docs 1 term

A

SGDR: Stochastic Gradient Descent with Warm Restarts Pap… 1 term

A

Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates (the one-cycle policy) Pap… 1 term

A

Symbolic Discovery of Optimization Algorithms (Lion paper) Pap… 1 term

T

TensorFlow guide to writing a training loop Docs 1 term

T

TensorFlow Keras RMSprop documentation Docs 1 term

T

tf.keras.mixed_precision.LossScaleOptimizer (TensorFlow) Docs 1 term

D

torch.amp: autocast and gradient scaling (GradScaler) documentation Docs 1 term

P

Training with PyTorch (official training-loop tutorial) Docs 1 term