Deep Learning

• Activation Function

• Zero Initialization
• Random Initialization
• Xavier (Glorot) Initialization
• He Initialization
• LeCun Initialization
• Orthogonal Initialization
• Identity Initialization

• Backpropagation
• Gradient Flow
• Local Gradient
• Upstream Gradient
• Backpropagation Through Time (BPTT)
• Vanishing Gradients
• Exploding Gradients
• Gradient Checking

• Recurrent Neural Network (RNN)
• Hidden State
• Long Short-Term Memory (LSTM)
• Cell State
• Forget Gate
• Input Gate
• Output Gate
• Gated Recurrent Unit (GRU)
• Update Gate (GRU)
• Reset Gate (GRU)
• Bidirectional RNN
• Stacked RNN
• Truncated BPTT
• Teacher Forcing