Regularization: Fighting Overfitting

The goal

Regularization discourages a model from memorizing noise, so it performs better on new data.

Common techniques

• L2 / L1 — penalize large weights (weight decay).
• Dropout — randomly disable neurons during training (deep nets).
• Early stopping — stop when validation loss stops improving.
• Data augmentation — expand training data with realistic variations.
• More data — the most reliable regularizer of all.

How to apply

Add one technique at a time and watch the gap between training and validation scores shrink. A small, well-regularized model often beats a big, unconstrained one.