Batch Normalization

Batch normalization standardizes the activations within a mini-batch so they maintain a more consistent distribution during training. The normalized values are then scaled and shifted using learnable parameters. This process can make neural network training more stable and allow models to use higher learning rates while reducing sensitivity to parameter initialization.

Training deep neural networks can become unstable as values change across layers. Batch normalization helps stabilize these changes, often enabling faster training and more reliable convergence. It can also reduce sensitivity to initial weights and, in some cases, provide a mild regularizing effect.