Rectified Linear Unit (ReLU)
Rectified Linear Unit (ReLU) is an activation function commonly used in neural networks that outputs zero for negative inputs and the input value itself for positive inputs.
What is Rectified Linear Unit (ReLU)?
ReLU is defined as f(x) = max(0, x). It introduces non-linearity into neural networks, allowing them to learn complex patterns. Unlike some traditional activation functions, ReLU is computationally simple and helps neural networks train efficiently.
Why is Rectified Linear Unit (ReLU) Important?
ReLU helps deep neural networks learn complex relationships while generally reducing some of the optimization difficulties associated with older activation functions. It is widely used in hidden layers of modern deep learning architectures.
Common use cases
ReLU is commonly used in convolutional neural networks, feed-forward neural networks, computer vision systems, and other deep learning models.