ReLU
Technology & Computing abbreviation · Artificial Intelligence
| Stands for | Rectified linear unit |
|---|---|
| Categories | Technology & Computing |
ReLU stands for Rectified linear unit, an artificial intelligence abbreviation.
An activation function that outputs the input if it is positive and zero otherwise, written f(x) = max(0, x). Its simplicity and resistance to vanishing gradients made it the default activation in deep networks during the 2010s.
Example: Replace the sigmoid layers with ReLU and the deep network finally starts learning.
Usage notes
Popularized for deep learning by Vinod Nair and Geoffrey Hinton in 2010 and by AlexNet in 2012. Units stuck at zero output are called "dead ReLUs". Smoother variants such as GELU and SwiGLU are common in modern transformers. Pronounced "REL-you".
Related abbreviations
More artificial intelligence abbreviations
- PEFT – Parameter-efficient fine-tuning
- POS – Part of speech
- PPO – Proximal policy optimization
- PR – Precision-recall (curve)
- QA – Question answering
- RAG – Retrieval-augmented generation
- RF – Random forest
- RL – Reinforcement learning
- RLAIF – Reinforcement learning from AI feedback
- RLHF – Reinforcement learning from human feedback
- RMSE – Root mean square error
- RNN – Recurrent neural network