MMLU
Technology & Computing abbreviation · Artificial Intelligence
| Stands for | Massive Multitask Language Understanding |
|---|---|
| Categories | Technology & Computing |
MMLU stands for Massive Multitask Language Understanding, an artificial intelligence abbreviation.
A benchmark of about 16,000 multiple-choice questions across 57 subjects, from elementary math to law and medicine, used to compare the general knowledge and reasoning of language models.
Example: The model card reports an MMLU score of 86 percent, close to expert level.
Usage notes
Published by Dan Hendrycks and colleagues in 2020. Leading models now score near the ceiling, and audits found errors in some questions, so harder variants such as MMLU-Pro and newer benchmarks have taken over. Contamination, where test questions leak into training data, complicates comparisons.
Related abbreviations
More artificial intelligence abbreviations
- MCP – Model Context Protocol
- MDP – Markov decision process
- ML – Machine learning
- MLE – Maximum likelihood estimation
- MLOps – Machine learning operations
- MLP – Multilayer perceptron
- MoE – Mixture of experts
- MSE – Mean squared error
- MT – Machine translation
- NER – Named-entity recognition
- NeRF – Neural radiance field
- NLG – Natural language generation