LLMcompare

Search

Search for a command to run...

Reasoning & knowledge

MMLU-Pro

Harder multi-task language understanding across STEM and professional domains; reported results depend on prompt, shot count, and reasoning/tool settings.

How models are tested

Scores are published numbers from the sources below. They are only comparable when the evaluation version, tools, agent harness, and sampling setup match. Missing scores are omitted rather than treated as zero.

Category
Reasoning & knowledge
Metric
Percent
Direction
Higher is better
Coverage
88 / 318Models in this catalog with a published score
Updated
Catalog snapshot date
Source
github.com/TIGER-AI-Lab/MMLU-Pro

Scores

  1. 1
    Kimi K2.5

    Moonshotopen

    87.1%
  2. 2
    Nemotron 3 Ultra

    NVIDIAopen

    86.8%
  3. 3
    Grok 4

    xAIclosed

    86.6%
  4. 4
    Gemini 2.5 Pro

    Googleclosed

    86.2%
  5. 5
    DeepSeek V3.2

    DeepSeekopen

    86.2%
  6. 6
    o3

    OpenAIclosed

    85.3%
  7. 7
    Gemma 4 31B

    Googleopen

    85.2%
  8. 8
    DeepSeek R1-0528

    DeepSeekopen

    85.0%
  9. 9
    DeepSeek V3.1

    DeepSeekopen

    84.8%
  10. 10
    o1 Preview

    OpenAIclosed

    84.8%
  11. 11
    Qwen3 Max

    Alibabaclosed

    84.1%
  12. 12
    o1

    OpenAIclosed

    84.1%
  13. 13
    DeepSeek R1

    DeepSeekopen

    84.0%
  14. 14
    Gemma 4 26B

    Googleopen

    82.6%
  15. 15
    Kimi K2

    Moonshotopen

    81.1%
  16. 16
    Mistral Large 3

    Mistralopen

    80.7%
  17. 17
    GPT-4.1

    OpenAIclosed

    80.6%
  18. 18
    Qwen3-Next 80B-A3B

    Alibabaopen

    80.6%
  19. 19
    Llama 4 Maverick

    Metaopen

    80.5%
  20. 20
    Grok 3

    xAIclosed

    79.9%
  21. 21
    DeepSeek-R1-Distill-Llama-70B

    DeepSeekopen

    79.5%
  22. 22
    o3-mini

    OpenAIclosed

    79.1%
  23. 23
    Qwen3 Coder 480B

    Alibabaopen

    78.8%
  24. 24
    Nemotron Nano 3 30B

    NVIDIAopen

    78.3%
  25. 25
    GPT-4.1 Mini

    OpenAIclosed

    78.1%
  26. 26
    Granite 4.2 30B

    IBMopen

    77.6%
  27. 27
    Claude 3.5 Sonnet (Oct)

    Anthropicclosed

    77.2%
  28. 28
    QwQ-32B

    Alibabaopen

    76.4%
  29. 29
    Devstral 2

    Mistralopen

    76.2%
  30. 30
    DeepSeek V3

    DeepSeekopen

    75.9%
  31. 31
    OLMo 3 32B Think

    Ai2open

    75.9%
  32. 32
    Sonar Pro

    Perplexityclosed

    75.5%
  33. 33
    Claude 3.5 Sonnet

    Anthropicclosed

    75.1%
  34. 34
    Llama 4 Scout

    Metaopen

    74.3%
  35. 35
    Phi-4 Reasoning

    Microsoftopen

    74.3%
  36. 36
    o1 Mini

    OpenAIclosed

    74.2%
  37. 37
    Granite 4.2 8B

    IBMopen

    74.0%
  38. 38
    GPT-4o (May 2024)

    OpenAIclosed

    74.0%
  39. 39
    DeepSeek-R1 Distill Qwen 14B

    DeepSeekopen

    74.0%
  40. 40
    DeepSeek-R1-Distill-Qwen-32B

    DeepSeekopen

    73.9%
  41. 41
    Llama 3.1 405B

    Metaopen

    73.3%
  42. 42
    Command A

    Cohereclosed

    71.2%
  43. 43
    Qwen2.5 72B

    Alibabaopen

    71.1%
  44. 44
    Qwen3-Coder 30B-A3B

    Alibabaopen

    70.6%
  45. 45
    Phi-4

    Microsoftopen

    70.4%
  46. 46
    Pixtral Large

    Mistralclosed

    70.1%
  47. 47
    Claude 3 Opus

    Anthropicclosed

    69.6%
  48. 48
    GPT-4 Turbo

    OpenAIclosed

    69.4%
  49. 49
    Ministral 3 14B

    Mistralopen

    69.3%
  50. 50
    Mistral Small 3.2

    Mistralopen

    69.1%
  51. 51
    Qwen2.5 32B Instruct

    Alibabaopen

    69.0%
  52. 52
    Llama-3.1-Nemotron-70B Instruct

    NVIDIAopen

    69.0%
  53. 53
    Llama 3.3 70B

    Metaopen

    68.9%
  54. 54
    Sonar

    Perplexityclosed

    68.9%
  55. 55
    Mistral Medium 3.1

    Mistralclosed

    68.3%
  56. 56
    Gemma 3 27B

    Googleopen

    67.5%
  57. 57
    Reka Flash 3

    Rekaopen

    66.9%
  58. 58
    Llama 3.1 70B

    Metaopen

    66.4%
  59. 59
    GPT-4.1 Nano

    OpenAIclosed

    65.7%
  60. 60
    GPT-4o Mini

    OpenAIclosed

    64.8%
  61. 61
    Ministral 3 8B

    Mistralopen

    64.2%
  62. 62
    Qwen2.5 14B Instruct

    Alibabaopen

    63.7%
  63. 63
    Claude Haiku 3.5

    Anthropicclosed

    63.4%
  64. 64
    Gemma 3 12B

    Googleopen

    60.6%
  65. 65
    Claude 3 Sonnet

    Anthropicclosed

    57.9%
  66. 66
    Jamba 1.5 Large

    AI21open

    57.2%
  67. 67
    Gemini 1.5 Flash-8B

    Googleclosed

    56.9%
  68. 68
    Qwen2.5 7B

    Alibabaopen

    56.3%
  69. 69
    GPT-4

    OpenAIclosed

    56.2%
  70. 70
    DeepSeek-R1 Distill Llama 8B

    DeepSeekopen

    54.3%
  71. 71
    Mixtral 8x22B

    Mistralopen

    53.7%
  72. 72
    Mistral Small

    Mistralclosed

    52.9%
  73. 73
    Phi-4-mini

    Microsoftopen

    52.8%
  74. 74
    Ministral 3 3B

    Mistralopen

    52.4%
  75. 75
    Mistral Large

    Mistralclosed

    51.5%
  76. 76
    OLMo 2 32B

    Ai2open

    51.1%
  77. 77
    Claude 2.1

    Anthropicclosed

    49.5%
  78. 78
    Claude 2

    Anthropicclosed

    48.6%
  79. 79
    Phi-4-multimodal

    Microsoftopen

    48.5%
  80. 80
    Llama 3.1 8B

    Metaopen

    48.3%
  81. 81
    GPT-3.5 Turbo

    OpenAIclosed

    46.2%
  82. 82
    Granite 4.0 Micro

    IBMopen

    44.7%
  83. 83
    Gemma 3 4B

    Googleopen

    43.6%
  84. 84
    Phi-3 Mini 3.8B

    Microsoftopen

    43.5%
  85. 85
    Gemini 1.0 Pro

    Googleclosed

    43.1%
  86. 86
    DeepSeek Coder V2 Lite

    DeepSeekopen

    42.9%
  87. 87
    DBRX Instruct

    Databricksopen

    39.7%
  88. 88
    Mixtral 8x7B

    Mistralopen

    38.7%

Back to all benchmarks