LLMcompare

Search

Search for a command to run...

Coding

BigCodeBench

Realistic function-level coding benchmark with 1,140 tasks across 7 languages; pass@1 depends on the release and prompting.

How models are tested

Scores are published numbers from the sources below. They are only comparable when the evaluation version, tools, agent harness, and sampling setup match. Missing scores are omitted rather than treated as zero.

Category
Coding
Metric
Percent
Direction
Higher is better
Coverage
45 / 318Models in this catalog with a published score
Updated
Catalog snapshot date
Source
bigcode-bench.github.io/

Scores

  1. 1
    GPT-4o (May 2024)

    OpenAIclosed

    51.1%
  2. 2
    DeepSeek V3

    DeepSeekopen

    50.0%
  3. 3
    Llama 4 Maverick

    Metaopen

    49.7%
  4. 4
    Qwen2.5-Coder 32B

    Alibabaopen

    49.0%
  5. 5
    GPT-4.1 Mini

    OpenAIclosed

    48.9%
  6. 6
    GPT-4 Turbo

    OpenAIclosed

    48.2%
  7. 7
    DeepSeek Coder V2

    DeepSeekopen

    48.2%
  8. 8
    Llama 3.3 70B

    Metaopen

    46.9%
  9. 9
    Claude 3.5 Sonnet

    Anthropicclosed

    46.8%
  10. 10
    GPT-4o Mini

    OpenAIclosed

    46.1%
  11. 11
    Claude Haiku 3.5

    Anthropicclosed

    46.1%
  12. 12
    Llama 3.1 70B

    Metaopen

    46.1%
  13. 13
    Qwen2.5 72B

    Alibabaopen

    45.8%
  14. 14
    Phi-4

    Microsoftopen

    45.5%
  15. 15
    Claude 3 Opus

    Anthropicclosed

    45.5%
  16. 16
    Qwen2.5 32B Instruct

    Alibabaopen

    45.0%
  17. 17
    Claude 3.5 Sonnet (Oct)

    Anthropicclosed

    44.6%
  18. 18
    DeepSeek-R1-Distill-Qwen-32B

    DeepSeekopen

    43.9%
  19. 19
    Llama 3 70B

    Metaopen

    43.6%
  20. 20
    Gemma 2 27B

    Googleopen

    42.8%
  21. 21
    Claude 3 Sonnet

    Anthropicclosed

    42.7%
  22. 22
    Codestral 22B

    Mistralopen

    41.8%
  23. 23
    Code Llama 70B Instruct

    Metaopen

    40.7%
  24. 24
    Mixtral 8x22B

    Mistralopen

    40.6%
  25. 25
    Qwen2.5 14B Instruct

    Alibabaopen

    39.8%
  26. 26
    Claude 3 Haiku

    Anthropicclosed

    39.4%
  27. 27
    Llama-3.1-Nemotron-70B Instruct

    NVIDIAopen

    38.7%
  28. 28
    DeepSeek-R1 Distill Qwen 14B

    DeepSeekopen

    38.1%
  29. 29
    Yi-Large

    01.AIclosed

    37.7%
  30. 30
    Phi-3 Medium 14B

    Microsoftopen

    37.6%
  31. 31
    Qwen2.5 7B

    Alibabaopen

    37.6%
  32. 32
    DeepSeek Coder V2 Lite

    DeepSeekopen

    36.8%
  33. 33
    DeepSeek-R1-Distill-Llama-70B

    DeepSeekopen

    35.3%
  34. 34
    Gemma 2 9B

    Googleopen

    34.7%
  35. 35
    Yi-1.5 34B Chat

    01.AIopen

    33.9%
  36. 36
    Command R (08-2024)

    Cohereclosed

    33.8%
  37. 37
    Llama 3.1 8B

    Metaopen

    32.8%
  38. 38
    Llama 3 8B

    Metaopen

    31.9%
  39. 39
    Mistral Large

    Mistralclosed

    30.0%
  40. 40
    Phi-3 Mini 3.8B

    Microsoftopen

    29.6%
  41. 41
    Code Llama 34B Instruct

    Metaopen

    29.0%
  42. 42
    Llama 3.2 3B

    Metaopen

    23.4%
  43. 43
    Mistral 7B

    Mistralopen

    19.5%
  44. 44
    DeepSeek-R1 Distill Llama 8B

    DeepSeekopen

    10.6%
  45. 45
    Llama 3.2 1B

    Metaopen

    8.2%

Back to all benchmarks