LLMcompare

Search

Search for a command to run...

Tool use & function calling

τ³-Banking

97 fintech customer-support tasks (disputes, account freezes, provisional credits, product changes) where an agent must navigate roughly 700 interconnected policy documents (~195k tokens) and execute multi-step tool calls. Graded pass@1 against backend database state rather than conversational quality. This is an agent-system result and is distinct from τ²-bench Telecom and from earlier τ-bench releases.

How models are tested

Scores are published numbers from the sources below. They are only comparable when the evaluation version, tools, agent harness, and sampling setup match. Missing scores are omitted rather than treated as zero.

Category
Tool use & function calling
Metric
Percent
Direction
Higher is better
Coverage
101 / 318Models in this catalog with a published score
Updated
Catalog snapshot date
Source
artificialanalysis.ai/evaluations/tau3-banking

Scores

  1. 1
    Grok 4.6

    xAIclosed

    50.7%
  2. 2
    Muse Spark 1.3

    Metaclosed

    50.5%
  3. 3
    GLM-5.3

    Zhipu AIopen

    50.3%
  4. 4
    Qwen3.8 27B

    Alibabaopen

    48.0%
  5. 5
    Qwen3.8 Max

    Alibabaopen

    47.8%
  6. 6
    Claude Fable 5.1

    Anthropicclosed

    47.2%
  7. 7
    GLM-5.3 Flash

    Zhipu AIopen

    47.2%
  8. 8
    Kimi K3

    Moonshotopen

    46.0%
  9. 9
    Qwen3.8 Flash Next

    Alibabaopen

    45.4%
  10. 10
    Gemini 3.8 Flash

    Googleclosed

    44.9%
  11. 11
    GPT-5.6 Sol

    OpenAIclosed

    44.3%
  12. 12
    Claude Opus 5

    Anthropicclosed

    42.1%
  13. 13
    Grok 4.5

    xAIclosed

    42.1%
  14. 14
    GPT-6 Astra

    OpenAIclosed

    41.4%
  15. 15
    Atria Dawn Preview

    Shanghai AI Labopen

    41.2%
  16. 16
    GPT-5.6 Terra

    OpenAIclosed

    40.2%
  17. 17
    GPT-5.4

    OpenAIclosed

    39.6%
  18. 18
    DeepSeek V4 Pro 0813

    DeepSeekclosed

    39.6%
  19. 19
    DeepSeek V4 Flash 0731

    DeepSeekopen

    39.4%
  20. 20
    GPT-5.5

    OpenAIclosed

    39.0%
  21. 21
    Claude Fable 5

    Anthropicclosed

    38.1%
  22. 22
    Claude Sonnet 5

    Anthropicclosed

    37.3%
  23. 23
    Muse Spark 1.2

    Metaclosed

    34.8%
  24. 24
    GLM-5.2

    Zhipu AIopen

    34.6%
  25. 25
    Claude Opus 4.7

    Anthropicclosed

    34.6%
  26. 26
    Claude Sonnet 4.6

    Anthropicclosed

    34.4%
  27. 27
    Claude Opus 4.8

    Anthropicclosed

    34.2%
  28. 28
    K2 Horizon 375B A23B

    MBZUAI Institute of Foundation Modelsopen

    34.2%
  29. 29
    Gemini 3.7 Flash

    Googleclosed

    32.8%
  30. 30
    Gemini 3.5 Flash

    Googleclosed

    32.2%
  31. 31
    Muse Spark 1.1

    Metaclosed

    31.8%
  32. 32
    GPT-5.6 Luna

    OpenAIclosed

    31.1%
  33. 33
    DeepSeek V4 Flash

    DeepSeekopen

    30.9%
  34. 34
    DeepSeek V4 Pro

    DeepSeekopen

    30.1%
  35. 35
    Gemini 3.6 Flash

    Googleclosed

    29.9%
  36. 36
    Inkling

    Thinking Machinesopen

    29.1%
  37. 37
    GPT-5.4 Nano

    OpenAIclosed

    27.4%
  38. 38
    Ling-3.0 Flash

    InclusionAIopen

    27.2%
  39. 39
    GPT-5.4 Mini

    OpenAIclosed

    25.6%
  40. 40
    Muse Glimmer

    Metaopen

    23.5%
  41. 41
    Solar Pro 4

    Upstageclosed

    23.3%
  42. 42
    Kimi K2.6

    Moonshotclosed

    23.3%
  43. 43
    Hy3

    Tencentopen

    22.9%
  44. 44
    GPT-5

    OpenAIclosed

    22.1%
  45. 45
    Gemini 3 Flash

    Googleclosed

    20.8%
  46. 46
    Qwen3.6 Plus

    Alibabaclosed

    20.8%
  47. 47
    Ling-3.0 Tiny

    InclusionAIopen

    20.8%
  48. 48
    Kimi K2.7 Code

    Moonshotopen

    20.2%
  49. 49
    Inkling-Small

    Thinking Machinesopen

    18.8%
  50. 50
    Ring-2.6-1T

    InclusionAIopen

    17.9%
  51. 51
    Gemini 3.5 Flash-Lite

    Googleclosed

    17.5%
  52. 52
    Qwen3.7 Plus

    Alibabaclosed

    17.5%
  53. 53
    Qwen3.6 27B

    Alibabaopen

    16.7%
  54. 54
    GPT-5 Mini

    OpenAIclosed

    15.5%
  55. 55
    MiniMax M3

    MiniMaxopen

    15.3%
  56. 56
    Mistral Medium 3.5

    Mistralopen

    15.1%
  57. 57
    Gemma 4 31B

    Googleopen

    14.8%
  58. 58
    Granite 4.2 30B

    IBMopen

    14.4%
  59. 59
    Kimi K2.5

    Moonshotopen

    14.2%
  60. 60
    Nemotron 3 Ultra

    NVIDIAopen

    14.2%
  61. 61
    GLM-5.1

    Zhipu AIopen

    13.6%
  62. 62
    Qwen3.5 397B-A17B

    Alibabaopen

    13.4%
  63. 63
    LongCat-2.0

    Meituanopen

    13.2%
  64. 64
    gpt-oss-120B

    OpenAIopen

    12.8%
  65. 65
    Grok 4.3

    xAIclosed

    12.4%
  66. 66
    GLM-4.7

    Zhipu AIopen

    12.2%
  67. 67
    Qwen3.7 Max

    Alibabaclosed

    11.8%
  68. 68
    Devstral 2

    Mistralopen

    10.5%
  69. 69
    MiMo-V2.5-Pro

    Xiaomiopen

    9.9%
  70. 70
    MiniMax M2.7

    MiniMaxclosed

    9.9%
  71. 71
    Gemini 2.5 Pro

    Googleclosed

    9.7%
  72. 72
    Gemini 3.1 Flash-Lite

    Googleclosed

    9.7%
  73. 73
    Mercury 2

    Inception Labsclosed

    9.5%
  74. 74
    Qwen3.6 35B A3B

    Alibabaopen

    9.3%
  75. 75
    Nemotron 3.5 Lightning

    NVIDIAopen

    8.9%
  76. 76
    MiMo-V2.5

    Xiaomiopen

    8.7%
  77. 77
    Solar Pro 3

    Upstageopen

    8.7%
  78. 78
    Mistral Medium 3.1

    Mistralclosed

    8.2%
  79. 79
    Granite 4.2 8B

    IBMopen

    7.6%
  80. 80
    gpt-oss-20B

    OpenAIopen

    7.0%
  81. 81
    Ministral 3 14B

    Mistralopen

    6.6%
  82. 82
    DeepSeek R1

    DeepSeekopen

    6.4%
  83. 83
    Mistral Small 3.2

    Mistralopen

    6.2%
  84. 84
    Qwen3-Next 80B-A3B

    Alibabaopen

    6.2%
  85. 85
    Mistral Large 3

    Mistralopen

    5.8%
  86. 86
    Trinity Large Thinking

    Arcee AIopen

    5.8%
  87. 87
    Granite 4.2 3B

    IBMopen

    5.6%
  88. 88
    GPT-4.1 Mini

    OpenAIclosed

    5.4%
  89. 89
    Qwen3 32B

    Alibabaopen

    5.4%
  90. 90
    Mistral Small 4

    Mistralopen

    4.9%
  91. 91
    Ministral 3 3B

    Mistralopen

    4.7%
  92. 92
    DeepSeek V3

    DeepSeekopen

    4.7%
  93. 93
    Llama 4 Maverick

    Metaopen

    3.7%
  94. 94
    Ministral 3 8B

    Mistralopen

    3.7%
  95. 95
    GPT-4.1 Nano

    OpenAIclosed

    3.5%
  96. 96
    Llama 4 Scout

    Metaopen

    3.3%
  97. 97
    GPT-4o Mini

    OpenAIclosed

    2.9%
  98. 98
    Ling-2.6 Flash

    InclusionAIclosed

    2.9%
  99. 99
    Gemma 3 12B

    Googleopen

    0.8%
  100. 100
    Gemma 3 27B

    Googleopen

    0.8%
  101. 101
    Gemma 3 4B

    Googleopen

    0.4%

Back to all benchmarks