Skip to results
AI evaluation & trainingEvaluation libraryHUMAINE Leaderboard

HUMAINE Leaderboard

54 models ranked by 40,000+ people in the UK and US, on conversations of their own choosing, with results for 22 demographic groups.

At a glance

HUMAINE ranks models by human preference. A hierarchical Bayesian Bradley–Terry–Davidson model with demographic random effects keeps differences between groups instead of averaging them away, and reports uncertainty on every score and rank.

Pairwise comparisons
253,849
Models
54
Demographic groups
22
Ranking model
Hierarchical Bayesian BTD
Convergence
R-hat ≤ 1.02
Data updated
15 Jul 2026

Results

Choose a measure and a participant group to rank all 54 models.

Choose a measure
Overall

Which of two anonymous models participants preferred overall, after a conversation on a topic of their choice. Groups disagree noticeably on this ranking.

54models, All participants

Model results

54

Scroll horizontally to compare scores

HUMAINE Overall ranking for All participants
PositionCurrent position in this specific leaderboard (#1, #2, #3...). This is where the model sits right now, different from Expected Rank which shows their typical performance accounting for uncertainty.ModelScoreWinshare (BTD): expected total points in a round-robin vs. all other models (win = 1, tie = 0.5). With 54 models, max = 53, field average ≈ 26.5. Higher = better. Error bars show posterior credible intervals.Expected RankThe average ranking position this model would achieve across many possible tournaments. The ± value shows the posterior standard deviation, indicating how much the rank might vary due to statistical uncertainty. Lower values = more confident in the ranking.P(best)Probability this model would rank #1 if we repeated the evaluation many times. With 54 models competing, random chance would give each model 1.9% probability. Higher values indicate the model is genuinely superior to its competitors.Sample SizeNumber of evaluations for this model. Each evaluation is a head-to-head comparison where a participant judged this model against another model. More evaluations = more reliable rankings.
Top tier11 models — clearly stronger than the mid-pack (beats a typical model ≥60% of the time).
1
Gemini 3.1 Pro (preview)
Google
35.38 ± 0.52
1.3 ± 0.5
73.1%2,006
T22
Gemini 3.5 Flash
Google
34.81 ± 0.70
2.1 ± 1.0
25.1%1,302
T33
Gemini 3 Prodeprecated
Google
33.80 ± 0.47
4.0 ± 1.4
0.6%2,866
4
Claude Fable 5
Anthropic
33.52 ± 0.73
5.0 ± 2.2
0.9%1,329
5
Grok 4.20 Beta
xAI
33.32 ± 0.55
5.5 ± 1.9
<0.1%1,999
T46
Gemini 2.5 Pro
Google
33.23 ± 0.51
5.9 ± 2.0
0.1%2,498
7
Qwen 3.7 Max
Alibaba Cloud
33.03 ± 0.71
6.7 ± 2.5
<0.1%1,295
8
Mistral Large 3
Mistral AI
32.68 ± 0.44
8.0 ± 1.8
<0.1%3,366
T59
DeepSeek V4 Flash
DeepSeek
32.39 ± 0.65
9.1 ± 2.5
<0.1%1,519
10
Qwen3 235B A22B 2507
Alibaba Cloud
32.16 ± 0.47
10.1 ± 1.9
<0.1%3,022
11
GPT-5.2 Chat
OpenAI
32.12 ± 0.46
10.2 ± 1.8
<0.1%3,280
Mid-pack33 models — statistically close — within a coin-flip of one another (≈40–60% vs a typical model), so small rank gaps here aren't meaningful.
12
Claude Opus 4.7
Anthropic
31.76 ± 0.64
11.5 ± 2.3
<0.1%1,524
T713
DeepSeek V4 Pro
DeepSeek
31.05 ± 0.75
14.2 ± 2.9
<0.1%1,268
14
DeepSeek Chat V3 0324
DeepSeek
30.95 ± 0.52
14.4 ± 2.0
<0.1%2,377
15
Magistral Medium 2506deprecated
Mistral AI
30.88 ± 0.59
14.7 ± 2.3
<0.1%1,973
T816
GPT-5.4
OpenAI
30.40 ± 0.57
16.7 ± 2.6
<0.1%2,064
17
Claude Opus 4.6
Anthropic
30.16 ± 0.48
17.7 ± 2.4
<0.1%2,972
18
Kimi K2.5
Moonshot AI
30.14 ± 0.49
17.9 ± 2.4
<0.1%3,015
T919
GPT-5.5
OpenAI
29.71 ± 0.66
20.2 ± 3.5
<0.1%1,524
20
Grok 4deprecated
xAI
29.63 ± 0.57
20.6 ± 3.2
<0.1%2,180
21
DeepSeek V3.2
DeepSeek
29.41 ± 0.47
21.8 ± 2.8
<0.1%3,326
22
Gemini 2.5 Flash
Google
29.22 ± 0.54
22.8 ± 3.1
<0.1%2,319
T1023
Kimi K2.6
Moonshot AI
29.14 ± 0.76
23.1 ± 3.8
<0.1%1,291
24
Grok 3deprecated
xAI
29.00 ± 0.55
24.0 ± 2.9
<0.1%2,137
25
GLM 4.7
Z.ai
28.94 ± 0.50
24.4 ± 2.7
<0.1%2,939
26
DeepSeek R1 0528
DeepSeek
28.83 ± 0.55
24.9 ± 2.8
<0.1%2,323
27
GPT-5.6 Terra
OpenAI
28.70 ± 0.79
25.1 ± 3.5
<0.1%1,186
28
GPT-4.1
OpenAI
28.69 ± 0.55
25.6 ± 2.6
<0.1%2,297
T1129
Claude Opus 4.8
Anthropic
27.90 ± 0.76
28.1 ± 2.3
<0.1%1,283
T1230
Gemma 3 27B IT
Google
26.94 ± 0.57
30.3 ± 1.1
<0.1%2,176
31
Claude Sonnet 4.5
Anthropic
26.54 ± 0.45
31.1 ± 1.0
<0.1%3,390
T1332
Gemini 2.0 Flash 001deprecated
Google
26.16 ± 0.59
32.0 ± 1.3
<0.1%2,072
T1433
OLMo 3.1 32B Thinkdeprecated
Allen AI (Ai2)
25.55 ± 0.68
33.4 ± 1.6
<0.1%1,561
34
Claude Opus 4.5
Anthropic
25.08 ± 0.45
34.6 ± 1.1
<0.1%3,503
35
Claude Opus 4deprecated
Anthropic
25.06 ± 0.55
34.6 ± 1.3
<0.1%2,146
T1536
GPT-5.6 Luna
OpenAI
24.92 ± 0.84
34.8 ± 1.7
<0.1%1,139
T1637
Claude Sonnet 4deprecated
Anthropic
23.38 ± 0.56
37.5 ± 0.9
<0.1%2,153
38
GPT-5.6 Sol
OpenAI
23.25 ± 0.81
37.9 ± 1.5
<0.1%1,168
T1739
MiniMax M2.1
MiniMax
22.26 ± 0.47
39.8 ± 1.4
<0.1%3,025
40
o3
OpenAI
22.12 ± 0.55
40.2 ± 1.5
<0.1%2,071
41
GPT-5 mini
OpenAI
21.71 ± 0.50
41.5 ± 1.6
<0.1%2,384
T1842
GPT-5
OpenAI
21.59 ± 0.52
41.9 ± 1.6
<0.1%2,396
43
o1-minideprecated
OpenAI
21.40 ± 0.58
42.4 ± 1.8
<0.1%1,746
44
Llama 3.3 70B Instruct
Meta
21.14 ± 0.52
43.3 ± 1.6
<0.1%2,403
Bottom tier10 models — clearly weaker than the mid-pack (loses to a typical model ≥60% of the time).
45
Command A
Cohere
20.44 ± 0.49
45.3 ± 1.3
<0.1%2,463
46
Claude 3.7 Sonnetdeprecated
Anthropic
20.01 ± 0.52
46.4 ± 1.2
<0.1%2,162
47
Kimi K2
Moonshot AI
20.00 ± 0.50
46.5 ± 1.2
<0.1%2,326
T2048
o4-mini
OpenAI
19.67 ± 0.50
47.2 ± 1.1
<0.1%2,425
T2149
Llama 4 Maverick
Meta
18.44 ± 0.47
49.0 ± 0.3
<0.1%2,666
T2250
Mistral Nemo
Mistral AI
17.20 ± 0.45
50.0 ± 0.3
<0.1%2,807
T2351
o1
OpenAI
16.31 ± 0.45
50.9 ± 0.3
<0.1%2,730
T2452
GPT-4o
OpenAI
14.98 ± 0.41
52.0 ± 0.1
<0.1%2,882
T2553
o3-mini
OpenAI
13.28 ± 0.39
53.1 ± 0.3
<0.1%3,218
T2654
Command R7B 12 2024
Cohere
12.65 ± 0.36
53.9 ± 0.3
<0.1%3,582
Top tierclearly stronger than the mid-pack (beats a typical model ≥60% of the time).
1
Gemini 3.1 Pro (preview)
Google
Score
35.38
Exp. Rank
1.3
P(best)
73.1%
2
Gemini 3.5 Flash
Google
Score
34.81
Exp. Rank
2.1
P(best)
25.1%
3
Gemini 3 Prodeprecated
Google
Score
33.80
Exp. Rank
4.0
P(best)
0.6%
4
Claude Fable 5
Anthropic
Score
33.52
Exp. Rank
5.0
P(best)
0.9%
5
Grok 4.20 Beta
xAI
Score
33.32
Exp. Rank
5.5
P(best)
<0.1%
6
Gemini 2.5 Pro
Google
Score
33.23
Exp. Rank
5.9
P(best)
0.1%
7
Qwen 3.7 Max
Alibaba Cloud
Score
33.03
Exp. Rank
6.7
P(best)
<0.1%
8
Mistral Large 3
Mistral AI
Score
32.68
Exp. Rank
8.0
P(best)
<0.1%
9
DeepSeek V4 Flash
DeepSeek
Score
32.39
Exp. Rank
9.1
P(best)
<0.1%
10
Qwen3 235B A22B 2507
Alibaba Cloud
Score
32.16
Exp. Rank
10.1
P(best)
<0.1%
11
GPT-5.2 Chat
OpenAI
Score
32.12
Exp. Rank
10.2
P(best)
<0.1%
Mid-packstatistically close — within a coin-flip of one another (≈40–60% vs a typical model), so small rank gaps here aren't meaningful.
12
Claude Opus 4.7
Anthropic
Score
31.76
Exp. Rank
11.5
P(best)
<0.1%
13
DeepSeek V4 Pro
DeepSeek
Score
31.05
Exp. Rank
14.2
P(best)
<0.1%
14
DeepSeek Chat V3 0324
DeepSeek
Score
30.95
Exp. Rank
14.4
P(best)
<0.1%
15
Magistral Medium 2506deprecated
Mistral AI
Score
30.88
Exp. Rank
14.7
P(best)
<0.1%
16
GPT-5.4
OpenAI
Score
30.40
Exp. Rank
16.7
P(best)
<0.1%
17
Claude Opus 4.6
Anthropic
Score
30.16
Exp. Rank
17.7
P(best)
<0.1%
18
Kimi K2.5
Moonshot AI
Score
30.14
Exp. Rank
17.9
P(best)
<0.1%
19
GPT-5.5
OpenAI
Score
29.71
Exp. Rank
20.2
P(best)
<0.1%
20
Grok 4deprecated
xAI
Score
29.63
Exp. Rank
20.6
P(best)
<0.1%
21
DeepSeek V3.2
DeepSeek
Score
29.41
Exp. Rank
21.8
P(best)
<0.1%
22
Gemini 2.5 Flash
Google
Score
29.22
Exp. Rank
22.8
P(best)
<0.1%
23
Kimi K2.6
Moonshot AI
Score
29.14
Exp. Rank
23.1
P(best)
<0.1%
24
Grok 3deprecated
xAI
Score
29.00
Exp. Rank
24.0
P(best)
<0.1%
25
GLM 4.7
Z.ai
Score
28.94
Exp. Rank
24.4
P(best)
<0.1%
26
DeepSeek R1 0528
DeepSeek
Score
28.83
Exp. Rank
24.9
P(best)
<0.1%
27
GPT-5.6 Terra
OpenAI
Score
28.70
Exp. Rank
25.1
P(best)
<0.1%
28
GPT-4.1
OpenAI
Score
28.69
Exp. Rank
25.6
P(best)
<0.1%
29
Claude Opus 4.8
Anthropic
Score
27.90
Exp. Rank
28.1
P(best)
<0.1%
30
Gemma 3 27B IT
Google
Score
26.94
Exp. Rank
30.3
P(best)
<0.1%
31
Claude Sonnet 4.5
Anthropic
Score
26.54
Exp. Rank
31.1
P(best)
<0.1%
32
Gemini 2.0 Flash 001deprecated
Google
Score
26.16
Exp. Rank
32.0
P(best)
<0.1%
33
OLMo 3.1 32B Thinkdeprecated
Allen AI (Ai2)
Score
25.55
Exp. Rank
33.4
P(best)
<0.1%
34
Claude Opus 4.5
Anthropic
Score
25.08
Exp. Rank
34.6
P(best)
<0.1%
35
Claude Opus 4deprecated
Anthropic
Score
25.06
Exp. Rank
34.6
P(best)
<0.1%
36
GPT-5.6 Luna
OpenAI
Score
24.92
Exp. Rank
34.8
P(best)
<0.1%
37
Claude Sonnet 4deprecated
Anthropic
Score
23.38
Exp. Rank
37.5
P(best)
<0.1%
38
GPT-5.6 Sol
OpenAI
Score
23.25
Exp. Rank
37.9
P(best)
<0.1%
39
MiniMax M2.1
MiniMax
Score
22.26
Exp. Rank
39.8
P(best)
<0.1%
40
o3
OpenAI
Score
22.12
Exp. Rank
40.2
P(best)
<0.1%
41
GPT-5 mini
OpenAI
Score
21.71
Exp. Rank
41.5
P(best)
<0.1%
42
GPT-5
OpenAI
Score
21.59
Exp. Rank
41.9
P(best)
<0.1%
43
o1-minideprecated
OpenAI
Score
21.40
Exp. Rank
42.4
P(best)
<0.1%
44
Llama 3.3 70B Instruct
Meta
Score
21.14
Exp. Rank
43.3
P(best)
<0.1%
Bottom tierclearly weaker than the mid-pack (loses to a typical model ≥60% of the time).
45
Command A
Cohere
Score
20.44
Exp. Rank
45.3
P(best)
<0.1%
46
Claude 3.7 Sonnetdeprecated
Anthropic
Score
20.01
Exp. Rank
46.4
P(best)
<0.1%
47
Kimi K2
Moonshot AI
Score
20.00
Exp. Rank
46.5
P(best)
<0.1%
48
o4-mini
OpenAI
Score
19.67
Exp. Rank
47.2
P(best)
<0.1%
49
Llama 4 Maverick
Meta
Score
18.44
Exp. Rank
49.0
P(best)
<0.1%
50
Mistral Nemo
Mistral AI
Score
17.20
Exp. Rank
50.0
P(best)
<0.1%
51
o1
OpenAI
Score
16.31
Exp. Rank
50.9
P(best)
<0.1%
52
GPT-4o
OpenAI
Score
14.98
Exp. Rank
52.0
P(best)
<0.1%
53
o3-mini
OpenAI
Score
13.28
Exp. Rank
53.1
P(best)
<0.1%
54
Command R7B 12 2024
Cohere
Score
12.65
Exp. Rank
53.9
P(best)
<0.1%

What’s included

What the published leaderboard and evaluation dataset contain.

  1. 01

    Conversations

    Multi-turn transcripts with two anonymous models, on topics participants chose.

  2. 02

    Judgements

    A, B or tie on the overall winner and four dimensions.

  3. 03

    Participant profile

    Self-reported age, ethnic group, political affiliation and country.

  4. 04

    Conversation labels

    Task type, domain and 1–5 scores for complexity, goal achievement and engagement.

  5. 05

    Rankings

    Scores and rank ranges with 95% intervals, per group and measure.

How it was built

Snapshot taken 15 Oct 2026 from the public HUMAINE data, generated 15 Jul 2026.

PeopleAutomated

  1. Recruit

    People

    Participants join from Prolific's verified pool, balanced across 22 demographic groups in the UK and US.

    Captured
    • Self-reported age, ethnic group, political affiliation and country

  2. Pair models

    Automated

    A tournament design based on information gain chooses which two models each participant meets. Both stay anonymous.

    Captured
    • Model A and model B, hidden from the participant

  3. Converse

    PeopleAutomated

    The participant talks with both models about whatever matters to them, over several turns.

    Captured
    • Messages the participant types
    • Each model's replies

  4. Compare

    People

    The participant judges the two conversations side by side.

    Captured
    • A, B or tie on overall winner and four dimensions

  5. Rank

    Automated

    A hierarchical Bayesian Bradley–Terry–Davidson model, post-stratified to census data, turns votes into rankings.

    Captured
    • Scores and rank ranges with 95% intervals, per group and dimension

Run this on your own models

HUMAINE ran on 40,000+ Prolific participants in the UK and US, stratified by age, ethnicity and education.

Tell us the decisions you need to make. We’ll scope tasks, reviewers and grading with you.

Talk to the team

More from the library

HUMAINE HealthBenchmark, HealthcareAlignment LeaderboardBenchmark, AlignmentElicitation‑sensitive coding environmentsSample dataset, Coding