II.
TestSet overview
Reference · livetest-set:gpqa-diamond-2024
GPQA Diamond — 2024 release overview
The frozen Diamond split is the standard reported number for vendor announcements (Anthropic, OpenAI, Google) when citing "GPQA Diamond" scores.
Attributes
displayName
GPQA Diamond — 2024 release
benchmarkId
caseCount
198
releasedAt
2023-11-29
composition
The "diamond" subset of GPQA — 198 graduate-level questions in
biology, chemistry, and physics written and validated by domain
experts. Diamond is the hardest tier; in-domain experts (PhD
students) achieve ~65% accuracy, while non-expert humans with
web access score ~34%.
homepageUrl
description
The frozen Diamond split is the standard reported number for
vendor announcements (Anthropic, OpenAI, Google) when citing
"GPQA Diamond" scores.
Outgoing edges
None.
Incoming edges
uses_test_set8
- eval-run:gpqa-diamond.claude-opus-4-5.2025-09·EvalRun
- eval-run:gpqa.deepseek-r1.2025-01·EvalRun
- eval-run:gpqa-diamond.gemini-2-5-pro.2025-06·EvalRun
- eval-run:gpqa-diamond.gemini-3-1-pro.2026-02-19·EvalRun
- eval-run:gpqa-diamond.gemini-3-pro.2025-11-18·EvalRun
- eval-run:gpqa-diamond.gpt-5.2025-08·EvalRun
- eval-run:gpqa-diamond.gpt-5-4.2026-03-17·EvalRun
- eval-run:gpqa-diamond.gpt-5-4-mini.2026-03-17·EvalRun