About

What ChessQA measures

ChessQA is a 3,500-item benchmark from CSSLab at the University of Toronto (arXiv:2510.23948) that probes LLM chess understanding across five categories of ascending abstraction: Structural (can the model read a board at all — piece placement, legal moves, checks), Motifs (recognizing pins, forks, skewers, batteries), Short Tactics (finding the best move in puzzle positions), Position Judgement(classifying a position's engine evaluation into five buckets), and Semantic (multiple-choice questions built from human game commentary). Every question is grounded in a concrete position, so answers are mechanically checkable.

What this smoke campaign was

In July 2026 we ran sixteen frontier-model configurations (fourteen distinct fleet configs plus two reused baselines) against a 50-task sample — exactly one position per task type, spanning all five categories. The point was to validate the harness, measure per-model cost and trace fidelity, and surface behavior worth studying before committing to the full 3,500-task run. That is why this site is a behavior browser, not a leaderboard: with one position per task type, category numbers are anecdotes. Counts ("23/24") are shown everywhere instead of percentages.

How answers were scored

Answers are extracted from the model's final FINAL ANSWER:line and scored by exact match (set match for multi-answer questions), following the paper's protocol. Outcomes shown here collapse the harness's error taxonomy: correct, wrong, illegal move (a wrong single-move answer that is not even legal in the position — verified with python-chess at export time), ran out of tokens(the model burned its entire 32K thinking budget without producing an answer), and format error (no parseable answer). The legality check is the first mechanical trace/answer verification from our Phase 3 roadmap, shipped early.

Thinking traces and fidelity

The centerpiece of this site is the unedited reasoning stream behind each answer. Providers differ in what they return: full_text is the raw, unedited chain of thought (Gemini 3.x and the open-weight reasoners return this); summary is a provider-generated summary of reasoning the provider keeps private (OpenAI models, and Anthropic models from the adaptive-thinking generation onward — Claude 5 family and Opus ≥ 4.7 — never expose raw CoT, so their traces here are provider-summarized); plain is reasoning the model wrote directly into its visible output; none means no trace was returned. Each trace drawer is labeled with its fidelity tier, and every model page records the exact reasoning payload sent.

Roadmap

Next: the full 3,500-task run with proper uncertainty quantification, then reasoning-trace root-cause analysis — mechanically verifying every calculated line in the thinking traces with python-chess (illegal-line rates, break depth, phantom pieces) plus a failure-mode taxonomy calibrated against hand labels by a FIDE Master. This site will grow to browse those results.

Attribution

The ChessQA benchmark is by CSSLab, University of Toronto — upstream at github.com/CSSLab/chessqa-benchmark(MIT license). Our harness fork, results, and the exporter that produced this site's data live at github.com/Ellipsoul/chessqa-benchmark. All costs shown are real measured API spend from the runs, as reported by the gateway per call.