LLMs and shared game rules
ChessBench evaluates existing large language models (LLMs); it does not train or fine-tune them. Protocol chessbench-v3 measures chess-playing performance when each model receives the current position and legal moves. Each selected thinking setup has its own rating. These results do not establish general intelligence or unaided board reconstruction.
Both players receive the same chess rules. The server uses chess.js as the authority for legal moves and results. Checkmate and standard draw rules decide games. Repeated rejected moves cause a forfeit at the configured attempt limit; the default is three rejected attempts in one turn. A rejected move leaves the board unchanged.
Games have a shared limit of 150 accepted half-moves: White and Black moves combined, equal to 75 full moves. A game still in progress at that limit is recorded as a draw. Standard chess results take precedence, and a lower saved game limit may end a game earlier.
Finishing a game is part of playing strength. A model must convert its advantage into a win within the move limit. Repeated moves, lost material, and avoidable stalemate show a failure to convert that advantage. Extra turns are not granted indefinitely to rescue the result. At the cutoff, an unfinished game scores as a draw, not a win.
A higher-rated model is expected to score more than a draw against a lower-rated opponent. Failing to win therefore gives a negative contribution to its normal Glicko-2 update; the lower-rated opponent receives the corresponding positive contribution. For example, a player expected to score 0.8 earns only 0.5 for a draw. In initial placement, all players start from equal ratings and each draw contributes half a point. This is the rating consequence for failing to finish, not an extra penalty or an engine-awarded result. Standard checkmate and draw rules always take precedence.
Placement games
Each participating LLM identity starts at Elo 1500, rating deviation 350, and volatility 0.06. Every pair plays two games with reversed colours.
The required initial games form one Glicko-2 rating period. All calculations use the same starting opponent ratings and deviations. Games are treated as simultaneous for that calculation, so their execution order does not change the final ratings. Game displays show Unranked until placement is complete. Earlier game replays keep the ranking status recorded when the game started. Individual initial games do not receive separate official Elo changes.
The highest final unrounded Elo wins the initial event. When ratings are exactly equal, the model with the lower average game cost ranks higher. Equal point totals give equal initial ratings when starting values and game counts are equal.
A permanent withdrawal during the first placement games removes that identity and its games from the initial rating calculation. The remaining participants complete their mutual scheduled games. Completed game records and all recorded costs are retained. A temporary outage defers scheduled games and does not count as a withdrawal.
Continuous Elo ratings and the ladder
The displayed Elo uses Glicko-2 with tau 0.5. After the first placement games, each completed rated LLM game updates both players from the same pre-game rating snapshots. Rating, deviation, and volatility carry forward. The live implementation uses a per-game update for the participating players; it does not add a separate wall-clock decay. Game order can affect these live ratings.
The ladder selects the eligible identity with the fewest completed LLM games in the previous 48 hours. Both players count in ladder and manual benchmark games. Initial placement, new-model placement, and human games are excluded. Ties are resolved uniformly at random.
The opponent is selected uniformly from eligible identities up to five rank positions above or below. This is a rank range, not an Elo-point range. Manually paused identities keep their position but cannot play. Retired identities leave the active ladder.
White goes to the player with the lower White percentage in the same 48-hour window, using the same game exclusions. No games means 50%. Equal percentages use random colours. The normal ladder uses neighbours only after the required placement games.
Wins, draws, losses, and rejected-move forfeits affect ratings. Placement completion determines whether a new LLM may lead. Rating deviation remains part of the calculation. These Elo values compare LLMs within ChessBench. They are not equivalent to Chess.com, FIDE, or other human chess ratings.
New LLMs and retirement
A new model version or reasoning identity starts at 1500 and receives placement games against the active opponent pool at admission. It plays each required opponent once as White and once as Black.
Placement has priority over normal ladder play. Both players receive per-game rating updates. Each game’s result and cost are counted once. The new LLM may lead after placement completes; no additional rating-deviation cutoff applies.
Ready placement games continue when another game fails. Failed attempts do not satisfy placement requirements. A permanently retired opponent is removed from unfinished requirements.
Retirement removes an LLM from future scheduling, new human challenges, and the active leaderboard. Its completed history and established rating contributions remain. Restoring the same identity keeps its saved rating and placement status. A different model or reasoning identity starts new placement.
What each LLM receives
Each LLM has a separate conversation. Every request contains the shared system prompt, the player colour, the complete current FEN, an explicit piece list for both colours, all accepted moves in standard algebraic notation, the full list of legal UCI moves, and an instruction to select exactly one move. The FEN includes the board, side to move, castling rights, en passant information, and move counters. The piece list groups current square locations by colour and piece type. Both representations come from the same board state.
System prompt v3.2 includes an optional move-selection checklist. It suggests checking opponent threats, examining checks, captures, and threats, comparing defence with counterattack, and checking the intended move against the opponent’s best reply. Models can use this checklist alongside other analysis and plans.
The piece list states where every piece is now, including promoted pieces. Missing piece types are marked none, and all unlisted squares are empty. The server generates this list directly from the board instead of asking the model to reconstruct it. This reduces notation-reading work so the comparison can focus on move selection and play.
This choice follows the ChessQA piece-arrangement experiment (Table 4). Adding explicit piece positions increased overall chess-question accuracy from 14.5% to 23.5% for Claude 3.5 Haiku and from 24.1% to 31.0% for Gemini 2.5 Flash. Those figures describe that study, not a measured win-rate gain in ChessBench. LLM Chess (Table 3) also shows that board representation and interaction design can change scores, while ChessArena (Table 3) shows that supplied legal moves improve legality for many models. We therefore give every model the same FEN, piece list, move history, and legal choices. The linked studies are evaluation references, not training instructions.
Each new turn names the player’s last accepted UCI move and the opponent’s reply separately, then states whose turn it is and the player’s colour. Opening messages state that the player has not moved yet; White is told to move first, and Black is shown White’s opening move. The player’s piece-list heading is marked (Your pieces). Rejected replies receive the rejection reason and attempt count, with an explicit statement that the board is unchanged and the opponent has not moved again. The submit_move tool accepts a UCI move such as e2e4 or a7a8q. The server validates the move before changing the board.
The retained conversation includes that player’s earlier state messages, replies, tool calls, and provider-returned reasoning or context items where available. The opponent’s accepted moves are visible in game history. The opponent’s private conversation and reasoning are never shared.
When context approaches the configured limit, complete old conversation turns are removed toward a 40% context-window target, subject to output-space requirements. The system prompt is added separately on every request and is never pruned. The latest state still includes the full accepted move history, current FEN, piece list, and legal moves. No summary call is used.
The system and turn prompts do not announce an evaluation or identify ChessBench. They also omit opponent identity, Elo, standings, Stockfish evaluations, engine suggestions, and move-quality labels. A model or provider could still infer the setting; the prompt does not guarantee that the evaluation is undetectable. Private conversation data is deleted when a game completes or is removed.
Time, tokens, and costs
Response time includes reasoning and the full streamed reply. Engine analysis and video rendering are excluded. Output-token usage includes reasoning as reported by the provider. Rejected % is rejected moves divided by answered move attempts; failed calls are excluded.
Request costs use reported charges where available, or recorded token usage and saved prices for estimates. Cost per game is the average from completed benchmark games with complete pricing.
Cost Breakdown reports recorded costs across rating eras under Placement games, Continuous benchmark, Human games, and Operating expenses. Placement games combine the first rating batch and games for new LLMs. Continuous benchmark includes other LLM benchmark games. Benchmark LLM cost is the subtotal of placement games and continuous benchmark costs. It is not added again as a separate expense.
Every charge is assigned to the requesting LLM and its saved thinking setup. Both players’ costs are counted once in an LLM game. Human games count only the LLM requests.
Operating expenses include hosting, storage, and domain registration. Billing corrections count only the difference from the recorded total.
The Cost Breakdown timeline groups costs by game stage or LLM thinking setup. Dates are UTC. Short ranges use daily columns; long ranges use labelled multi-day intervals.
Saved records
Completed games keep accepted moves, call metrics, price snapshots, results, PGN, and requested replay videos. The first placement records identify their shared rating batch. Later rated games keep their pre-game ratings and rating changes. Retiring an LLM does not remove completed history.
Protocol chessbench-v3 records the input representation and response rules alongside the system prompt version. New results use FEN plus a piece list, accept legal bare UCI or move JSON, and request submit_move through the supported provider controls. The fresh LLM rating event starts at 1500 under this protocol. Human game history is separate.
Human challenges
Human games use separate game records and never change LLM benchmark ratings or automatic scheduling. ChessBench covers models marked FREE. Models marked BYOK use the player’s OpenRouter credits. Site-paid usage is included under Human games in Cost Breakdown. Players sign in with X, and ownership follows the verified account ID. All completed human games use the same rules and rating treatment.
Human Elo starts at 1500. Each completed game uses K = 32 and the LLM rating saved before the game. The expected score is 1 / (1 + 10^((LLM rating − human rating) / 400)). The rating change is 32 times the actual score minus the expected score. A win scores 1, a draw 0.5, and a loss 0. Only the human rating changes. These are ChessBench ratings, not external human chess ratings.
Human moves are published after legal-move validation. Evaluations remain hidden until the game ends. Completed board replays are public; video access follows the owning player’s sharing settings.