A first comparison suggested that a development version of CUBE Backgammon chose better moves than the version it was tested against. We asked Open Sage, a separate backgammon engine, to evaluate the same decisions. It did not confirm the improvement, so we set the proposed change aside.
We later found an error in the code that sent positions to Open Sage: on half the sample, it supplied the wrong player to roll next. The engine was evaluating a different situation from the one we intended to compare.
The checkers are only part of a position
A screenshot tells you where the checkers are. It may not tell you who rolls next. That missing detail can change a position’s value dramatically, even when the board is perfectly symmetrical.
Consider this deliberately small example. Each player has borne off 14 checkers and has one left on their own 4-point. Both have the same pip count: four pips from finishing. We play this example without the doubling cube, and neither player can lose a gammon.
Who rolls next?
Ivory and Red each have one checker four pips from the exit.
Read the position as text
Ivory, point: number of checkers. 4: 1. Red: 21: 1. On the bar: Ivory 0, Red 0. Borne off: Ivory 14, Red 14.
Ivory rolls first.
Ivory wins94.8%
Red wins5.2%
Exact count: 307/324 for the first player, 17/324 for the other. The board stays the same.
Whoever rolls first has a chance of about 94.8% to win. The other player has about 5.2%. Switching the player to roll changes the answer by almost 90 percentage points without moving a checker.
Why the first player is so heavily favoured
Only two ordered rolls fail to bear off a lone checker from the 4-point immediately: 1–2 and 2–1. Every other roll succeeds. Even 1–1 succeeds, because a double gives four moves.
The first player therefore wins immediately on 34 of the 36 rolls. On either miss, the checker reaches the 1-point. The opponent then wins on 34 rolls too; if the opponent also misses, the first player finishes on any subsequent roll.
So the first player’s winning chance is 34/36 plus (2/36 × 2/36), or 307/324. The remaining 17/324 belongs to the other player. This is an exact count of dice possibilities, not a neural-network estimate.
We constructed this teaching example for the article; it does not come from the benchmark. It isolates the effect of whose turn it is, with winning chances we can calculate exactly.
The question our benchmark was actually asking
Our comparison took the board after a candidate move and asked an independent engine to evaluate what came next. The wrapper around that engine always said the opponent was about to roll.
That was correct when the move had just been made by the player our code called “you”. But the sample also contained decisions made by the other player. After those moves, “you” should have been next. The wrapper supplied the wrong side to roll on half the sample. Handling of finished games had a related player-perspective error.
This was more complicated than a minus sign accidentally turning wins into losses. Half the evaluations answered a different question. Combining them with the valid evaluations made real differences harder to see and undermined the comparison.
A simple race made the problem visible
Results from simple races sent us back to the code. In a complicated position, two engines can disagree for reasons that take time to understand. These races were easier to check, and tracing their results exposed our mistake: we were telling the reference engine the wrong player was about to roll.
After correcting the player to roll and the handling of finished games, Open Sage also rated the new version above the old one on that set of decisions. The initial test and the independent check now agreed on which version did better.
That did not mean we had discovered a guaranteed upgrade over everything released afterwards. A later comparison against a newer shipped baseline did not establish an additional gain. The correction tells us what the earlier experiment meant; it is not a new strength claim about today’s app.
The referee needs a test too
To compare two versions of CUBE Backgammon using an independent engine, we must transmit each position accurately. That includes the player to roll, the player whose chances are being measured and whether the game has already ended.
A small position such as the race above lets us check that transmission. Knowing each side’s exact winning chance for either player to roll helps expose a player-perspective error without analysing a whole match.
Our experiment with a higher training score and worse move choices had a different problem: we were counting how often CUBE Backgammon agreed with GNU Backgammon. Here, we were sending the checking engine incorrect information about the game.
For anyone reading an engine-strength claim, it is worth asking how the comparison was made and checked. A recognisable reference engine gives a test credibility, but it cannot repair a position described incorrectly.
This account follows the corrected development record, including the retraction of the original benchmark conclusion. We credit Open Sage and GNU Backgammon, the independent reference projects involved. The example board and exact dice calculation were constructed for this article and checked separately against CUBE Backgammon’s legal-move rules.