What does a setting cost? Difficulty measured by the AI

Audamir can be set up in dozens of ways: the size of the map, the shape of the tiles, the number of opponents, the strength of the artificial intelligence, the limit of dice per tile, fortresses, lakes, teams, fog of war. Every combination is a different game. And a player who wins wants to know what exactly they have won.

A win in a game for two on a small map is not the same as a win against seven on a huge one. A leaderboard that counts them the same is nonsense. A leaderboard that counts them by my estimate is only an opinion. That is why the score for a win is measured.

The principle: the AI in place of a person, one change at a time

The difficulty of a setting can be defined as how much the chance to win falls when the setting is changed. In place of a person, who would have to play thousands of games, the AI plays. We use the AI at two levels: level 2, Normal ("the average player"), and level 3, Strong ("the good player"). The opponents are played by the artificial intelligence according to the setting.

The procedure has three steps:

1. The base configuration. A medium map, hexagons, three opponents, a Strong AI. It is chosen so that the AI in the player's place has a reasonable chance (about 12% for level 2 and 37% for level 3). In a game that is too easy or too hard the wins would "saturate" at 0 or 100% and the differences could not be measured.
2. A change of a single setting. Every value of every setting is measured separately, always 2,400 games per level of the AI. One measurement is, for example, reserve=false@2.
3. Conversion to points. The share of wins is converted to odds (the logarithm of the ratio win : loss, log-odds) and the difference against the easiest value of the setting is multiplied by eight. The easiest value therefore always has 0 points. One point corresponds to a drop of 0.125 in log-odds, which at a chance of about 50% is roughly three percentage points. Differences below 0.12 are taken as noise and counted as zero.

The real output of the tool for a quick comparison, run today (tools/experiment.js): a defender bonus of +4 against a game without it.

A: {"defBonus":4}
  AI 2: 4.4 % (±1.6) of 600 games · a win takes 32.3 rounds on average
  AI 3: 18.3 % (±3.1) of 600 games · a win takes 24.5 rounds on average
B: {}
  AI 2: 13.9 % (±2.8) of 600 games · a win takes 36.3 rounds on average
  AI 3: 35.2 % (±3.8) of 600 games · a win takes 37.7 rounds on average

Difference (mean over the AI): Δ log-odds 1.07 → A is harder than B by ≈ 8.6 pts

The output of the tool is in English and uses a point as the decimal mark; the only change here is that the word "bot" of the output is replaced by the word "AI". A defender bonus of +4 is therefore roughly 8 to 9 points of difficulty. The calibration table records 8 points for it.

The same number after three days

Notice that these numbers (4.4%, 18.3%, 13.9%, 35.2%, 8.6 points) are exactly the same as the ones in the documentation written on 17 October, right on the first morning of the project. Today is the 20th, and in the meantime a lot has changed in the code.

It is not a coincidence and it is not a mistake. The measurement is deterministic: the seed of every game is derived from its sequence number and the name of the measurement. The same code and the same parameters give exactly the same numbers, regardless of the number of processor cores. From this follows a useful property: the tool calibrate.js --verify replays 18 control measurements (9 areas of the game times two levels of the AI, about two minutes) and says one of three sentences:

It is a test of balance. An ordinary test checks that a function returns the right value. This one checks that the game is as hard as it was yesterday. In ordinary software a similar trick is called a golden master or a snapshot test.

What the measurement admits

This is not an exact science, and the documentation says so openly. Three main limitations:

The points do not add up exactly. The score of a game is the sum of the points of the chosen settings, which assumes that their effects add up in log-odds. In reality they influence one another. A check with three combinations:

The score is therefore indicative, typically off by 15 to 40% of the difference from the base. It is good for comparing settings, not for predicting a chance.

The AI is not a person. It does not use hints and does not play "like a human". The clearest example: the Tactician bonus (the player places the dice themselves) is underrated for the AI, because the AI in the player's place places them in the same way as the opponents do, and a person knows more. Likewise Reroll: the AI spends it at once, a person saves it for an important attack.

Saturation. When the AI wins less than 2% or more than 98% of the games, the share is clipped and the points are underrated. Fog of war cannot be measured this way at all, since the AI sees the whole map, so it has a manual estimate of +3 points. In the table it is marked as an estimate, not as a measurement.

To this belongs one tax: every change of the AI changes all the measured shares of wins, because the AI is the opponent in every measurement. After a change to the AI everything has to be measured again (26 minutes).

One currency of difficulty for everything

Once a currency exists ("a point = a drop of 0.125 in log-odds"), anything can be priced in it. After the settings, the bonuses of the player, the traits of the bosses, the companions and whole campaign missions were measured in this way. Because of that, the design of the campaign is a matter of a table, not of feeling. A few rows from it (average points, a plus means that the bonus helps):

Design can be read from such a table. Armour is a weak, "safe" choice. Supplies grow almost linearly, and Plunder is strong right from the first level. And the table was useful negatively too: defensive bonuses lead to a stalemate, because the maps saturate, and that is why Armour is only for small stacks.

The AI as a malicious tester

The most instructive cases are those where the measurement showed that a rule is broken. The companions of the game have abilities, and one of them, Passage, allows moving dice from one of one's own tiles to another. In the first version everything but one die could be moved. The measurement: +19 points at the lowest level. The AI kept pulling stacks from the inside of the map to the front and attacked with a full stack, so in the second mission it won 97 to 100% of the games. Even moving two or three dice did not help (+13, +17).

After the limit to one die per move, the three levels came out at +5.7, +11.9 and +14.7 points, which is roughly the same as the Supplies bonus (+4.1, +9.6, +14.5 in another measurement, docs/TUNING.md §14.2). The rule could be balanced because it could be measured.

The opposite example: the weakness of a companion, "Cautious advance", was originally to read after a win only half of the dice move. For the AI it was devastating, −11 points (Brynja^[1] playing alone won 7 to 20% in some missions), because the AI builds on chain attacks with a full stack. Today it is mild (−3.5): a stack of at least eight dice leaves one extra die at home.

Two things follow from this. The AI finds a hole in a rule faster than a person does. And with a new rule it pays to measure the parts separately, not only the whole, because the whole often hides one strong and one weak part.

The ordinary game must not change

Through all of this one rule stands above the others: anything new may happen only behind a flag (game.special, game.puzzle). The ordinary game never visits that branch. This is verified by two guards:

• tools/baseline.js plays 172 ordinary games and compares the "fingerprint" of each. A change in the rules, the AI or the adjustments of the player must leave it the same.
• calibrate.js --verify must report "identical".

It is a discipline that a programmer would often excuse themselves from ("it does not concern this anyway"). The measurement does not allow skipping it, because it takes minutes and the result is binary.

What I take from it

• An estimate without a measurement is an opinion. An opinion is fine until you mistake it for a fact.
• A good measurement is deterministic. When the same inputs give the same numbers, the measurement becomes a test.
• Admit the limits. Three rows in a table where the prediction and the reality do not match are worth more than twenty where they do.
• Mark a manual estimate as manual. Fog of war has +3 points, and nobody will learn that it is an estimate off the top of the head if the source does not say so.

Chess programs are tested in a similar way today. Since 2013 Stockfish has used the distributed system Fishtest, in which volunteers run tens of thousands of games of the new version against the old one, and the sequential probability ratio test (SPRT) decides whether the change is better. The lesson is the same: whoever plays against themselves long enough learns the truth about themselves.

Sources and notes

• Procedure: docs/TUNING.md §3 (the principle, the calculation of points, accuracy and limits) and §8.1 (quick A/B experiments). Measurements: node tools/experiment.js '{"defBonus":4}' --vs '{}' --games 600 and node tools/calibrate.js --list of 20 October 2026.
• The table of bonuses and missions: docs/TUNING.md §13.2 (measured on 17 October 2026, 500 games). Passage and Cautious advance: docs/TUNING.md §14.2, the tool node tools/missions.js --heroes.
• Stockfish, Fishtest and SPRT: Fishtest FAQ and the Chess Programming Wiki on SPRT.