Bugs of the week: five things that could not be repeated on request

The worst bug in tests is not the one that always fails. That one can be fixed in half an hour. The worst is the one that fails once in ten runs, once after the date changes, and once only on a machine that is busy at the time. The week from 23 to 27 October 2026 brought five such cases: three checks that depend on chance or on the clock, one loop that got stuck, and one visual bug that the eye no longer saw after the first fix, although it was still there.

This article follows the article on testing, which said what the tests guard. Here the opposite question is asked: what they do when they cannot be trusted. All numbers come from commits and from the documentation of the game (the sources are at the end). Nothing of it was run again while the article was written; the exception is the pictures of the cube in section 4, which come from old commits of the game and show only what can be seen in them.

Overview of the cases

1. A test that played a different game on every run

While working on steps M4 (the application reads the table of modes, see the article on realms), a test of the mode Watch a game showed up, in which the computer plays against itself. The test waited at most 600 virtual seconds for the end of the game. One run in ten the game did not finish in time.

The first suspicion was the load of the machine, because other heavy tools were running at the time. A measurement refuted it: 40 runs one at a time, without load, 4 failed (DESIGN §54.3). The load only made more runs happen, so the bug was seen more often.

The real cause was in the game itself. A watched game has no player who could “fix” the rolls of the dice, so Flow draws a random salt for the rolls. The map is given by a seed, but the game on it lasted, depending on the salt, 13 to 29 rounds, which is 138 to more than 600 virtual seconds (median 343). So the test with its limit of 600 seconds sometimes did not manage to finish a game that is entirely regular.

The fix is simple: the test page of this mode got a fixed salt (Watch.makeFlow wrapped in beforeApp). The salt MODES-B gives the game 13 rounds and 288 seconds. After the fix 40 runs one at a time, 0 failures, and the original scenario (the file with six other interface tests next to calibrate --verify) three times, also without a failure.

2. A migration check that read the clock

The migration checks in this game work so that the old code (from git archive of an old commit) plays the real history, the new code is started over its data, and the result is compared. Why this matters is in the article on the daily challenge. The check tools/daily-hub-migration.js sometimes failed with the message that the day's record shows today's challenge as lost, although a win was expected.

Two things changed between runs:

• Chance. Each game drew its seed (Principalities^[2], free play) and the salt of the rolls.
• Date. The daily challenge is a different map every day. The AI sometimes won today's, sometimes lost, and the check demanded a win.

The fix (26 Oct 2026): the clocks of both processes are pinned to 2026-10-24 (a fixed offset, so the time still runs), the seeds and the salt of the old application come from a counter, and the record of the day is compared with what the old code wrote: a win when the old profile has that day among the won ones, otherwise a loss. The test also checks that the clocks are really pinned. Five runs of the whole tool give the same output byte for byte, even when the system date is moved by +4, +30 and −10 days.

Pinning the clock revealed a second bug

After the clocks were pinned, the check, unfortunately, failed on every run: the commit message gives the pair of numbers 108/110 against 65/108 (without a closer explanation of what exactly they count). The explanation is in another weekly story, the score by the length of a game. A game won on points by the new code is counted with the new formula and with the length factor of that day (1.13 to 1.89 by the measured days), so its record no longer has the same numbers as the one the old code made. The check, however, demanded equality. Before the clocks were pinned, it “passed or did not pass” by the date of the run, and so nobody noticed.

The fix solves it: the newest record of the daily challenge may differ in the sum, but its base must equal the points of the settings of that day (Score.compute from Settings.daily(day)); the other records are compared as before.

3. A migration check that waited for an answer nobody gave

The rules migration (tools/rules-migration.js) had the same kind of check too, and something worse troubled it on top of that. The test (tests/tools.test.js) sometimes ended with the message null !== 0. In Node this means that the subprocess did not finish and was killed by a signal: the tool hit its limit of 120 seconds.

According to the development notes there were at least three causes, and not all of them are certain:

1. Stuck after a stalemate. A game of the old code that ends in a stalemate (the Principalities in round 14, or the daily challenge) asks “End on points?” and holds the game until somebody answers. The helper playOut had no answer, nor a guard against getting stuck, so the AI waited 20,000 times, 20 seconds each. On a clean development version of the game the tool got stuck in 3 of 6 runs.
2. Memory limit. Under ulimit -v 24000000 the tool took, according to one session, about 480 seconds instead of about 11 seconds without the limit. This is not a rule (other runs with the limit passed), but it is worth keeping in mind when looking for the cause.
3. Load of the machine. When other sessions ran whole test suites, two runs in a row failed after 120 seconds, even on the clean development version of the game. On a free machine the tool takes about 12 to 14 seconds.

The fix (26 Oct 2026) used the helpers of the previous fix: the pinned clocks, the seeds from a counter, the answer “End on points” to the question about the stalemate, and a game that gets stuck is at once an error, not 20,000 waits. The test gives the tool 280 seconds (on a quiet machine it takes 14). Eight runs are byte for byte the same (five ordinary and three with a moved date). The path of the answer was verified by ten sets of seeds: two of them reached a stalemate (the Principalities in round 11 and the daily game in round 68), and the check ended with the message “the old and the new code agree”.

4. A cube you can see through

The last two cases are not about tests, but they show where tests do not help. The hero of the game is a cube made of CSS 3D (transform-style: preserve-3d, six .face elements, no canvas; the description is in the article on boxes and in docs/HERO-DICE.md). The walls have rounded corners (18 %) and behind them there was nothing. When the cube was turned with a corner towards the viewer, the pips of the far wall showed through the corners and the edges. My note from playing drew attention to it (26 Oct 2026).

The first fix (version 0.50.6). Behind each wall a square plate in the colour of the wall's edge was placed. It was looked at in the browser on the home screen, in the game and in the card Die, in the poses with a corner towards the viewer and also after dragging. It looked good.

The second fix (version 0.50.7). According to my report the show-through persisted, most of all at the light pips of the stone and the iron die. The cause: the plate had inset: 0, which stops at the inside of the edge, so it was 2 px (the hero's die) or 4 px (the die in the card) smaller than the wall, and the ray at the corner went past it. This time it was not looked at, but measured. One die in each cell got a random pose, only the pips of the most turned-away wall were lit, and every lit pixel was a leak. Measured in Firefox in 80 poses for sizes 64, 30 and 90 px, for the die in the card and the die of the hero:

In Chromium no leak in 60 poses. The first fix therefore removed only about a third of the leak (from 16 to 11 poses), and from looking at a few poses it could not be seen.

The third adjustment (version 0.50.8). Where two walls of different materials met, the edge and the plate had the colour of their own material, so the corners of the iron die with brown and silver walls looked like patches. The cube now carries its world and the level of its edge (data-world, data-rim), and all edges and plates have the colour of the lowest level among its walls. A test keeps the table of edges in style.css equal to --face-edge in the rules of the levels. Looked at in Firefox (dark and light theme, eight dice of three worlds in four poses) and in Chromium; the measurement of the show-through stayed the same, because only the colours changed.

The test that still fails, and why

Not everything was fixed. At step M4 the whole suite (node --test --test-concurrency=3, 2,508 tests, 7 minutes) ended with one failure: tests/rollwin.test.js (the test “a move to make”). It fails about once in thirty runs even on the clean development version of the game: 1 in 30 on a clean working tree of version 0.50.19 and 1 in 12 on a tree with an unfinished change. The test starts the Principalities on a random map and asks the AI for the first move of the human's seat. This is the same pattern as in case 1: a random map in the test. The documentation records that the test was not changed. So the fix waits.

One more curiosity comes from the same suite. In a run during which games were played in the browser pane, two tests of tests/autopilot-ui.test.js also failed; they wait for the AI in real time. Run alone they passed (3 of 3). A test that depends on real time depends on the load, even when it does not look like it.

And one more story from the same step: the page of the test runs in a vm context, where object literals have another prototype of Object than in the module itself. The check of “plain data” in the realm manager therefore refused all three manifests on the first run with the message “value is not a plain object”. In a browser this would not happen. The check now accepts the root prototype of any realm, and still refuses classes, Object.create(x), look-alikes and a getter that throws. Here the test behaved differently from the real environment, and that is a bug of another kind, one that is worth knowing about.

What I take from it

• A test plays one fixed game. If the game knows the seed and the salt of the rolls, the test sets both. The case Watch a game: 4 failures in 40, a fixed salt, 0 failures in 40. The unfixed case rollwin is so far exactly the opposite example.
• A check does not read the clock or roll dice. A pinned time and a counter instead of chance (cases 2 and 3). Pinning has a side effect: checks that “passed by the date” start to fail always, and they reveal bugs that were hidden until then (108/110 against 65/108 in the commit message). The good news is that the bug is now repeatable.
• Every wait has a limit and can say why it stopped. 20,000 waits of 20 seconds is not waiting, but getting stuck. The guard against getting stuck and the answer to the question about the stalemate changed “hangs once in a while” into “ends with an error at once”.
• A visual bug is measured, not looked at. The first fix of the cube passed a look at three screens and removed a third of the leak. Only the measurement of a lit pixel in 80 poses gave a number that can be compared: 16, 11, 0.
• A difference between environments is a bug of the test, not of the game. The vm context and the browser have different prototypes; that the code works in the browser is not enough while the tests run somewhere else.

Sources and notes

• The test of the watched game: commit of 27 Oct 2026 and docs/DESIGN.md §54.3 (40 runs, 13 to 29 rounds, median 343 s).
• The check of the daily screen: two commits and a merge of 26 Oct 2026, docs/DESIGN.md §48.9. The length factor (1.13 to 1.89) is from §51; the numbers 108/110 and 65/108 are from the message of the second commit.
• The check of the rules: a commit and a merge of 26 Oct 2026; the time of 14 s and the limit of 120 s are from the message of the commit. The number of runs, the times with ulimit -v and the stuck processes (section 3) are from the development notes, not from an independent measurement.
• The cube: versions 0.50.6 to 0.50.8 of 26 Oct 2026; docs/HERO-DICE.md, ch. 11 (the geometry of the cube). The measurements 16/80, 11/80 and 0/640 in Firefox (tools/firefox-check.js in the game) and 0/60 in Chromium are from the message of the commit of version 0.50.7.
• **The failing rollwin and the vm context:** docs/DESIGN.md §54.5 and §54.2 (27 Oct 2026).
• What was not repeated: build:check, tools/smoke.js and i18n:scan, because Chromium does not start from a terminal on the development machine (snap-confine; docs/DESIGN.md §50.10 and §54.5).