How to test a game we will never stop changing

In 1993, before the game Doom was uploaded to the server, it was tested, by recollection, for 30 hours in a row. Today we have 1,234 tests for Audamir that pass in one to three minutes. The right feeling about such a number is distrust rather than calm: if there are many tests, why is a bug still found?

This article is about what we found out in testing the game. The number of tests is not important. What is important is which kinds of protection hold the game together, and where it turned out that green is not proof.

This is how the number of tests grew by day from 17 October:

What guards the game

The protection of the game was assembled gradually, always after it became clear what was missing. Today it can be summed up in six layers:

1. Logic without a screen. The rules, the AI, the puzzles, the campaign and the economy are pure modules without the DOM (the same code runs in the browser, in a Worker and in Node). They can be tested quickly and deterministically.
2. Screens against a fake DOM. The real scripts of the user interface run in Node against tests/fake-dom.js. Fast, but the fake DOM does not know CSS.
3. Fingerprints. A golden fingerprint of a map from a fixed seed, a fingerprint of 172 ordinary games (baseline), 18 check measurements (calibrate --verify), game records that must replay the same way.
4. Measurements. Every tool has --check: missions, minigames, puzzles, the economy of the dice, achievements. They say whether the game has changed in difficulty, not only in correctness.
5. A smoke test of the finished product. A real browser (headless Chromium) plays the real build with real clicks and keys.
6. Independent review. After every larger phase the code is read by another agent, which changes nothing and only reports.

The same six layers in one picture, each with one number from the game of 6 November (the text has the numbers of 20 October):

Of these, the fifth and the sixth layers are the interesting ones, because without them the first four would only claim what we expected of them.

Mutations: testing the tests

A test that never fails need not be a good one. It may merely be blind. That is why a mutation check is done after every phase: errors are deliberately inserted into a copy of the code (one change at a time: an inverted condition, a different limit, a skipped step) and the suite of tests is run. If the change does not make it fail, a test is missing.

One mutation looks like this (a real line of the game and a real test from today's state):

Real numbers from this project:

• The window "Report a bug or idea": 126 mutations (every limit, the cleaning of the text, the addresses in the errors, links, the fallback ways of copying, focus, Esc and Tab). After the first rounds the tests did not catch 8 of them. Every surviving mutation led to a new test. In the end all were caught.
• The territory generator: 37 mutants, 35 caught, three of them only after the tests were extended with a golden fingerprint. Two survived because the code was redundant. It was removed from the project.
• According to the records from the development (these counts are not in the documentation of the project): the Iron Run approximately 55 mutants, Puzzle Labs approximately 66, the Collection 27 and 88, the Threat ladder around 175. Everything was caught after tests were added, except for one equivalent mutant.

The first two checks together with today's third (the renaming of the companions in the campaign, 6 November) in one picture:

An important sentence that can be read from these numbers: a surviving mutant is either a missing test or useless code. It is worth knowing both.

The same principle is also applied to the smoke test of the finished build. It was verified that it fails when it should: 18 deliberately broken builds (a missing link, an Esc that does not close the window, a copy that claims success and copies nothing, links out in the build for portals, a missing service worker, a broken path to the Worker, an error in the console at start and others). The smoke test caught all 18, and each for a good reason. A fire alarm is not tested by looking at it, but by lighting a match.

Three traps we ran into

1. A test that never fails

The helper function waitFor waits until a condition is met. When the time runs out it **only returns false** and does not throw an exception. A test that calls it without assert.ok therefore passes even when the end card never appears. Only a mutation revealed this. The rule since then: the condition is always written as assert.ok(await waitFor(…)).

A similar trap is in Node itself. A cancelled test (the time limit expired) reports fail 0 and cancelled 1, but the process exits with a non-zero code. A mutation script that looked at the number of failures declared such mutants "survived". What decides is the return code of the process.

2. A test that brings down the machine

During a mutation check one test process grew to approximately 33 GB of memory and the machine froze. Twice, because after the first crash the same thing was run again before the cause was found.

The cause: a failing assert.equal(element, null), where element was an element of the fake DOM. On a failure Node prints and compares both operands, and the graph of a single element is the entire fake document. A test that was supposed to announce "this does not match" tried to print the whole world.

Three rules have applied since then:

• **A DOM element is never passed to assert.equal.** Identity is compared (assert.ok(a === b, 'description')), and the header of fake-dom.js warns about it.
• Heavy runs are started one process at a time and in small batches, with a memory limit (ulimit -v).
• After a crash the cause is looked for first. The same thing is not run again.

The last rule is probably the most expensively acquired of the whole series and the most usable outside games.

3. A bug that a human found

The Puzzle Labs screen could not be scrolled, so part of the list stayed out of reach. In the CSS overflow-y: auto was missing. The tests could not find this bug, because the fake DOM does not know CSS. And the manual check missed it: the screenshot "list at the bottom" came out exactly like "at the top" and nobody noticed.

Three things came out of it:

• a test layout.test.js that statically verifies that every screen except the game has overflow-y: auto,
• one-off scripts that scroll with a real wheel and touch (via the DevTools protocol) and compare the position before and after (only the static test stayed in the repository),
• a rule: the screenshots "before and after scrolling" must come out different; if they are the same, it is a bug, not a coincidence.

An independent review and a hundred tests can stay silent, and a person with a mouse finds it at once. That is why the verification of the finished build is also done with real input: a click, a wheel, a key and a touch on a mobile.

When test data lies: migration

The game stores the profile, the campaign, the backup and games in progress in the browser. When the IDs change (in this project Slovak identifiers were changed to English ones), it must be guaranteed that the old save still works and nothing is paid out twice.

A test with hand-written data passed. A real profile, however, revealed a bug: four Puzzle Labs achievements were paid out a second time after the labs were renamed, because the achievement ID is derived from the lab ID and was missing from the migration table. It was found out by creating, with the old code (from the last commit before the change, via git archive), a profile, a campaign, a backup and a saved game and loading them with the new code. An independent review found the same problem only after about twenty minutes; a comparison of two sets of IDs finds it in a second.

The lesson: test data written by hand reflects what the author wants, not what exists. If the format that a player already has saved changes, the data must be made by the old code. Whoever does not do this finds the bug only with the first user.

The same principle stands behind two tools (talents-migration.js, integration-check.js) that produce profiles at several stages from the archive of the old tree and verify that the new code handles them.

Determinism as a superpower

The best thing the game did for testing is not any test. It is that all the randomness of the game is derived from the state (seed, round, turn, order of the attack). That is why every game is reproducible:

• tools/replay.js --games 300 --check plays 300 random games with the AI, records them and replays them. Every checkpoint and the result must match.
• A saved game does not roll the dice again on return (a roll cannot be "repeated" by leaving to the home screen).
• An error report carries the seed and the "salt" of the rolls, so the bug can be reproduced.

Time is another source of randomness. The tests of the puzzle of the day had the clock of the page pinned to an "easy day", because the difficulty grows during the week and the suite of tests would otherwise depend on which day it is run (this was verified by setting the clock to the hardest Sunday). In the tests with a Worker a two-minute timer stayed hanging, which kept the Node process alive.

Reviews by an agent: what they know and what they do not

After every larger phase (Puzzle Labs, the Collection, the Avatar^[1] H1 to H4, the game record, achievements, the analysis) the code is read by another agent, which only reports. It is a cheap form of a second pair of eyes. According to the records from the development (not all the numbers are in the documentation): for the campaign the review found 7 real bugs, for the Collection 16 findings (7 medium and 9 low), for the achievements 12, for the analysis 13, and for Puzzle Labs no serious one, but a few real ones (the help drew the chances even in a puzzle, the end card could crash with a full list of results).

What the review did not find is just as instructive: it did not work out that the Puzzle Labs screen cannot be scrolled. On the other hand it found edge cases: two browser tabs overwriting one state, a "dead" run after the last attempt was discarded, a failure of copying. It is a good complement. It is not a replacement for a person who actually plays the game.

What I take from it

1. Green is a claim, not proof. A test that has never failed must be broken to prove that it can fail.
2. A surviving mutant is a gift. Either a test is missing or there is something unnecessary in the code.
3. Tests made up out of one's head lie. If a saved format changes, create the data with the old code.
4. Play with real input. Wheel, touch, key. Whoever does not scroll does not know whether it scrolls.
5. After a crash do not run the same thing again. The cause first.
6. Speed is a feature. A minute or two can be run after every change. An hour would be run only before handing in.

On the last point: this is what the one command that verifies the whole game today looks like:

And where the time goes in it:

Sources and notes

• Counts: node --test in the main tree of the game of 20 Oct 2026 (1,234 tests, 1,233 passed, 1 skipped, 82 test files, --test-concurrency=3: 161 s). On a copy of the project from the previous morning (1,187 tests) the suite took 61 s with the default parallelism. The length of the tests approximately 29,000 lines.
• Mutations and the smoke test of the build: docs/DESIGN.md §22 (126 mutations, 18 broken builds), §19.6 (37 mutants); the rest from the notes on the phases.
• The memory accident: docs/DESIGN.md §22.3 ("A lesson about tests and memory") and an entry in the history (a commit of 18 Oct 2026, 23:24).
• The pictures (indicative, the numbers are today's, not of 20 Oct): the numbers of the layers and the output are from one whole run of node tools/verify.js on the tree of the game ce303e2 of 6 Nov 2026 (dev shortly before version 0.119.0; 32 min 5 s, all phases passed). The growth of the number of tests is the number of calls of test( in the files tests/**/*.test.js at the last commit of each day of the branch dev (approximately; 17 Oct from the saved copy of the project). The mutations are from the text of the article and from the check of the renaming of the companions in the campaign (27 mutations, 6 Nov 2026). The line of code and the test in the sketch of the mutation are real (js/core/feedback.js and tests/core/feedback.test.js on dev 0.119.0, run on 6 Nov 2026).
• Doom and 30 hours of testing: research for the opening article (Kushner, Masters of Doom, via Wikipedia); it is a recollection, not a measurement.