A test for a multiplayer game in Godot printed "ALL TESTS OK" and exited with code 0. Inside that same run there were six SCRIPT ERROR lines. Two of them said Compilation failed and another Trying to assign value of type 'CanvasLayer' to a variable of type 'character_select_panel.gd'. The runner looked at the exit code, the exit code was 0, and nobody looked any further.
A test that can't fail is worse than no test. Without a test you know you don't know. With a false green you think you know.
Six errors inside a green
The failing block instantiated the full level scene to count the collectibles it declares. That scene drags in two UI zones whose scripts use autoloads that, when you boot the engine with --script, don't exist. The scripts didn't compile, the node came out as CanvasLayer instead of the expected type, and Godot spat the errors to stderr without bringing down the process.
The ugly part isn't the error, it's that the test carried on and the later assertions were evaluated against a half-built tree.
The fix was to stop instantiating. The scene is read with PackedScene.get_state(), which gives you the node names, the instance= and the overridden properties without running any _ready or loading the zone scripts. The precondition is still anchored to the production file: if someone removes an object from the level, the test goes red.
The assertion that was a tautology
That same test had another, worse problem, because it makes no noise on stderr. The precondition said "the level ships N templates" with N derived from the scene itself, and then checked that the conversion returned N. That always passes. It doesn't check that it converts the templates that exist, it checks that the number you just read equals itself.
Now that number is written by hand:
const PLAYGROUND_TEMPLATE_COUNT := 7 # the 7 pickups placed in the editor
And every converted payload is matched one by one against its declaration in the .tscn by position, with a 1 px tolerance because the host had already integrated a frame of gravity before converting them, plus its type and its amount. Hardcoding the 7 is what separates "converts however many there are" from "converts the 7 that are there".
How you check that a red is possible
The way to know whether a test is worth anything is to break the code on purpose and see if it notices. In that fix I wrote down the three mutations I tried: removing the remove_child from the function that detaches the templates turned three assertions red; forcing the amount to 1 in the payload turned two red; deleting an object from the scene turned the precondition red.
Thirty seconds per mutation, and it's the only real evidence that a test protects anything. A test you've never seen red is a hypothesis.
The race that made the two-instance test pass
A script launched two instances of the game, host and client over ENet on localhost, and captured the state of both. It came out green. It had three false greens at once, and I reproduced all three before touching anything:
- The host died halfway through and returned 0. The client had already captured its frames with a local player, so the script declared itself satisfied. Now it also requires the host to die from the script's own
SIGTERM; the code measured is 143. pkill -ffed unescaped paths into an ERE. With a+in the path it didn't match and the host lived until the timeout: three minutes instead of twenty seconds. With a.it could kill the Godot from another run. Now it kills by the direct child's PID.- The client guessed by frame number when the host existed. And you can't guess, because before hosting Godot installs an
OfflineMultiplayerPeer:get_unique_id()returns 1 andis_server()returns true in the menu, including on the instance that's about to be the client. The only real marker for "it's hosting" is its JSON having a player withis_local.
Along the way it turned out that the autopilot's at values count drawn frames, not seconds: with vsync, a covered or unfocused window waits on the compositor and drops to 1-3 fps. Measured: 85 frames in 90 seconds. With --disable-vsync --max-fps 60, a steady 51 fps.
Test the artifact you ship, not the one you build
The case that cost me most was in a desktop transcription app. CI was green and shipping a .deb with a dead microphone, because nothing ever ran what it built.
PyInstaller bundled the PortAudio from the sounddevice wheel, compiled without a PulseAudio backend, plus a libasound.so.2 without its alsa-lib modules. Inside the bundle that wipes out the pulse, pipewire and default PCMs and leaves only the hw: ones in exclusive access. PipeWire already has the mic, so recording failed with Invalid sample rate [PaErrorCode -9997].
It only reproduced on the installed .deb: in development you link against the system PortAudio, which does have Pulse. The fix was to exclude libportaudio, libasound and libjack from the bundle, since they were already required in Depends.
What changed in CI matters more than the fix. Now, before packaging, it checks that there are no audio libraries inside the bundle and that the binary starts under xvfb. And the .deb declared libc6 (>= 2.43), a floor that installs on no Ubuntu: it came from taking the highest GLIBC tag in the bundle, that is, from libraries copied off the build machine. It described the builder, not the app. Now it's declared by hand as what it is, a release decision: 2.35, the runner's.
I also removed the Windows build from CI: I commented it out entirely and took build-windows out of the release's needs. Keeping a green artifact nobody tested cost more than it gave.
What I ask myself now about a new test
Two things, and if either one fails I don't trust it. First: have I seen it red from a change that actually breaks the behavior, not one that breaks the shape? Second: does the expected value come from somewhere other than the code under test, or am I deriving it from that code?
And a third one for CI: what you test has to be the file someone will download, not the one that comes out of your run.