BallOrcs Battle: 5,000 orcs at 60fps — and six tests that were only pretending

In plain words

Two questions got real answers this week. First: can this thing actually run an army? Five thousand units, on a laptop's built-in graphics chip — not a gaming card — held a steady 60 frames per second. That's the target met on modest hardware, and the simulation itself was using about a quarter of its time budget.

Second, and less comfortable: how much of our automated testing was telling the truth? We went looking, and found six separate checks that reported success while checking nothing at all — a robot soak that passed with zero robots, a screenshot comparison that passed against an empty reference set, a performance gate that could never fail. All six now fail properly. A test that can't go red isn't a test, it's decoration, and a green wall of those is worse than no wall.

The other headline: the promise that two different computers simulate the same battle identically is now proven on two different compilers, not one.

Under the hood

The dev-GPU number (9163ba9, closes #4). 5,000 units on stress_flats, windowed, Vulkan Forward+ on integrated Intel Iris Xe: fps pegged at the 60 Hz vsync cap, process_ms p50 17.6 ms, draw_calls p50 14. The ceiling is unmeasured because vsync bounds it. The finding worth acting on is in the sim column: sim_step_ms p50 is 1.83 ms windowed against 0.78 ms headless at the same unit count — 2.3×, because the sim now shares cores and memory bandwidth with the renderer. Every headless row in docs/perf.md is an optimistic reading of the tick budget by roughly a factor of two. Still 23% of the 8 ms Phase 3 ceiling, but half the margin the headless numbers implied.
The 5,000 in "5,000 units" was 4,608 (b9007ba, #4). debug_scatter_spawn drops 8 units per passable hex, so dev_flats (24×24) topped out at 4,608 and every "5k" perf row was silently a 4.6k row. stress_flats (32×32, 8,192 capacity) fixes it and match.gd grew --map=NAME, resolved as a stem inside res://data/maps so a typo fails with the name in the log.
And "slightly better than linear" was wrong (97f5416). The numbers in that same sentence refuted it — 4,608 → 5,000 units is +8.5% units for +39% time — because it compared a dev_flats row against stress_flats rows. Map area is not free. Re-run on one map throughout: 0.164 / 0.144 / 0.156 / 0.150 µs per unit at 2,000 / 4,608 / 5,000 / 8,000. Flat within noise, linear over 4× of range.
Determinism, on two compilers (42edd79, closes #1/#2). Goldens confirmed on MSVC x64 after re-verification on GCC 11.4, 64/64. Also CHECK-continues-after-failure made failed preconditions UB — missing data segfaulted the suite instead of failing red — so fatal REQUIRE/REQUIRE_EQ landed and the hazard sites converted. Then (194a7d9) the GCC build was held to MSVC /W4 parity: compiling all 26 sim/test units with -Wconversion -Wsign-conversion -Wshadow produced zero warnings, so those flags are now on the GCC path permanently — drift shows up in every Linux build log rather than waiting for someone to trigger the Windows job by hand.
Roster reads are deterministic now (7747d85, #6). Tick-driven bots decide at a decision tick but observed at whatever tick the frame's catch-up loop reached — frame-rate dependent the moment units can despawn. SimWorld gained a bounded spawn/despawn event log (ROSTER_HISTORY_TICKS=64 ≫ MAX_CATCHUP_STEPS=6) and alive_ids_at(t), reconstructing the roster exactly as a live read at tick t saw it by undoing newer events in reverse so slot reuse unwinds correctly. It's derived bookkeeping — not hashed, not snapshotted — so the goldens didn't move.
The six pretenders (483be3b, #7 follow-up). all([]) is True, so run_swarm reported PASS with zero runs. An empty screenshot baseline made the graphics check a no-op that reported clean — and CI then promoted that run's shots over the baseline, so one probe dying after mkdir would wipe the reference set forever. A metric with a 0 baseline can never regress (compare() requires base > 0) and the report printed "ok" for it; the report now states how many of its seven metrics are actually gating — it's two. trace_corpus --only <typo> reported PASS, which is exactly the flag a failure report tells you to re-run with. And a corpus where every case SKIPped passed, because ok = not failed; the checked guard had even been written and never wired up. 94/94 python tests, 7 of them new — none of these paths had a test before.
A heredoc that fed Python nothing (97f5416). The [perf-windowed] log lines never existed: a heredoc piped to python3 - inside $engine run --rm — no -i, so the container gets /dev/null for stdin, Python read EOF, ran an empty program and exited 0, with || true hiding even that. Replaced with a real script invoked like its neighbours, which now reports "capture did not run" for an empty directory rather than printing nothing.
Windows CI, and a build that only failed in CI (94e7d66, 0488c0b, #5/#10). infra/windows/register-msvc-runner.ps1 front-loads the failures that otherwise surface hours later — notably that the runner service runs as LocalSystem while preflight runs as the interactive admin, so pip install --user scons passes preflight and then fails in CI with No module named SCons. Separately, pipelines are now refused on the gitlab.com mirror: both instances share one self-hosted runner at concurrent = 1, so every push compiled the same commit twice, in series, on one box. And the leading suspect for #10 — CI producing a broken GDExtension from source that built fine locally — is the shared scons cache handing back godot-cpp interface files generated against a different engine pin. The engine keeps its cache; the extension no longer does.

Green this week: 68/68 sim tests (4 new for roster churn), goldens unmoved, 94/94 tooling tests, shellcheck clean. Phase 4 lockstep multiplayer is still ahead; this was the week of making the measurements trustworthy before building on top of them.