A one-character typo in a comment took down a Project Zomboid mod’s entire item set, and every automated check we had said the change was fine. The interesting part isn’t the typo — it’s the two independent reasons nothing caught it.
The symptom
A routine balance change: two numbers, plus a comment explaining why. Lint passed. The full headless suite passed — forty-odd assertions, green. The change was committed, cherry-picked to the release branch, and deployed.
The next server boot died:
WARN : unknown script object "/*"
ERROR: Item.Load> ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
LOG : [item] removing script due to load error = BeerRefill
LOG : [item] removing script due to load error = WhiteLightning
ERROR: WorldDictionaryException: World loading could not proceed
LOG : Server Terminated.Two items gone, world refuses to load.
Why one stray character took out half a file
The comment stripper does not scan forward. It seeds from the last */ in the file and walks backwards, pairing each terminator with its opener. When it hits a */ it cannot pair, the search index runs off the front of the file, goes negative, and the whole strip loop aborts.
Not “skips that comment” — aborts. Everything above that point stays in the parse stream as if it were code.
The change had introduced one extra */, which closed a comment early and orphaned its real terminator further down. Everything above the typo was now unstripped prose being read as script.
That explains the otherwise baffling blast radius. The parser splits statements on commas and then splits each fragment on =. Prose with a comma but no equals sign produces a one-element array, and the loader indexes element two:
"...pass, not flavour" -> split("=") -> ["...pass, not flavour"] -> [1] -> boomSo the casualties weren’t the items near the typo. They were the items above it whose comments happened to contain a comma and no equals sign. Exactly two items matched that description. Exactly two items died. The model predicted the log before we read it.
Why lint didn’t catch it
The linter strips comments too — with its own forward-scanning implementation. It saw a well-formed file, because by its rules the file was well-formed. It was checking a different grammar from the one the engine actually implements.
A linter that reimplements the parser it’s protecting you from will agree with itself.
Why the test suite didn’t catch it — the part worth internalising
The engine loads script files once, at boot. Everything afterwards runs against objects already in memory.
The dev loop was: edit the file, deploy it, run the suite. But the running server had loaded the previous version of that file at its last boot, and nothing in “deploy” makes a running server re-read scripts. So the suite was faithfully testing the old file while the new one sat unparseable on disk.
Every green result was real. It was just answering a question nobody was asking.
The detail that stays with me: that commit’s own message explained away twelve failing assertions as an expected pre-restart mismatch. The author knew the tests were running against stale scripts. They shipped anyway, because the failures had a plausible story and the rest was green.
What we changed
A script change is unverified until a boot succeeds. Not “tests pass” — boots. That is now a written rule rather than folklore, because it had been folklore and folklore lost.
The check is more than “did the server come up”. The engine drops individual bad scripts and carries on, so a server can look perfectly healthy while a mod item has silently vanished:
SERVER STARTED <- necessary, not sufficient
removing script due to load error = 0 <- the one that actually catches it
unknown script object = 0And a warning count is not an instance count. The log deduplicates identical messages, so one line can conceal any number of broken items. On a related bug the same week, a single logged warning was hiding seven.
The general shape
Three checks, three different blind spots, one root cause each:
- The linter reimplemented the grammar it was verifying, so it could only find bugs it already agreed were bugs.
- The test suite measured a stale copy of the thing it was testing, and had no way to know.
- The human had a plausible explanation for the failures that did appear.
None of these is a missing test. Each is a test that ran, passed, and meant nothing — which is strictly worse than no test, because it comes with confidence attached.
The cheap defence isn’t more assertions. It’s asking, of any green result: what exact bytes was this measured against, and how do I know?