My Mutation Score Was 100%. That's How I Knew Something Was Wrong.

I was adding a new module to a simulation project and ran the mutation testing harness to check my new tests were any good.

killed 165/165   mutation score 100.0%

A perfect score. Every single mutant caught.

I did not celebrate. I went looking for the bug — and there was one.

What mutation testing actually does

If you have not used it: mutation testing answers a question code coverage cannot.

Coverage tells you a line ran. It says nothing about whether anything checked what that line did. You can execute every line in a codebase with a test suite that asserts nothing at all and score 100%.

Mutation testing works differently. It deliberately breaks your code — flips a <= to <, changes a +1 to -1, deletes a statement — and then runs your tests.

• If the tests fail, the mutant is killed. Something was watching that line.
• If the tests pass, the mutant survived. That line ran, and nothing cared what it did.

Your mutation score is the percentage killed. It is the closest thing I know to an honest measure of whether a test suite is doing its job.

Why 100% is a red flag

Here is the mechanism stated plainly, because the bug lives inside it:

A mutant is judged killed when the test suite fails with it applied.

Read it again with a hostile eye. It has a corollary that my harness never accounted for:

If the suite already fails, every mutant is killed.

The harness applies a mutation, runs the tests, sees red, and records a kill. It has no idea the red was already there. Do that 165 times and you get a flawless score generated entirely by a broken build.

And that is what happened. A stale number in my README made three assertions fail — nothing to do with the code being mutated. Every subsequent mutant was scored killed against an already-failing suite, and genuinely surviving mutants went unrecorded.

The number that should have raised an alarm was the reassuring one.

That inversion is what makes this worth writing down. If the harness had printed 83%, I would have investigated immediately. It printed 100%, which invites you to close the terminal and feel good.

The fix is four lines

Run the suite once, before mutating anything, and refuse to continue if it is red:

if (!runSuite()) {
  console.error(
    "REFUSING TO RUN: the test suite fails before any mutation is applied.\n\n" +
      "Every mutant would be scored KILLED and the result would read 100%,\n" +
      "because a mutant is killed by the suite failing — and it already does."
  );
  process.exit(1);
}

The error message matters as much as the check. "Suite failed" would send someone hunting for a broken test. Explaining why the result would be meaningless is what stops them from working around it.

The real score, once the suite was green: 87.3% — with twenty untriaged survivors to go and fix. Considerably less flattering, and considerably more useful. (That number was itself under-read: the negation operator was splicing if (! without parentheses, turning if (a !== b) into (!a) !== b — always true — so vacuous mutants "survived". After fixing the operator and triaging every remaining survivor with a written equivalence argument, and re-running after two later audits that widened the campaign to the oracle and the core: 352 of 362 killed, 97.2%.)

The survivor that was not a missing test

While fixing those survivors I hit a second, better problem.

A mutant changed attempt <= MAX_RETRIES to attempt < MAX_RETRIES and survived. My first instinct was the obvious one: I am missing an assertion about the retry count. So I sat down to write it.

I could not.

The branch that handles running out of retries could never execute. In my model, all the contending writers committed during the first attempt, so the compare-and-set could lose at most once and then always won. A reachability probe across every input shape confirmed it: the retry-exhaustion path never fired, not once.

That is dead code wearing the costume of defensive programming. It looks careful. It reviews well. It does nothing.

The fix was not to delete the branch. Deleting it was defensible on the evidence — it genuinely never ran — but the retry bound is correct, and the reason it never ran was that my model let contention politely stop after one attempt. Real contention does not do that. A busy row stays busy.

So I fixed the model: one contender commits per attempt, sustained. The branch is now reachable, the test that pins it is real, and the model is more faithful into the bargain.

When a mutant survives, the interesting question is not always "which assertion am I missing?" Sometimes it is "why does this code never run?" — and the answer is that your model is too polite, not your test too weak.

The part that generalises

Both of these are the same shape, and I had already hit it once before in the same project. An invariant checker was reading the system's own report of what it had done, rather than the raw facts — so a correctly-refused operation looked like a violation. A checker that trusts the thing it is checking is not a checker.

The mutation harness is that failure one level further out. It is the tool that grades the tests, and it could not distinguish "these tests are excellent" from "these tests are broken."

Every layer that grades another layer needs someone grading it. Eventually that someone is you, asking a question that feels faintly paranoid:

What would this output look like if the tool itself were wrong?

For a mutation harness, the answer is "a perfect score." For a code coverage tool, it is "100% coverage of assertions that assert nothing." For a green CI badge, it is "the job passes because the test step was never wired up."

I have found all three of those this month, in my own code.

What I would tell someone adopting mutation testing

Do not chase 100%. It is not achievable on real code and, as above, achieving it is more likely to mean something is broken. Some mutants are genuinely equivalent — they change the code without changing its behaviour, and no test can kill them. A healthy score is somewhere in the eighties with every survivor either fixed or explicitly argued as equivalent.

Triage every survivor, and write the argument down. "This mutant is equivalent because…" is a sentence worth keeping in the repository. Six months later, neither you nor anyone else will remember whether that survivor was fine or forgotten.

Assert on outcomes, not on tallies. Several of my survivors changed arithmetic inside a data structure while every test still passed — because my tests counted results from a log of events and nothing ever inspected the structure itself. That was a design gap, not a missing assertion, and mutation testing is what surfaced it.

Run it on a green suite. Now enforced, in my case, by the harness itself.

---

The project is BALLAST, a deterministic simulation of a multi-tenant control plane. Every finding above — and six more — is written up in its LEDGER.md, including which ones turned out to be in the checker rather than the system. Eighteen of twenty-nine were.