2026-05-13

Encoding mutations directly into the test code

I sometimes annotate a test case with the (potential) mistake that caused me to introduce the test in the first place, as it can make it easier to understand why a test looks the way that it does.

Consider the following module, written in F#:

open System

module Validator =
    let isAllowed (now: DateTime) (date: DateTime) = now.Date <= date

I might, for example, write the following comment inside of one of its test cases:

open Xunit

[<Fact>]
member _.``You're allowed to pick the current day``() =
    // This catches the bug where we wrote `now <= date` instead of `now.Date <= date`
    let now = DateTime(2026, 5, 12, 17, 17, 13)
    let date = DateTime(2026, 5, 12)
    Assert.True (Validator.isAllowed now date)

I’ve always found that to be slightly dissatisfying, however.

I don’t like tieing tests to implementation details, and this is about as close as you can get to tieing a test(’s comments) to an implementation detail, which might very well go out-of-date in the future, and completely lose its relevance.

As a compromise, I’ve sometimes taken to including the mutation in the commit message, so that it at least shows up in the git blame, together with the original version of the code for which the mutation was relevant. I’ve even written an article about formalizing that notion into something that can be verified automatically. Ensuring that these claims show up in exactly the right place in your commit history is quite cumbersome, however, and I’m increasingly starting to second-guess that strategy. We’re also missing out on the ability to (automatically) retry the mutation in the future, which is especially relevant if we ever change the test.

Let’s take a few steps back, and consider again if we can just embed the mutation in the test code directly. What if we formalize it in a way that allows to check if it the mutation is still relevant today, so that we can make sure that it never goes out-of-date?

[<Fact>]
[<ShouldCatch("""
--- a/Example/Validator.fs
+++ b/Example/Validator.fs
@@ -3,4 +3,4 @@ namespace Example
 open System

 module Validator =
-    let isAllowed (now: DateTime) (date: DateTime) = now.Date <= date
+    let isAllowed (now: DateTime) (date: DateTime) = now <= date
""")>]
member _.``You're allowed to pick the current day``() =
    let now = DateTime(2026, 5, 12, 17, 17, 13)
    let date = DateTime(2026, 5, 12)
    Assert.True (Validator.isAllowed now date)

We can now trivially verify that all patches still apply in a pre-commit hook, but what do we do if they don’t?

This is utterly frustrating to maintain, right?

Could it be that LLM’s are tipping the scale, however?

They can figure out an equivalent mutation, even if your code looks nothing like it did in the past. It is, of course, not bulletproof, but these annotations are our third line of defense¹, so it will get us a long way, and it will make it a lot less scary to change our tests in the future.

Prototype of (mut)ation (annot)ations for .NET: mutannot.