..

Should we start writing verifiable claims in commit messages?

Whether or not every commit should build and pass all of its tests is, arguably, a contentious issue:

Every commit should be runnable, that is we should be able to git checkout any commit and get a functional code base. This means no “WIP” commits or commit chains that only restore functionality at the end.

Cleaning Up Git History, Robin Schroer

Within a feature branch, not every commit necessary passes the tests (or even builds), and that is a useful property! Here’s some ways this can be exploited:

  • When fixing a bug, add a failing test first, as a separate commit. That way it becomes easy to verify for anyone that the test indeed fails without the follow up fix. […]
  • To refactor an API which has a lot of usages, split the work in two commits. In the first commit, change the API itself, but don’t touch the usages. In the second commit, mechanically adjust all call sites. […]

Git Things, Alex Kladov

I often lean towards the former, but context is everything, and the latter sometimes makes a change so much easier to review that is it hard to stick my ground.

I don’t have a definite answer.

What I do believe, however, is that there is a lot of value in simply letting your reviewers know what kind of commit they are looking at:

  • Is it a self-contained commit that fixes all call sites and tests, or is there something “missing” in the diff? Should I, as a reviewer, try to look for missing pieces, or should I trust that a fix for that weird thing that I just noticed is still coming, and march ahead to the next commit?
  • Does the test that you just added succeed with the current version of the program, or does it rely on a change that is introduced in the next commit?
  • Does the change that you just made cause the test that you added in the previous commit to succeed, or is it just a harmless re-factoring that doesn’t change the program’s behavior?

All of these things can be written down in commit messages, but I wonder if we can do even better.

Your CI/CD pipeline currently only checks that the end result passes the tests. Reasoning about a change becomes a lot easier if we can check how the individual commits behave as well.

Knowing that a commit builds successfully and isn’t missing anything is more valuable than just reading in the commit message that it is, and will occasionally save you from a wild-goose chase that ends with the realisation that it really doesn’t build after all.

Verifying that a commit that purports to add a failing test contains a test that actually fails for the reason that you expect is crucial1, and it can make the difference between fixing a production bug, and just thinking that you fixed it.

Can we somehow write an assertion in a commit message, and automatically verify it?

Edit: I gave it a go, check out git-check-assertions.

Next: Writing the steps to validate a test in the commit message


  1. This, of course, is more relevant than ever with the advent of agentic programming. An agent can claim that it saw a test fail, but if we force it to encode that fact in a commit message, then we can check that the test fails. ↩︎

sven [at] memcmp.org | mastodon | codeberg