Quick Primer

Let’s consider the following improvised very safety-critical piece with a sophisticated “test suite” with three “test cases”:

main.c

#include <assert.h>

// Returns 1 if value is in the closed interval [min, max], 0 otherwise
int in_range(int value, int min, int max) {
  return value >= min && value < max;
}

int main(void) {
  assert(in_range(5,  1, 10) == 1);  // mid-range: both conditions true
  assert(in_range(0,  1, 10) == 0);  // below min: first condition false
  assert(in_range(15, 1, 10) == 0);  // above max: second condition false
  return 0;
}

All the classical coverage metrics (line, statement, branch, MC/DC even) show 100% coverage, yet a couple of important test cases are missing, and there is also a classical bug in the implementation.

Let’s run the tests using Mull. For each mutant, Mull reports whether it was killed (the test suite caught the change and failed) or survived (the tests passed despite the change, indicating a gap in test coverage):

> clang-22 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  main.c -o range_tests
> mull-runner-22 range_tests
[info] Using config /workspaces/mull/demo/mull.yml
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 3ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Running mutants (threads: 7)
       [################################] 7/7. Finished in 3ms
[info] Survived mutants (2/7):
/workspaces/mull/demo/main.c:5:16: warning: Survived: Replaced >= with > [cxx_ge_to_gt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Survived: Replaced < with <= [cxx_lt_to_le]
  return value >= min && value < max;
                               ^
[info] Mutation score: 71%
[info] Surviving mutants: 2
[info] Total execution time: 34ms

The Survived Mutants section above shows precisely the places which are not covered with tests.

Why and how does it work?

Mull is a tool for Mutation Testing (also known as Mutation Analysis), which is a form of Fault Injection.

The premise of mutation testing is that a good test suite should catch changes to the semantics of the underlying implementation.

Mull works by generating several versions of the original program, so called mutants, by introducing slight changes. Whenever a mutant behaves differently from the original program, it is said to be detected (of killed). If the change in the behavior is not observed, then the mutant is considered as survived.

In the example above, Mull tells us that changing certain conditions doesn’t affect the test results.

You can easily see the list of all applied mutants.

> mull-runner-22 -ide-reporter-show-killed range_tests
[info] Using config /workspaces/mull/demo/mull.yml
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Running mutants (threads: 7)
       [################################] 7/7. Finished in 7ms
[info] Killed mutants (5/7):
/workspaces/mull/demo/main.c:5:16: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Killed: Replaced < with >= [cxx_lt_to_ge]
  return value >= min && value < max;
                               ^
/workspaces/mull/demo/main.c:9:30: warning: Killed: Replaced with != [cxx_eq_to_ne]
  assert(in_range(5,  1, 10) == 1);  // mid-range: both conditions true
                             ^
/workspaces/mull/demo/main.c:10:30: warning: Killed: Replaced with != [cxx_eq_to_ne]
  assert(in_range(0,  1, 10) == 0);  // below min: first condition false
                             ^
/workspaces/mull/demo/main.c:11:30: warning: Killed: Replaced with != [cxx_eq_to_ne]
  assert(in_range(15, 1, 10) == 0);  // above max: second condition false
                             ^
[info] Survived mutants (2/7):
/workspaces/mull/demo/main.c:5:16: warning: Survived: Replaced >= with > [cxx_ge_to_gt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Survived: Replaced < with <= [cxx_lt_to_le]
  return value >= min && value < max;
                               ^
[info] Mutation score: 71%
[info] Surviving mutants: 2
[info] Total execution time: 13ms

Some mutations in this example make less sense, but Mull has many knobs and dials to control the behavior precisely.

Take a look at the guides and the references to learn more.