Setup & Configuration

Mutation testing with Mull consists of at most three steps:

• mutant generation
• mutant execution
• report generation (optional, can be done during execution)

Let’s reuse the same code from the Quick Primer:

main.c

#include <assert.h>

// Returns 1 if value is in the closed interval [min, max], 0 otherwise
int in_range(int value, int min, int max) {
  return value >= min && value < max;
}

int main(void) {
  assert(in_range(5,  1, 10) == 1);  // mid-range: both conditions true
  assert(in_range(0,  1, 10) == 0);  // below min: first condition false
  assert(in_range(15, 1, 10) == 0);  // above max: second condition false
  return 0;
}

Mutant Generation

Mull is implemented as a Clang/LLVM plugin, so mutants are generated at compile time.

In the Quick Primer example we’ve seen that enabling Mull was as easy as adding a few compiler flags to the compile command:

# mutant generation
clang-22 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  main.c -o range_tests

More specifically, the important bits are:

• -fpass-plugin=/usr/lib/mull-ir-frontend-22 this is the core of Mull, the plugin version must match the Clang version used to compile the code
• -g -grecord-command-line to function properly, Mull needs debug information such that mutants generated at the very low-level can be mapped back to the high-level source code.

That’s all it takes, but Mull uses some default configuration which is a good start, but your project likely needs special treatment.

The plugin is configured using mull.yml file placed at the current working directory, alternatively you can pass the path to the file using environment variable MULL_CONFIG=path-to-file.yml:

# create some configs
touch mull.yml
touch mull-ci.yml

# uses mull.yml by default
clang-22 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  main.c -o range_tests

# uses mull-ci.yml
env MULL_CONFIG=mull-ci.yml clang-22 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  main.c -o range_tests

There are quite a few options of what goes into the config, but you’ll normally just need a couple of those. See the full reference here: YAML Configuration.

Mutant Execution

Once the binary is compiled we can run mutation analysis using mull-runner.

This time let’s look at both killed and survived mutants (using -ide-reporter-show-killed):

# mutant generation
> clang-22 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  main.c -o range_tests
# mutant execution
> mull-runner-22 -ide-reporter-show-killed range_tests
[info] Using config /workspaces/mull/demo/mull.yml
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Running mutants (threads: 7)
       [################################] 7/7. Finished in 3ms
[info] Killed mutants (5/7):
/workspaces/mull/demo/main.c:5:16: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Killed: Replaced < with >= [cxx_lt_to_ge]
  return value >= min && value < max;
                               ^
/workspaces/mull/demo/main.c:9:30: warning: Killed: Replaced with != [cxx_eq_to_ne]
  assert(in_range(5,  1, 10) == 1);  // mid-range: both conditions true
                             ^
/workspaces/mull/demo/main.c:10:30: warning: Killed: Replaced with != [cxx_eq_to_ne]
  assert(in_range(0,  1, 10) == 0);  // below min: first condition false
                             ^
/workspaces/mull/demo/main.c:11:30: warning: Killed: Replaced with != [cxx_eq_to_ne]
  assert(in_range(15, 1, 10) == 0);  // above max: second condition false
                             ^
[info] Survived mutants (2/7):
/workspaces/mull/demo/main.c:5:16: warning: Survived: Replaced >= with > [cxx_ge_to_gt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Survived: Replaced < with <= [cxx_lt_to_le]
  return value >= min && value < max;
                               ^
[info] Mutation score: 71%
[info] Surviving mutants: 2
[info] Total execution time: 9ms

Note

Please, note the Using config /workspaces/mull/demo/mull.yml in the output.

mull-runner (and mull-reporter) also uses mull.yml for configuration when applicable, some options are controlled via CLI flags.

Full reference to the CLI flags can be found here:

• mull-runner CLI ref
• mull-reporter CLI ref.

It is recommended to use the same config for mutant generation and mutant execution. Using different configs can lead to mismatches — for example, mutants that were generated but not expected by the runner, or filters applied at execution that were not reflected at compile time — resulting in confusing or incorrect results.

In total, Mull generated 7 mutants, 5 of which were killed.

Strictly speaking, the cxx_eq_to_ne mutants belong to the test suite and usually you want to avoid mutating the test suite itself.

There are several ways to disable/ignore those mutants (see Filters), but for this example we are just going to disable mutation operators producing those mutants.

Modify mull.yml as follows:

mull.yml

mutators:
  - cxx_boundary    # all mutators from the group
  - cxx_comparison  # all mutators from the group
ignoreMutators:
  - cxx_eq_to_ne    # single mutator

Rebuild & rerun:

# mutant generation
> clang-22 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  main.c -o range_tests
# mutant execution
> mull-runner-22 -ide-reporter-show-killed range_tests
[info] Using config /workspaces/mull/demo/mull.yml
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Running mutants (threads: 4)
       [################################] 4/4. Finished in 2ms
[info] Killed mutants (2/4):
/workspaces/mull/demo/main.c:5:16: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Killed: Replaced < with >= [cxx_lt_to_ge]
  return value >= min && value < max;
                               ^
[info] Survived mutants (2/4):
/workspaces/mull/demo/main.c:5:16: warning: Survived: Replaced >= with > [cxx_ge_to_gt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Survived: Replaced < with <= [cxx_lt_to_le]
  return value >= min && value < max;
                               ^
[info] Mutation score: 50%
[info] Surviving mutants: 2
[info] Total execution time: 6ms

Now we only have relevant mutants.

The full list of support mutation operators can be found here: Mutation Operators.

Report Generation

For simple cases mull-runner has all you need. In fact, it supports all the Reporters and combines mutant execution and report generation in a single tool.

However, there are cases when you may want to aggregate data across several runs, several binaries, or even several machines.

To do so, one needs to “export” mutant execution state and then use mull-reporter to generate a combined report.

This is especially useful for cases with Multiple Test Targets.

The usage is simple:

# mutant execution
> mull-runner-22 -reporters SQLite -report-name range_tests range_tests
[info] Using config /workspaces/mull/demo/mull.yml
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 2ms
[info] Running mutants (threads: 4)
       [################################] 4/4. Finished in 32ms
[info] Results can be found at './range_tests.sqlite'
[info] Surviving mutants: 2
[info] Total execution time: 75ms

# report generation
> mull-reporter-22 -ide-reporter-show-killed range_tests.sqlite
[info] Using config /workspaces/mull/demo/mull.yml
[info] Killed mutants (2/4):
/workspaces/mull/demo/main.c:5:16: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Killed: Replaced < with >= [cxx_lt_to_ge]
  return value >= min && value < max;
                               ^
[info] Survived mutants (2/4):
/workspaces/mull/demo/main.c:5:16: warning: Survived: Replaced >= with > [cxx_ge_to_gt]
  return value >= min && value < max;
               ^
/workspaces/mull/demo/main.c:5:32: warning: Survived: Replaced < with <= [cxx_lt_to_le]
  return value >= min && value < max;
                               ^
[info] Mutation score: 50%
[info] Surviving mutants: 2
[info] Total reporting time: 1ms

Now you have everything you need to apply mutation testing to your project.

For more use cases, consider looking at the guides and references.