Skip to content

Source Level Mutations

Since 0.39.0, Mull comes with an extra tool, mull-rewriter-N.

It is used for differential testing of mull itself and has already found some critical bugs, but it can also be helpful if you want to generate a mutated version of the program at the source level.

The rest of the guide is based on CMake Integration. Make sure you have a local build ready.

Mull generates mutations at the LLVM IR/Bitcode level, and the only way to actually see the mutated version of the original program is by applying patches.

With mull-rewriter it is possible to generate a “metamutant”. It takes several steps:

  1. Ensure that the build system generated a compilation database, e.g. -DCMAKE_EXPORT_COMPILE_COMMANDS=ON for CMake.

  2. Generate an SQLite report; a dry run is sufficient:

Terminal window
> mull-runner --dry-run --reporters SQLite --report-name dry-run build.dir/bin/chrono-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 445ms
[info] Applying file path filter (threads: 8)
[################################] 147/147. Finished in 1ms
[info] Dry run (threads: 1)
[################################] 1/1. Finished in 0ms
[info] Results can be found at './dry-run.sqlite'
[info] Total dry run time: 455ms
  1. Run mull-rewriter to modify the original program:
Terminal window
> mull-rewriter-23 --report dry-run.sqlite -p build.dir --root $PWD --in-place
[1/2] Processing file /tmp/sc-U7p0Z3/fmt/test/chrono-test.cc.
[2/2] (1/2) Processing file /tmp/sc-U7p0Z3/fmt/src/format.cc.
[2/2] (2/2) Processing file /tmp/sc-U7p0Z3/fmt/src/format.cc.
[1/30] Processing file /tmp/sc-U7p0Z3/fmt/test/xchar-test.cc.
[2/30] Processing file /tmp/sc-U7p0Z3/fmt/test/ranges-odr-test.cc.
[3/30] Processing file /tmp/sc-U7p0Z3/fmt/test/compile-test.cc.
...
[27/30] Processing file /tmp/sc-U7p0Z3/fmt/test/format-test.cc.
[28/30] Processing file /tmp/sc-U7p0Z3/fmt/test/unicode-test.cc.
[29/30] Processing file /tmp/sc-U7p0Z3/fmt/test/os-test.cc.
[30/30] Processing file /tmp/sc-U7p0Z3/fmt/test/c-test.c.
Not rewritten (expression not found, e.g. in a macro): cxx_add_to_sub:/private/tmp/sc-U7p0Z3/fmt/test/chrono-test.cc:1085:71:1085:72
Rewrote 146 of 147 mutants in 5 files
  1. Rebuild the program:
Terminal window
cmake --build build.dir

That’s it: now the compiled artifacts should contain (almost) the same mutations as the ones generated at the LLVM IR/Bitcode level.

Here is an example of a modified program: https://github.com/fmtlib/fmt/compare/main…mull-project:fmt:mull-source-mutation