Multiple Test Targets

The integration guides (CMake, Makefile) are focusing on a single test suite, but in the real world scenarios this is usually not the case.

What we typically have is a piece of software and several test targets, covering different aspects of the said software. Both fmt and openssl test suites cover such a use case.

To handle this Mull has a separate tool called mull-reporter which can be used for offline analysis of results.

The usage is straightforward: accumulate the mutation results across several runs into a single SQLite database, and then let the reporter analyze the database and show the results.

Here is an example of how to use it with fmt. The example is based on the CMake guide.

# build fmt tests
make scan-test std-test
# run the tests
mull-runner-22 bin/scan-test
mull-runner-22 bin/std-test

Running these commands would generate two reports. However, a mutant A might survive in the context of scan-test, but be killed by std-test. In this case, the mutation report is misleading.

To tackle this problem, Mull has a separate tool mull-reporter which allows aggregation of multiple results into a single one using SQLite reporter as an intermediary. The following commands

# run the tests
mull-runner-22 --reporters SQLite --report-name fmt bin/scan-test
mull-runner-22 --reporters SQLite --report-name fmt bin/std-test
# generate combined report
mull-reporter-22 fmt.sqlite

produce the following report:

[info] Using config /tmp/fmt/mull.yml
[info] Survived mutants (3/16):
/tmp/fmt/include/fmt/format-inl.h:1136:62: warning: Survived: Replaced + with - [cxx_add_to_sub]
            (cache.high() >> (num_significand_bits<double>() + 2))) >>
                                                             ^
/tmp/fmt/include/fmt/format-inl.h:1143:62: warning: Survived: Replaced + with - [cxx_add_to_sub]
            (cache.high() >> (num_significand_bits<double>() + 1))) >>
                                                             ^
/tmp/fmt/include/fmt/format-inl.h:1789:34: warning: Survived: Replaced + with - [cxx_add_to_sub]
    auto lower_end = lower_start + s.lower_count;
                                 ^
[info] Mutation score: 81%
[info] Surviving mutants: 3
[info] Total reporting time: 9ms

Mull collects all the results into a single SQLite database, and then uses aggregated data to produce a final result, leading to much more accurate report.