CMake Integration: fmtlib
This guide demonstrates how to integrate Mull into a CMake-based project.
We use fmtlib as an example.
Step 1. Check out the source code
Section titled “Step 1. Check out the source code”git clone https://github.com/fmtlib/fmt --depth 1Step 2. Create sample Mull config
Section titled “Step 2. Create sample Mull config”Create a file fmt/mull.yml with the following contents:
mutators: - cxx_add_to_subStep 3. Configure and build fmtlib
Section titled “Step 3. Configure and build fmtlib”mkdir fmt/build.dircd fmt/build.direxport CXX=clang++-22cmake \ -DCMAKE_CXX_FLAGS="-O0 -fpass-plugin=/usr/lib/mull-ir-frontend-22 -g -grecord-command-line" \ ..make scan-test -jStep 4. Run Mull against fmtlib tests
Section titled “Step 4. Run Mull against fmtlib tests”mull-runner-22 ./bin/scan-testYou should see similar output:
> mull-runner-22 ./bin/scan-test[info] Using config /tmp/fmt/mull.yml[warning] Could not find dynamic library: libstdc++.so.6[warning] Could not find dynamic library: libm.so.6[warning] Could not find dynamic library: libgcc_s.so.1[warning] Could not find dynamic library: libc.so.6[info] Warm up run (threads: 1) [################################] 1/1. Finished in 4ms[info] Baseline run (threads: 1) [################################] 1/1. Finished in 3ms[info] Running mutants (threads: 8) [################################] 175/175. Finished in 320ms[info] Survived mutants (158/175):/tmp/fmt/include/fmt/base.h:1277:74: warning: Survived: Replaced + with - [cxx_add_to_sub] return static_cast<int>((0x3a55000000000000ull >> (2 * (c >> 3))) & 3) + 1; ^/tmp/fmt/include/fmt/base.h:1290:24: warning: Survived: Replaced + with - [cxx_add_to_sub] value = value * 10 + unsigned(*p - '0'); ^/tmp/fmt/include/fmt/base.h:1299:33: warning: Survived: Replaced + with - [cxx_add_to_sub] return num_digits == digits10 + 1 && ^/tmp/fmt/include/fmt/base.h:1300:31: warning: Survived: Replaced + with - [cxx_add_to_sub] prev * 10ull + unsigned(p[-1] - '0') <= max ^/tmp/fmt/include/fmt/base.h:1805:23: warning: Survived: Replaced + with - [cxx_add_to_sub] try_reserve(size_ + 1); ^/tmp/fmt/include/fmt/base.h:1818:27: warning: Survived: Replaced + with - [cxx_add_to_sub] grow_(*this, size + count); ^/tmp/fmt/include/fmt/base.h:2056:19: warning: Survived: Replaced + with - [cxx_add_to_sub] return count_ + this->size(); ^ <truncated>/tmp/fmt/test/gtest/gmock-gtest-all.cc:12211:36: warning: Survived: Replaced + with - [cxx_add_to_sub] IsUTF8TrailByte(s[i + 2]) && ^/tmp/fmt/test/gtest/gmock-gtest-all.cc:14394:26: warning: Survived: Replaced + with - [cxx_add_to_sub] argv[j] = argv[j + 1]; ^/tmp/fmt/test/gtest/gmock/gmock.h:6218:33: warning: Survived: Replaced + with - [cxx_add_to_sub] return ilhs * num_matchers_ + irhs; ^/tmp/fmt/test/gtest/gtest/gtest.h:3758:19: warning: Survived: Replaced + with - [cxx_add_to_sub] return ~sam + 1; ^/tmp/fmt/test/scan.h:24:44: warning: Survived: Replaced + with - [cxx_add_to_sub] if (c >= 'A' && c <= 'F') return c - 'A' + 10; ^[info] Mutation score: 9%[info] Surviving mutants: 158[info] Total execution time: 385msStep 5. Ignore mutants coming from gtest
Section titled “Step 5. Ignore mutants coming from gtest”We’ve got lots of survived mutants.
We can ignore some of them (specifically the ones coming from gtest and gmock) by extending the config file as follows:
# mull.ymlmutators: - cxx_add_to_subexcludePaths: - .*gtest.* - .*gmock.*No recompilation needed in this case, Mull will pick the changes from the config file and ignore the mutants under gtest/gmock folders.
> mull-runner-22 ./bin/scan-test[info] Using config /tmp/fmt/mull.yml[info] Using config /tmp/fmt/mull.yml[warning] Could not find dynamic library: libstdc++.so.6[warning] Could not find dynamic library: libm.so.6[warning] Could not find dynamic library: libgcc_s.so.1[warning] Could not find dynamic library: libc.so.6[info] Warm up run (threads: 1) [################################] 1/1. Finished in 3ms[info] Applying file path filter (threads: 8) [################################] 175/175. Finished in 4ms[info] Baseline run (threads: 1) [################################] 1/1. Finished in 3ms[info] Running mutants (threads: 8) [################################] 105/105. Finished in 54ms[info] Survived mutants (99/105):/tmp/fmt/include/fmt/base.h:1277:74: warning: Survived: Replaced + with - [cxx_add_to_sub] return static_cast<int>((0x3a55000000000000ull >> (2 * (c >> 3))) & 3) + 1; ^/tmp/fmt/include/fmt/base.h:1290:24: warning: Survived: Replaced + with - [cxx_add_to_sub] value = value * 10 + unsigned(*p - '0'); ^/tmp/fmt/include/fmt/base.h:1299:33: warning: Survived: Replaced + with - [cxx_add_to_sub] return num_digits == digits10 + 1 && ^/tmp/fmt/include/fmt/base.h:1300:31: warning: Survived: Replaced + with - [cxx_add_to_sub] prev * 10ull + unsigned(p[-1] - '0') <= max ^/tmp/fmt/include/fmt/base.h:1805:23: warning: Survived: Replaced + with - [cxx_add_to_sub] try_reserve(size_ + 1); ^/tmp/fmt/include/fmt/base.h:1818:27: warning: Survived: Replaced + with - [cxx_add_to_sub] grow_(*this, size + count); ^<truncated>/tmp/fmt/include/fmt/format.h:3554:63: warning: Survived: Replaced + with - [cxx_add_to_sub] (has_decimal_point ? 1 : 0) + ^/tmp/fmt/include/fmt/format.h:3560:69: warning: Survived: Replaced + with - [cxx_add_to_sub] ptr = format_decimal<Char>(ptr, significand, significand_size + 1); ^/tmp/fmt/include/fmt/format.h:3564:38: warning: Survived: Replaced + with - [cxx_add_to_sub] *ptr++ = static_cast<Char>('0' + significand); ^/tmp/fmt/include/fmt/format.h:3574:38: warning: Survived: Replaced + with - [cxx_add_to_sub] *ptr++ = static_cast<Char>('0' + abs_exponent / 100); ^/tmp/fmt/test/scan.h:24:44: warning: Survived: Replaced + with - [cxx_add_to_sub] if (c >= 'A' && c <= 'F') return c - 'A' + 10; ^[info] Mutation score: 5%[info] Surviving mutants: 99[info] Total execution time: 82msWe get fewer mutants, but the number can be reduced even further.
Step 6. Filter out unreachable mutants
Section titled “Step 6. Filter out unreachable mutants”To filter out unreachable mutants we need to recompile the tests with coverage enabled (via -fprofile-instr-generate -fcoverage-mapping), and then simply rerun the tests.
# recompile with coverage enabledcmake \ -DCMAKE_CXX_FLAGS="-O0 -fpass-plugin=/usr/lib/mull-ir-frontend-22 -g -grecord-command-line -fprofile-instr-generate -fcoverage-mapping" \ ..make scan-test -jRerunning tests shows that all the mutants are actually killed:
> mull-runner-22 -ide-reporter-show-killed ./bin/scan-test[info] Using config /tmp/fmt/mull.yml[warning] Could not find dynamic library: libstdc++.so.6[warning] Could not find dynamic library: libm.so.6[warning] Could not find dynamic library: libgcc_s.so.1[warning] Could not find dynamic library: libc.so.6[warning] Could not find dynamic library: ld-linux-aarch64.so.1[info] Warm up run (threads: 1) [################################] 1/1. Finished in 8ms[info] Applying coverage filter (threads: 8) [################################] 105/105. Finished in 3ms[info] Applying file path filter (threads: 4) [################################] 4/4. Finished in 0ms[info] Baseline run (threads: 1) [################################] 1/1. Finished in 3ms[info] Running mutants (threads: 4) [################################] 4/4. Finished in 5ms[info] Killed mutants (4/4):/tmp/fmt/test/scan.h:23:44: warning: Killed: Replaced + with - [cxx_add_to_sub] if (c >= 'a' && c <= 'f') return c - 'a' + 10; ^/tmp/fmt/test/scan.h:404:16: warning: Killed: Replaced + with - [cxx_add_to_sub] n = n * 10 + static_cast<unsigned>(c - '0'); ^/tmp/fmt/test/scan.h:418:20: warning: Killed: Replaced + with - [cxx_add_to_sub] prev * 10ull + unsigned(prev_digit - '0') <= max) { ^/tmp/fmt/test/scan.h:435:18: warning: Killed: Replaced + with - [cxx_add_to_sub] n = (n << 4) + static_cast<unsigned>(digit); ^[info] Mutation score: 100%[info] All mutations have been killed[info] Total execution time: 97ms