Makefile Integration: OpenSSL

This guide demonstrates how to integrate Mull into a custom Makefile-based build system.

We use OpenSSL as an example.

Step 1. Check out the source code

git clone https://github.com/openssl/openssl.git \
  --branch openssl-3.0.1 \
  --depth 1

Step 2. Create sample Mull config

Create a file openssl/mull.yml with the following contents:

mutators:
  - cxx_remove_void_call
  - cxx_replace_scalar_call

Step 3. Configure and build OpenSSL

cd openssl
export CC=clang-22
./config -O0 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line
make build_generated -j
make ./test/bio_enc_test -j

Step 4. Run Mull against OpenSSL’s tests

mull-runner-22 ./test/bio_enc_test

You should see similar (and pretty long) output:

[info] Using config /tmp/openssl/mull.yml
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 123ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 89ms
[info] Running mutants (threads: 8)
       [################################] 1671/1671. Finished in 29.12s
[info] Killed mutants (1/1671):
/tmp/openssl/test/testutil/driver.c:367:24: warning: Killed: Replaced + with - [cxx_add_to_sub]
                j = (j + jstep) % all_tests[i].num;
                       ^
[info] Survived mutants (1670/1671):
/tmp/openssl/apps/lib/opt.c:1126:15: warning: Survived: Replaced + with - [cxx_add_to_sub]
        i = 2 + (int)strlen(o->name);
              ^
/tmp/openssl/apps/lib/opt.c:1128:20: warning: Survived: Replaced + with - [cxx_add_to_sub]
            i += 1 + strlen(valtype2param(o));
                   ^
/tmp/openssl/crypto/aes/aes_core.c:433:23: warning: Survived: Replaced + with - [cxx_add_to_sub]
        s[0] = s0[0*4 + r];
                      ^
/tmp/openssl/crypto/aes/aes_core.c:434:23: warning: Survived: Replaced + with - [cxx_add_to_sub]
        s[1] = s0[1*4 + r];
                  ^
<truncated>
/tmp/openssl/test/testutil/format_output.c:283:47: warning: Survived: Replaced + with - [cxx_add_to_sub]
    l2 = bn2 == NULL ? 0 : (BN_num_bytes(bn2) + (BN_is_negative(bn2) ? 1 : 0));
                                              ^
/tmp/openssl/test/testutil/format_output.c:301:32: warning: Survived: Replaced + with - [cxx_add_to_sub]
    len = ((l1 > l2 ? l1 : l2) + bytes - 1) / bytes * bytes;
                               ^
/tmp/openssl/test/testutil/random.c:24:54: warning: Survived: Replaced + with - [cxx_add_to_sub]
    test_random_state[pos] += test_random_state[(pos + 28) % 31];
                                                     ^
[info] Mutation score: 0%
[info] Surviving mutants: 1670
[info] Total execution time: 29.48s

Mull created 1671 mutants for this program. Almost all of them survived, and the reason is that most of these mutants are not relevant for the test cases we are running here.

The problem is that bio_enc_test only exercises a small part of OpenSSL, yet Mull generated mutants across the entire compiled codebase — including code that this test never even reaches. Surviving mutants in unreachable code are noise: they don’t tell us anything useful about test quality.

Step 5. Enable coverage to decrease the number of mutants

Mull can filter out irrelevant mutants by using coverage/profile information. When the binary is instrumented with coverage, Mull records which lines are actually executed during the test run and skips mutants in lines that were never reached. This reduces the mutant count to only those that are meaningful for the tests being run.

The only change needed is to tell clang to emit coverage info (via -fprofile-instr-generate -fcoverage-mapping):

make clean
./config -O0 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  -fprofile-instr-generate -fcoverage-mapping
make build_generated -j
make ./test/bio_enc_test -j

And then rerun tests using Mull again:

> mull-runner-22 --ide-reporter-show-killed  ./test/bio_enc_test
[info] Using config /tmp/openssl/mull.yml
[warning] Could not find dynamic library: libc.so.6
[warning] Could not find dynamic library: ld-linux-aarch64.so.1
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 98ms
[info] Applying coverage filter (threads: 8)
       [################################] 1671/1671. Finished in 10ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 85ms
[info] Running mutants (threads: 5)
       [################################] 5/5. Finished in 89ms
[info] Killed mutants (1/5):
/tmp/openssl/test/testutil/driver.c:367:24: warning: Killed: Replaced + with - [cxx_add_to_sub]
                j = (j + jstep) % all_tests[i].num;
                       ^
[info] Survived mutants (4/5):
/tmp/openssl/test/testutil/driver.c:383:53: warning: Survived: Replaced + with - [cxx_add_to_sub]
                                 subtest_case_count + 1, j + 1);
                                                    ^
/tmp/openssl/test/testutil/driver.c:383:60: warning: Survived: Replaced + with - [cxx_add_to_sub]
                                 subtest_case_count + 1, j + 1);
                                                           ^
/tmp/openssl/test/testutil/driver.c:398:66: warning: Survived: Replaced + with - [cxx_add_to_sub]
                test_verdict(verdict, "%d - %s", test_case_count + 1,
                                                                 ^
/tmp/openssl/test/testutil/random.c:24:54: warning: Survived: Replaced + with - [cxx_add_to_sub]
    test_random_state[pos] += test_random_state[(pos + 28) % 31];
                                                     ^
[info] Mutation score: 20%
[info] Surviving mutants: 4
[info] Total execution time: 2.25s

The output is much more concise and manageable now.