Custom Test Suites

The goal of this guide is to demonstrate how to use Mull with ‘non-standard’ test suites, such as when the test suite is a separate program. The best example is integration tests written in interpreted languages.

Tests in interpreted languages

Consider the following (absolutely synthetic) program under test:

main.c

extern int printf(const char *, ...);
extern int strcmp(const char *, const char *);

int test1(int a, int b) {
  return a + b;
}

int test2(int a, int b) {
  return a * b;
}

int main(int argc, char **argv) {
  if (argc == 1) {
    printf("NOT ENOUGH ARGUMENTS\n");
    return 1;
  }
  if (strcmp(argv[1], "first test") == 0) {
    if (test1(2, 5) == 7) {
      printf("first test passed\n");
      return 0;
    } else {
      printf("first test failed\n");
      return 1;
    }
  } else if (strcmp(argv[1], "second test") == 0) {
    if (test2(2, 5) == 10) {
      printf("second test passed\n");
      return 0;
    } else {
      printf("second test failed\n");
      return 1;
    }
  } else {
    printf("INCORRECT TEST NAME %s\n", argv[1]);
    return 1;
  }
  return 0;
}

The program accepts a command-line argument, and depending on the value of the argument it either runs one of the tests or exits with an error. Here is an example:

> clang-22 main.c -o test
> ./test
NOT ENOUGH ARGUMENTS
> ./test "first test"
first test passed
> ./test "second test"
second test passed
> ./test "third test"
INCORRECT TEST NAME third test

Running these tests manually is a tedious and error-prone process, so we create a separate test runner:

test.py

import sys
import subprocess

test_executable = sys.argv[1]

subprocess.run([test_executable, "first test"], check=True)
subprocess.run([test_executable, "second test"], check=True)

The script takes the program under test as its argument and runs the tests against that program.

> clang-22 main.c -o test
> python3 test.py ./test
first test passed
second test passed

Usage of Mull in this case is very similar to a “typical” use-case (see Setup & Configuration).

1. Create config file mull.yml:

mull.yml

mutators:
  - cxx_add_to_sub
  - cxx_mul_to_div

2. Generate mutated executable

clang-22 \
  -fpass-plugin=/usr/lib/mull-ir-frontend-22 \
  -g -grecord-command-line \
  main.c -o test.exe

3. Run analysis using mull-runner:

> mull-runner-22 ./test.exe -ide-reporter-show-killed \
  -test-program=python3 -- test.py ./test.exe
[info] Using config /tmp/sc-kGN35Gr1f/mull.yml
[info] Warm up run (threads: 1)
        [################################] 1/1. Finished in 347ms
[info] Filter mutants (threads: 1)
        [################################] 1/1. Finished in 0ms
[info] Baseline run (threads: 1)
        [################################] 1/1. Finished in 76ms
[info] Running mutants (threads: 2)
        [################################] 2/2. Finished in 81ms
[info] Killed mutants (2/2):
/tmp/sc-kGN35Gr1f/main.c:5:12: warning: Killed: Replaced + with - [cxx_add_to_sub]
  return a + b;
           ^
/tmp/sc-kGN35Gr1f/main.c:9:12: warning: Killed: Replaced * with / [cxx_mul_to_div]
  return a * b;
           ^
[info] All mutations have been killed
[info] Mutation score: 100%
[info] Total execution time: 509ms

Note, test.exe appears twice in the arguments list: the first appearance is required for mull-runner to extract the mutants generated at the second step. The second appearance is passed as an argument to the test program test.py.