Skip to content

Cache and Continuity

Since 0.39.0, Mull brings a new capability: caching of mutation results. Caching has two different modes: explicit caching and implicit caching (continuity).

The rest of the guide is based on CMake Integration. Make sure you have a local build ready.

Implicit caching is enabled by default and doesn’t require any changes to the config or Mull setup.

The first, cold run records the mutation results in a temporary cache file:

Terminal window
mull-runner --reporters SQLite build.mull/bin/chrono-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 142ms
[info] Applying file path filter (threads: 8)
[################################] 147/147. Finished in 1ms
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 126ms
[info] Running mutants (threads: 8)
[################################] 147/147. Finished in 4.50s
[info] Results can be found at './1795040071.sqlite'
[info] Surviving mutants: 101
[info] Total execution time: 4.87s

The second run reuses the same results and simply skips execution:

Terminal window
mull-runner --reporters SQLite build.mull/bin/chrono-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 144ms
[info] Applying file path filter (threads: 8)
[################################] 147/147. Finished in 1ms
[info] Reusing 147 cached result(s)
[info] Results can be found at './1795040151.sqlite'
[info] Surviving mutants: 101
[info] Total execution time: 216ms

The same mechanism applies when you want to restart an interrupted run:

Terminal window
mull-runner --reporters SQLite build.mull/bin/chrono-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 141ms
[info] Applying file path filter (threads: 8)
[################################] 147/147. Finished in 1ms
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 127ms
[info] Running mutants (threads: 8)
[############--------------------] 59/147. 1.59s
^C
mull-runner --reporters SQLite build.mull/bin/chrono-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 127ms
[info] Applying file path filter (threads: 8)
[################################] 147/147. Finished in 1ms
[info] Reusing 59 cached result(s)
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 126ms
[info] Running mutants (threads: 8)
[################################] 88/88. Finished in 2.77s
[info] Results can be found at './1795040222.sqlite'
[info] Surviving mutants: 101
[info] Total execution time: 3.07s

This can be disabled via the --no-cache flag.

Explicit caching has different behaviour and serves different use cases.

To enable explicit caching you have to specify the path to the cache via the --cache-path foobar.sqlite CLI flag.

Sharing of the results across different binaries

Section titled “Sharing of the results across different binaries”

The first use case covers a scenario where several test suites/binaries share some mutants. If test A killed a mutant, then there is no need to check whether test B also kills it.

The first run (format-test) kills ~100 mutants. The second run (chrono-test) can simply reuse that information and skip many already detected ones. The original, isolated run of chrono-test took ~4.8s, while the cached one completed in ~2.6s.

Terminal window
mull-runner --reporters SQLite --cache-path shared.sqlite build.mull/bin/format-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: shared.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 47ms
[info] Applying file path filter (threads: 8)
[################################] 141/141. Finished in 1ms
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 32ms
[info] Running mutants (threads: 8)
[################################] 141/141. Finished in 1.23s
[info] Results can be found at './1795040852.sqlite'
[info] Surviving mutants: 47
[info] Total execution time: 1.32s
Terminal window
mull-runner --reporters SQLite --cache-path shared.sqlite build.mull/bin/chrono-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: shared.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 137ms
[info] Applying file path filter (threads: 8)
[################################] 147/147. Finished in 1ms
[info] Reusing 81 cached result(s)
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 125ms
[info] Running mutants (threads: 8)
[################################] 66/66. Finished in 2.28s
[info] Results can be found at './1795040903.sqlite'
[info] Surviving mutants: 45
[info] Total execution time: 2.60s

The second use case targets incremental improvement of a test suite. Consider the following scenario: you run the test suite and it detects 90 out of 100 mutants, leaving 10 survivors. You add a new test case and only want to check whether any of those 10 mutants are killed now. The explicit cache enables this: mull detects the rebuilt binary and reruns only the mutants that were not killed (survivors and timeouts).

Terminal window
mull-runner --reporters SQLite --cache-path inc.sqlite build.mull/bin/format-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: inc.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 46ms
[info] Applying file path filter (threads: 8)
[################################] 141/141. Finished in 1ms
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 32ms
[info] Running mutants (threads: 8)
[################################] 141/141. Finished in 1.24s
[info] Results can be found at './1795041413.sqlite'
[info] Surviving mutants: 47
[info] Total execution time: 1.34s
# Rebuild the format-test binary
mull-runner --reporters SQLite --cache-path inc.sqlite build.mull/bin/format-test
[info] Using config /private/tmp/sc-U7p0Z3/mull.yml
[info] Using cache: inc.sqlite
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 455ms
[info] Applying file path filter (threads: 8)
[################################] 141/141. Finished in 1ms
[info] Reusing 86 cached result(s), 52 outdated by a rebuild
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 33ms
[info] Running mutants (threads: 8)
[################################] 55/55. Finished in 879ms
[info] Results can be found at './1795041473.sqlite'
[info] Surviving mutants: 46
[info] Total execution time: 1.40s

Caching is hard. mull fingerprints a run by hashing the test binary itself, the dynamic libraries that contain mutants, and the test program (when it is a file). Runs with different arguments keep separate results.

If the test results depend on anything else (e.g., env, date/time, network availability, etc.), then it is advised to disable caching altogether via the --no-cache CLI flag.

Wrappers such as ctest or scripts need extra care: mull doesn’t fingerprint the test binaries they run. If you only change tests outside of the binary with mutants, mull will reuse old survivors, so delete the cache or use --no-cache. Within a single fresh build (e.g., a CI job), a shared explicit cache is safe: a mutant killed via one binary is killed for all of them.