Mutation Testing in C#: Evaluate Unit Test Quality

Mutation testing is an advanced testing technique that evaluates the actual effectiveness of your unit tests by deliberately injecting bugs into your source code to see if your tests detect them. While traditional code coverage only measures which lines of code are executed during test runs, mutation testing assesses whether those tests are capable of identifying logical errors. This guide explains the core concepts of mutation testing, contrasts it with traditional code coverage, and demonstrates how to implement it in C# using Stryker.NET to ensure your test suite provides genuine quality assurance.

The Limitation of Traditional Code Coverage

Standard code coverage metrics—such as line, branch, or statement coverage—measure only execution, not verification. A test suite can achieve 100% code coverage without containing a single assertion, simply by executing application code paths without checking the outputs. Consequently, high code coverage often creates a false sense of security, leaving edge cases and critical business logic vulnerable to uncaught regressions.

What Is Mutation Testing?

Mutation testing shifts the focus from measuring test execution to measuring test quality. It operates on a simple premise: if a fault is introduced into your code, at least one of your unit tests should fail.

During mutation testing, a testing framework creates multiple variations of your compiled or source code, known as mutants. Each mutant contains a small, intentional syntactic modification (a mutation). The framework runs your existing test suite against each mutant:

  • Killed Mutant: At least one test fails when run against the modified code. This indicates that your tests successfully caught the fault.
  • Survived Mutant: All tests pass despite the introduced fault. This indicates a gap in your assertions or missing test cases.
  • Timed Out / Error: The mutant caused an infinite loop or compilation error and is accounted for separately.

The primary metric produced is the Mutation Score, calculated as:

\[\text{Mutation Score} = \left( \frac{\text{Killed Mutants}}{\text{Total Mutants}} \right) \times 100\]

A higher mutation score reflects a more resilient and thorough test suite.

Common Mutation Operators

Mutation frameworks apply predefined modification rules called mutation operators. Common examples in C# include:

  • Arithmetic Operators: Replacing + with -, or * with /.
  • Equality Operators: Changing == to !=, or >= to >.
  • Logical Operators: Swapping && with ||.
  • Boolean Literals: Flipping true to false.
  • Statement Removal: Removing a method call, assignment, or return value to check if side effects are tested.

Implementing Mutation Testing in C# with Stryker.NET

In the .NET ecosystem, Stryker.NET is the standard open-source framework for mutation testing. It supports .NET Core, .NET 5+, and .NET Framework, integrating with popular testing frameworks such as xUnit, NUnit, and MSTest.

Step 1: Install Stryker.NET

Stryker.NET can be installed globally as a .NET tool via the command line:

dotnet tool install -g dotnet-stryker

Alternatively, you can install it locally within a project directory by using a tool manifest:

dotnet new tool-manifest
dotnet tool install dotnet-stryker

Step 2: Run Stryker

Navigate to the directory of your unit test project and execute:

dotnet stryker

Stryker automatically identifies the target project under test, introduces mutations, executes the test suite for each mutant, and generates an execution report.

Step 3: Analyze the Results

Once the process completes, Stryker outputs a mutation score directly in the terminal and generates an interactive HTML report located in the StrykerOutput folder.

The report details:

  • The exact source code lines where mutations were generated.
  • The nature of each mutation (e.g., binary expression altered).
  • Which mutants survived and which were killed.
  • The specific tests that killed each mutant.

If a mutant survived, you should inspect the relevant code and test:

  1. Missing Assertion: The test executes the code but does not assert on the output or state change affected by the mutation.
  2. Missing Test Case: A specific boundary or conditional branch is not being tested at all.
  3. Equivalent Mutant: In rare cases, the mutation alters the code syntax without changing its semantic behavior; these do not indicate test deficiencies.

Best Practices for C# Mutation Testing

  • Focus on Domain Logic: Running mutation tests across entire solutions can be resource-intensive. Target core business logic, domain models, and calculation engines rather than auto-generated code, DTOs, or thin wrappers.
  • Integrate Selectively into CI/CD: Because running hundreds of variations takes significantly longer than standard unit test execution, configure Stryker to run on pull requests using its --since flag (which only mutates changed files) or schedule full runs during nightly builds.
  • Set Realistic Thresholds: Aiming for a 100% mutation score across an entire application is often impractical. Instead, establish baseline thresholds (e.g., a break threshold of 70% or 80%) using stryker-config.json to prevent quality regressions in critical assemblies.