Mutants are normally known only from biology and horror movies. Here they are the subject of a software testing method than can complete code coverage analysis.
Artificial faults (mutations) are applied to a program followed by a run of the test suite. Depending on whether any of the tests detects the mutation the suite can be considered complete or require extension. That is the theory at least.
If a mutation makes one or more tests fail (that includes crashes and hangs) the mutation is considered to be killed. That is the expected positive outcome denoting tests of quality.
Mutants not caught by running a test suite are considered to be survivors. Intuitively, one would consider the test suite to be incomplete and require a more thorough test to detect the mutation. Up unto the point where every single mutation is caught. Alas, not every mutation will have an impact that can be detected. See below.
Some mutations do not alter the semantics of a program. Therefore they cannot be detected automatically and require manual investigation. The resulting effort make up the biggest drawback of this testing methology.
A paper published by David Schuler and Andreas Zeller presents findings on the impact of equivalent mutants for seven sample Java programs:
- It takes 15 minutes to assess one single mutation.
- 45% of all uncaught mutations are equivalent
- The problem gets worse as the test suite improves.
How can code coverage analysis help?
Code coverage analysis can contribute in two valuable ways:
First, mutations affecting code not covered by any test cannot possibly be detected. By omitting the respective mutations from analysis compilation and execution time can be saved.
Secondly, the impact of a mutation on code coverage can help to tell apart equivalent from non-equivalent mutations. Again this results in saving precious time.