froglogic / Coco • Code Coverage and Test Coverage Tool / Coco Coverage Levels

Coco Coverage Levels

Coverage Levels

Supported Code Coverage Levels

Coco supports the following metrics:

Function Coverage

Definition

With this metric, Squish Coco counts which functions were called and how often. “Functions” includes also the member functions (or methods) in object-oriented programming languages like C++.

Use Case

While this metric does not provide many details it can nevertheless be useful for an initial assessment of a project’s test coverage status.

Relevance for Safety Standards

IEC 61508 highly recommends 100% structural test coverage of entry points (i.e. functions) for SIL 1, 2, 3 and 4.

Back to top

Line Coverage

Definition

The Line Coverage of a program is the number of executed lines divided by the total number of lines. Only lines that contain executable statements are considered, not those with pure declarations.

Advantage

Line Coverage is very suitable for both communicating and visualizing a coverage overview. A developer’s statement “I have 80% of my software’s lines covered through unit tests.” is easy to understand for the human recipient. Especially if it goes along with a view of the source code that has the lines colored in green and red, respectively.

The simplicity that is good for conveying a first impression has its drawbacks in terms of precision however. See the following sections for details.

Formatting Dependency

Line coverage is an unstable coverage measure since it depends strongly on the code formatting. Two variations of a simple C function will serve  as an example:

Variant 1: Concise version with two lines of executable code:

double kelvin_to_celcius(double kelvin)
{
    if (kelvin < 0.0) return -1;
    return kelvin - 273.15;
}


Variant 2: Reformatted version that got expanded to three lines of executable code:

double kelvin_to_celcius(double kelvin)
{
    if (kelvin < 0.0)
        return -1;
    return kelvin - 273.15;
}


A simple change of the coding style leads to a coverage change from 50% to 66.7% for a call like

kelvin_to_celcius(-10.0);


that tests the error handling!

Disguised Control Flow

Let’s assume an invocation of Variant 1 with a succesful conversion:

kelvin_to_celcius(250.0);


Both lines will be executed thus resulting in 100% coverage. Have we completed our testing? Nay! The error case needs as a test as well. Alas, the possible branching of the code flow is not part of what Line Coverage cares about.

Relevance for Safety Standards

None.  Due to the limitations described above Line Coverage does not provide any value over levels like Branch Coverage or – more importantly – MC/DC. Common standards like ISO 26262 for functional safety therefore speak out recommendations for the levels like the latter.

Alternatives to Line Coverage

The formatting dependency is remediated by Statement Coverage. One needs to switch to Branch Coverage to address execution flow branching. If one also cares about logic expressed through boolean expressions (leading to branching) we recommend to measure a Condition Coverage metric like MC/DC.

Back to top

Statement Coverage

Definition

Statement coverage tracks the executed program statements. The metric value of a program is the number of executed statements in it, divided by the total number of statements.

Statement Blocks

A block groups a sequence of statements and treats them like a single statement. In the C-family of languages curly braces ({ and }) are used for grouping.

A basic block is a statement block with no jump instructions in it. Given a single entry and a single exit point only Statement Coverage can be simplified by basing calculations on complete blocks.

Advantage

Unlike Line Coverage this metric is robust against plain code formatting changes. After all, many programming languages allow for multiple statements in a single line. Splitting of lines should not influence the coverage value.

Relation to other metrics

100% Statement Coverage implies 100% Line Coverage. But the opposite is not true: a line may contain multiple statements whereas a test may not succeed in executing all of them. The following example shows a line with two statements where the latter one is skipped if the variable done  is false from the start.

while (!done) { readMoreData(); }

Shortcomings

A plain count of executed statements does not take possible branching points and input conditions for branching decisions into consideration.

if (config->loggingEnabled())
    log("Will start to process data");
startProcessing();


A single test will be able to execute all of the above three statements. Thus resulting in 100% coverage at this level. But is it complete? The case of loggingEnabled() returning a false boolean value requires a second test.  To exercise the else branch that would become visible when transforming this code into a flow diagram.

Relevance for Safety Standards

ISO 26262 highly recommends Statement Coverage for ASIL A and B. ASIL C and D recommend it as well but the more strict Branch Coverage and Modified Condition/Decision Coverage are highly recommended instead.

EN 50128 recommends this metric for SIL 0.

IEC 61508 recommends this metric for SIL 1 and highly recommends it for SIL 2, 3 and 4.

Back to top

Copy link
Powered by Social Snap