Coco Questions and Answers

Webinar: C/C++ Code Coverage with Coco

You can view a a full recording of the webinar about C/C++ code coverage analysis with Squish Coco.

  • Watch it in MOV or WMV format.

During the Webinar, several questions where answered. Here is a list of all questions with answers.

Integrations

Is there any special integration between the Squish GUI testing tool and Coco – e.g. to correlate test suites/cases/scripts to coverage?

A Knowledge Base article describes how the correlation can be accomplished with a couple of additional script statements.

We’ll work on having the tools exchange their information automatically later this year.

What’s the difference between Squish Coco and QTest library shipped with Qt?

The QTest library is a framework for developing unit tests. Coco will then help to analyze what code was tested (or not) by the former framework.

This means that the two tools complement each other. Browse this report to see both of them in action. Also see the QtTestLib HowTo for technical details.

Is there a predefined possibility to integrate in Buildbot?

A Buildbot integration and other Continuous Integration (CI) systems can be accomplished using a set of command line tools. Reports can be generated in various formats such as HTML, JUnit, EMMA-XML, CSV and others.

For example, ready-made Jenkins solutions are already available, and our support engineers will be happy to assist you with a Buildbot setup.

Analysis

Can reports be run per class too?

The CoverageBrowser method browser contains a tree view for browsing through the classes. The EMMA-XML reports also generate statistics per class.

We are testing an old program with about 1200 classes and 32000 Methods. Is it too big for Coco?

No problem. Projects of that size have been analyzed before.

When it comes to large source code files: the next version comes with big performance enhancements for the code viewer and highlighter.

Overly many “hits” on code can also be an indication of a performance bug. Is there a way to get such a report?

While this is not the tool’s primary use case, such a report can nevertheless be produced: set the required “Coverage Level” to a high value. For example 10000. In this case CoverageBrowser will consider a statement to be validated only if it is executed 10000 times. With this trick is it possible to find the location with the high number of hits.

What is dead code and how is it found?

They are two kinds of dead code:

  • A function that is never called
  • Code which is not reachable in a function

A function which is never called cannot be detected automatically using Squish Coco, but after a test cycle Squish Coco will indicate that the coverage of it is 0%. Making it easy to identify the functions when reviewing the coverage information after the test cycle.

Dead code inside a function is detected using static analysis. For example:

int foo( int v ){if ( v )return 0;elsereturn 1;return -1; //

‘return -1;’ cannot be executed since when executing ‘foo()’ the line ‘return 0;’ or ‘return 1;’ will make the function return.

When running a batch of tests overnight, is there a way to separate the tests into separate executions in the coverage browser?

Yes.

By default, Squish Coco records (and so also displays it in CoverageBrowser) each test-application execution in its own execution.

Note: it is also possible:

  • to record an execution report after each unit test.
  • to label the execution report of each test with a name,
    status (passed/failed) and a comment (free HTML text)

Languages / Compilers

Is mingw-gcc supported?

Yes.

Are C++ template classes handled?

Yes.

Is it possible to use Coco also with older gcc compiler versions e.g. gcc 3.3.6?

Yes. Even versions as old as 2.x work.

Can I use Squish Coco on pure C++ code not using Qt?

Yes. Squish Coco can instrument every C++ application.

Is the tool only available for Qt 5 or are earlier Qt 4.x supported as well?

The switches that enable recognition of some Qt-specific features work with all Qt 3, 4 and 5 versions.

But remember, the tool works with any C and C++ application, even if Qt is not used at all.

Does Coco currently only support C++? Do you have plans to extend this capability to other languages?

A new edition supporting C# is in beta and will be released within a few weeks. Other candidate languages are Tcl, Java, Objective-C, JavaScript and QML. The order of tool support implementation will be based on customer demand.

Is there similar support for testing GUIs written in QML?

Not yet. But given the recent release of Qt 5 we’ll soon investigate possible technical solutions.

Note: Coco’s coverage analysis is language independent. As a result it would be possible to analyze hybrid QML/C++ applications.

Platforms

Does Coco run on Linux?

Yes. And with the exception of the Visual Studio Add-on the feature set is available on Windows, Linux and Mac OS X alike.

How about Android and other mobile and embedded devices?

Once the application is compiled with Squish Coco, executing it on a target produces a coverage report on the device. If generating a coverage report on the target device is not convenient, it is possible to provide custom I/O functions permitting for example the ability to store the information remotely. The following example illustrates using an SFTP server: Custom I/O using SFTP protocol.

Android support is currently limited to C/C++ components. Java support has been investigated and will be implemented based on customer demand.

Support for embedded Windows OS?

Embedded Windows systems are generally supported. One just has to alter the way to instrument the application:

  • If you use Visual Studio to build the application, it works out-of-the-box.
  • Other build environments (like Platform Builder) likely require additional setup steps. Please contact support@froglogic.com for assistance.

Instrumentation

How can I instrument an application when using QMake .pro files?

For Qt we have provided a complete tutorial based on the QTextEdit sample. The minimum requirement to have an application instrumented is to replace the compiler. This can be performed by adding the following lines to the .pro file:

 QMAKE_CC=cs$$QMAKE_CCQMAKE_LINK=cs$$QMAKE_LINKQMAKE_CXX=cs$$QMAKE_CXX

Do I always have to patch the Makefile for an instrumentation?

No. That was the approach taken for the live demonstration. But depending on your build system there are other possible
approaches.

For tools like CMake or QMake, only small additions to the input files are required The Visual Studio Add-On activates the instrumentations after the push of a button Prepend the path of Coco’s so called “compiler wrappers” to the PATH environment variable and set COVERAGESCANNER_ARGS=’–cs-on’.

The demonstration was based on a command line application. Does Coco also support GUI applications?

Yes.

What runtime impact does instrumenting an application have?

The overhead depends on the chosen coverage level to be analyzed. The by far most detailed level (condition/decision) causes the application to run an average of 25% slower.

See the following sample, demonstrating a Quicksort implementation: benchmark

Are projects with shared libraries (DLL / .so) covered? Can coverage of the code in libraries be instrumented?

Yes. Library and application code is instrumented in almost the same way.

Remark: To automatically dump coverage info from a library, the application using the library should be instrumented as well.

Is the instrumentation code something that can be changed in order to store the coverage information in an embedded system in our desired location/mechanism. We would then like to extract the date from the embedded system.

Squish Coco provides a C API which permits saving the coverage report using custom I/O functions. Here 2 samples:

  • Writing a report into a file. This sample helps to understand and play with the API.
  • Writing a report on a SFTP server. This is a real-life example, saving a report on a remote SFTP server.

Miscellaneous

What are the differences between Coco and Gcov/lcov?

The main differences are:

  • Gcov’s coverage level is limited to branch coverage. I.e. it does not include condition/decision coverage.
  • Because Gcov does not work at the source code level, it loses information and can produce inaccurate or incorrect results in optimized builds.
  • Coco has a graphical interface for interactive code coverage data analysis.
  • Coco is able to handle C++ templates and inline functions.
  • Coco compares the analysis between two software releases: finding the list of tests (to re-execute) impacted by a
    modification, verifying the modified code is correctly tested by hiding the analysis of untested code.
  • Coco records a code coverage report from each unit test or application test separately. This allows the selection and
    comparison of individual tests.
  • HTML reports for GCov are only available through an additional tool (lcov) and are less detailed.