Coco Case Study: InnovMetric

InnovMetric

Founded in 1994 and headquartered in Quebec, QC, Canada, with subsidiaries worldwide, InnovMetric Software Inc. is the leading provider of universal 3D metrology software solutions. The world’s largest industrial manufacturing organizations trust InnovMetric’s PolyWorks® software solutions to maximize the benefits of 3D measurement technologies for their engineering and manufacturing applications. One key element of PolyWorks is its powerful macro-programming language that allows customers to develop and deploy customized workflows and automated processes. This language is also heavily used internally to automate the testing process while developing new functionalities or refactoring software components.

 

Since 2003, InnovMetric has invested deeply into functional testing and continuous integration.  About 1,000 tests are written and added to a framework every year. There are currently over 15,000 test scripts, and to run them all takes over 14 hours. Running each of the tests daily helps them notice if there are any regressions, but until recently, it was impossible to know precisely how much code was covered by tests. They had a general idea, but without the right tools, it was impossible to say with certainty.

They decided that adding a code coverage tool would help them measure things more precisely. They did a serious evaluation of different tools, and ended up deciding on Coco for their daily workflow. Unlike Bullseye Coverage Analyzer, another tool they tried, which required them to make many changes to their code in order to instrument fully, Coco was able to instrument their entire code base.

What problem does it solve?

Immediately, Coco showed them which parts of the software were not yet covered by tests, but another problem to solve was that when a developer changes something in the code base, and wants to run the tests, they have to wait until the next morning before those changes can be pushed into version control. It takes too long and interrupts the work-flow of developers to run all of the tests before each possible change. InnovMetric ended up using Coco to get information on which tests are using a specific part of code, so that developers can run only the relevant tests, and get results faster. We like tools that save developer time.

Every two days, an instrumented version of the tests are run (close to 24 hours running time, hence the launch every 2 days). This gives them a full report with condition coverage, that can be used by developers to know if code they’re modifying is covered by tests, and which ones.

“Coco brings together the missing link between code and tests”

Coco seemed like a regular code coverage tool at first, but after we started using it, we found some very advanced features.

InnovMetric was able to write macros for Visual Studio to get the right HTML coverage report according to the current file in the editor. The next step is to get a full list of test executions that covers the modified code (at the function level) and possibly run those tests. Coco also has the ability to help optimize execution order of tests so that the maximum coverage is performed by the earliest tests executed.

“A software without good support is not a good software.”

For InnovMetric, support is very important, so this was one thing that made choosing Coco feel like the right choice. While getting up to speed, froglogic support staff helped InnovMetric in working out some of the initial kinks, and in the process, found and quickly fixed some bugs in Coco, making it possible to instrument their entire code base without modifications.

Any raw numbers to share?

The raw numbers are less interesting than the differences and trends. Using Coco, we can observe that an amount of work we do brings the coverage value up by a certain amount, and that is what is most important to us.

“It helps us see how good our tests are.”