Measuring QML Coverage

Measuring QML Coverage

Last year we started receiving the first requests for QML coverage. “Sure. We’ll look into it.”, we replied. It seemed like a logical extension of our cross-language coverage tool Squish Coco. At least on first sight.

At this year’s Qt Contributors’ Summit the question came up independently in one of the sessions. I had nothing to show back then. But now, there’s finally a prototype accomplishing a proof of concept. To be seen live in action at froglogic’s Qt Developer Days 2014 booth.

coco_coveragebrowser

With QML finding its way into “real” applications it’s a natural move for quality-aware developers to track its test coverage. A must for safety-critical software. Would it be much different from the current set of supported languages like C, C++, C# or the scripting language Tcl? When looking closer one quickly realizes two fundamental differences:

  • Besides some embedded JavaScript code QML is most about the UI. In fact, there can be zero JavaScript. So no if() statements or for() loops leading to execution flow branching. Just buttons, dials, menus, etc.
  • QML is a declarative language. Leaving the optional JavaScript aside nothing gets executed in the classic sense. By definition there is no control flow.

After some brainstorming and mind-bending sessions we ended up going for a cross-over solution: classic code coverage gets intertwined with what I’ll call “GUI coverage”. The latter measures test coverage by the degree to which control elements are being excercised. For a button this would be a click for example. Same for a menu item. Whether alternate keyboard usage is required as well can be configured.

To demonstrate the output of such an approach we grabbed the sample Unit Converter application Reggie is going to use during Tuesday’s session on Behavior-Driven Development (BDD):

unitconverter

After the GUI gets excercised through a dilligent human tester or a automated testing tool like Squish coverage data will be emitted. Analysis can be done using an interactive tool (see CoverageBrowser screenshot at the top) or an HTML report. The project leader might just care about the overall number. But a developer may want to study details. Was each ‘case’ of a ‘switch’ statement hit? Did each choice of a radio button group get used at least once? Here’s an overview showing percentages for control elements as well as JavaScript code:

Coverage Percent

Analog to coloring pieces of code the usage of graphical elements is visualized: the “Convert” button was clicked (green). The unit selectors remained unused (red). As a result the whole dialog was only partially tested (orange).

guicolored

Application components implemented in C++ and using QWidgets would be included in above metrics, too. For a live demo visit our booth in Berlin between October 6-8. Others are invited to provide feedback and ask questions via the comment field.