froglogic / All / Measuring QML Coverage

Measuring QML Coverage

Last year we started receiving the first requests for QML coverage. “Sure. We’ll look into it.”, we replied. It seemed like a logical extension of our cross-language coverage tool Squish Coco. At least on first sight.

At this year’s Qt Contributors’ Summit the question came up independently in one of the sessions. I had nothing to show back then. But now, there’s finally a prototype accomplishing a proof of concept. To be seen live in action at froglogic’s Qt Developer Days 2014 booth.

coco_coveragebrowser

With QML finding its way into “real” applications it’s a natural move for quality-aware developers to track its test coverage. A must for safety-critical software. Would it be much different from the current set of supported languages like C, C++, C# or the scripting language Tcl? When looking closer one quickly realizes two fundamental differences:

  • Besides some embedded JavaScript code QML is most about the UI. In fact, there can be zero JavaScript. So no if() statements or for() loops leading to execution flow branching. Just buttons, dials, menus, etc.
  • QML is a declarative language. Leaving the optional JavaScript aside nothing gets executed in the classic sense. By definition there is no control flow.

After some brainstorming and mind-bending sessions we ended up going for a cross-over solution: classic code coverage gets intertwined with what I’ll call “GUI coverage”. The latter measures test coverage by the degree to which control elements are being excercised. For a button this would be a click for example. Same for a menu item. Whether alternate keyboard usage is required as well can be configured.

To demonstrate the output of such an approach we grabbed the sample Unit Converter application Reggie is going to use during Tuesday’s session on Behavior-Driven Development (BDD):

unitconverter

After the GUI gets excercised through a dilligent human tester or a automated testing tool like Squish coverage data will be emitted. Analysis can be done using an interactive tool (see CoverageBrowser screenshot at the top) or an HTML report. The project leader might just care about the overall number. But a developer may want to study details. Was each ‘case’ of a ‘switch’ statement hit? Did each choice of a radio button group get used at least once? Here’s an overview showing percentages for control elements as well as JavaScript code:

Coverage Percent

Analog to coloring pieces of code the usage of graphical elements is visualized: the “Convert” button was clicked (green). The unit selectors remained unused (red). As a result the whole dialog was only partially tested (orange).

guicolored

Application components implemented in C++ and using QWidgets would be included in above metrics, too. For a live demo visit our booth in Berlin between October 6-8. Others are invited to provide feedback and ask questions via the comment field.

2 Comments

Hi,

Wondering if use of QML in HMI for safety critical embedded systems is suitable for example for IEC 61508 SIL2.
If yes, tools are needed to guarantee code coverage, will squish coco provide coverage for QML.

Is there a certification kit for Qt?

Thanks for your reply.

BR

Hi!

Squish Coco will soon support coverage of QML – next to C++ and C#. If you are interested in a live demo please contact squish@froglogic.com .

With regards to a certification kit for Qt I suggest to contact its maker: The Qt Company. They have serviced several customers with safety-critical embedded systems before. I’m unsure whether Qt and QML itself will pass the SIL2 requirements. Qt and QML are often used as a *frontend* to the safety-critical core of a system.

Leave a Reply

Your email address will not be published. Required fields are marked *

Copy link
Powered by Social Snap