More Useful Squish Test Reports

More Useful Squish Test Reports

Test reports matter. Especially so when something goes wrong because Squish detected a failing verification during test execution.

When a test fails, it’s typically interesting to figure out why a test failed (except if you’re fine with just ignoring the result). The first step to figuring out why something went wrong is to figure out what went wrong, and that’s where a good test report shines.

A common test report

Too many times, you’re facing reports like this:

Standard Test Report

Standard Test Report

The first ‘PASS’ is mostly a nuisance: False and False being equal is not exactly news. It actually becomes a bigger problem when a comparison fails: ‘True’ and ‘False’ are not equal.

Here, something is clearly not as expected. The report leaves us puzzled though as to what is not as expected. It just states (correctly so, but not being helpful) that True and False are not equal values. And while it’s true that this inequality triggered the ‘FAIL’ result, what we’re actually interested in is the origin of those values. So we need a bit more context. Let’s look at the script code which might have generated the above results:

def main():
    startApplication("addressbook")
    addButton = waitForObjectExists(":Address Book.Add_QToolButton")
    test.compare(addButton.enabled, False)
    clickButton(waitForObject(":Address Book.New_QToolButton"))
    test.compare(addButton.enabled, False)

 

What’s happening here is that an ‘addressbook’ application is started. After fetching a reference to an ‘Add’ button, the test verifies that the add button is initially not enabled (i.e. the ‘enabled’ property is False). The test then proceeds to click on the ‘New’ tool button to conclude with a final check for the enabled state of the ‘Add’ button, expecting the button to be disabled as before. In practice however, clicking the ‘New’ button also changes the enabled state of the ‘Add’ button – hence, the first verification passes, but the second one fails.

The test report is not saying any of this though. The reason for that is that in the test.compare invocations, the Python interpreter first evaluates the expression ‘addButton.enabled’ (to either False or True, before resp. after the click on the ‘New’ button) and then invokes the test.compare function. So test.compare just sees a ‘True’ or ‘False’ value.

An Improved Test Report

An alternative approach to implement this functionality is to define a function which does not get passed the property value but rather the property name. The name can then be used by the function to access the property value, but it can also be used when printing test report messages:

def verify(object, propName, expected):
    actual = getattr(object, propName)
    
    if actual == expected:
        test.passes("Property {} has expected value".format(propName),
                    "Property value is '{}' as expected".format(expected))
    else:
        test.fail("Property {} does not have expected value!".format(propName),
                  "Expected '{}' but got '{}'".format(expected, actual))


def main():
    startApplication("addressbook")
    addButton = waitForObjectExists(":Address Book.Add_QToolButton")
    verify(addButton, 'enabled', False)
    clickButton(waitForObject(":Address Book.New_QToolButton"))
    verify(addButton, 'enabled', False)

Note how instead of calling test.compare, a custom ‘verify’ function is defined and used. It uses the built-in getattr function to get the value of a property given the property name and then compares the property value with the (given) expected value. If they are equal, the Squish function test.passes is used to log a ‘PASS’ item. Otherwise test.fail is used to log a ‘FAIL’. Running this test yields a much better test report:

Improved Test Report

Improved Test Report

The improvement here is that the  test output not only shows the expected and actual values, it also

  • Clearly tells the name of the property being evaluated.
  • Clearly marks which of the two shown fails (in case of a FAIL result) is the expected and which is the actual value.

Associating Results With The Right Location

There is one downside though: the test results are now generate within the ‘verify’ function, so double-clicking the PASS and FAIL entries in the test report will not actually jump to the ‘verify’ calls generating those results but instead jump into the ‘verify’ function.

This is a perfect use case for the Squish functions test.fixateResultContext and test.restoreResultContext. The documentation explains:

These functions temporarily enforce that test results created by other Squish functions such as test.xcompare or test.verify will have their report entry rewritten such that the location of the result entry is no longer the place where text.xcompare was called but rather any of the ancestors frames of the current function.

Using these functions, we can make our ‘verify’ function generate test reports in which the PASS and FAIL statements are associated with the file location at which ‘verify’ is called:

def verify(object, propName, expected):
    actual = getattr(object, propName)
    
    test.fixateResultContext()
    if actual == expected:
        test.passes("Property {} has expected value".format(propName),
                    "Property value is '{}' as expected".format(expected))
    else:
        test.fail("Property {} does not have expected value!".format(propName),
                  "Expected '{}' but got '{}'".format(expected, actual))
    test.restoreResultContext()

 

…and finally, we get the test reports we wanted!

Further Development

It’s easy to imagine useful extensions to the ‘verify’ function. Here are some ideas:

  • Allow passing an optional ‘detail’ message to ‘verify’ which is then printed to the test report.
  • Extend the function to allow testing property values as well as the return values of method calls (use case: verify the return value of a standard method exposed on an object)
  • Introduce the notion of ‘timeouts’ to the function such that it won’t fail immediately if the property does not assume the expected value but rather waits a bit (and retries).

 

Software engineer working at froglogic since 2005. When he's not currently at the swimming pool or busy building awesome Lego creations with his son, he can typically be found playing video games.

0 Comments

Leave a reply

Your email address will not be published. Required fields are marked *

*