Verify objects that appear only for short durations

Many applications that carry out a long-running task in the background display a dialog indicating the progress of the task. The dialog may block access to the application too if other actions do not make sense while the task is running. These dialogs usually appear and disappear by themselves, based on the long-runnning tasks progress.

Since these dialogs are part of the application logic it is important that they are tested as well – in particular in error-scenarios. However since they automatically disappear it can be challenging to verify it inside a test as a recording or verification creation is not easily doable.

Tests may also want to use such a dialog as a synchronization point before carrying out the actions that are to be done once the task has finished. So detecting the existence and vanishing of the dialog can help determine when such tasks are fully done.

In the following sections we’ll look into how to detect the appearance and disappearance of the dialog as well as checking the progress. Finally I’ll give a few ideas on how the code can be improved/extended.

Test Setup

The following explanations are being done based on an example application provided as part of the Qt framework and can be automated using the Squish for Qt edition. The application we’re using is called findfiles and can be found in the Qt examples folder under qtbase/widgets/dialogs. The application has a single dialog for setting the search parameters like this:

We’re going to start off with a very simple recorded test that sets the findfiles application to look for all files containing the word QLabel anywhere in the Qt examples folder. Finally the number of results is verified. The paths being used here are of course bound to work only on my system and the number of found files may differ as well.

The script I ended up recording can be seen here:

# -*- coding: utf-8 -*-
def main():
    startApplication("findfiles")
    mouseClick(waitForObject(":Containing text:_QLineEdit"), 406, 6, 0, Qt.LeftButton)
    type(waitForObject(":Find Files.Containing text:_QComboBox"), "QLabel")
    mouseClick(waitForObject(":In directory:_QLineEdit"), 452, 13, 0, Qt.LeftButton)
    type(waitForObject(":Find Files.In directory:_QComboBox"), "<Ctrl+A>")
    type(waitForObject(":Find Files.In directory:_QComboBox"), "/Users/andreas/Qt/5.6/Src/qtbase/examples")
    type(waitForObject(":Find Files.In directory:_QComboBox"), "<Delete>")
    clickButton(waitForObject(":Find Files.Find_QPushButton"))
    test.compare(waitForObjectExists(":Find Files.Browse..._QTableWidget").rowCount, 241)

When executing this script it will fail on the last line – the rowCount value will be 0. The reason for this is that the comparison is being done as soon as the table is found, there’s nothing recorded that will wait for a dialog to appear and disappear again. The waitForObjectExists function does not wait for an object to become ready – like waitForObject would do – and hence is not blocked by the modal progress dialog.

Verify the dialog appears and disappears

In order to fix the synchronization problem and ensure that our test fails whenever the dialog fails to appear we’ll add some script code to wait for the dialog to appear and disappear.

Verifying the existence of the dialog can be done through the waitForObject function with a suitable object name for the dialog. The name can be obtained by recording a simple mouseClick on the dialog background area. When waitForObject fails to find the dialog, or its being blocked by some other object, it will throw an exception that will end the test script. This is not necessarily wanted, for example when the timeout was just too short another run is required just to find out that verifications in the dialog also fail. The error message is also harder to grasp when analyzing the test results, a clear message like ‘progress dialog was expected, but did not show up within x seconds’ is a lot easier to read.

The waitForObject invocation will be wrapped into a small function that returns True or False depending on wether the object name can be found. This in turn is then used to add a verification of the dialog existence into the script after the clickButton invocation. The timeout for waitForObject is intentionally short since the dialog should appear very fast and if it takes too long that is considered a bug in the application. The following two snippets show the function wrapping waitForObject and the verification added after clickButton:

def doesObjectExist(name, timeout):
    try:
        waitForObjectExists(name, timeout)
        return True
    except LookupError:
        return False
    clickButton(waitForObject(":Find Files.Find_QPushButton"))
    appearingTimeout = 2000
    if doesObjectExist(":Find Files_QProgressDialog", appearingTimeout):
        test.passes("Progress dialog appeared within %s seconds" % (appearingTimeout/1000,))
    else:
        test.fail("Progress dialog failed to appear within %s seconds" % (appearingTimeout/1000,))

The synchronization and verification of the progress dialog closing again is using the same function. However this time we want to wait for the dialog name to be not findable anymore. Hence we use the waitFor function to wait until doesObjectExist returns False. This is outlined in these lines that are being added to the script after the verification of the dialogs presence:

    disappearingTimeout = 10000
    dialogHasClosed = waitFor(lambda: not doesObjectExist(":Find Files_QProgressDialog", 500), disappearingTimeout)
    if dialogHasClosed:
        test.passes("Progress dialog has been closed within %s seconds" % (disappearingTimeout/1000,))
    else:
        test.fails("Progress dialog still shown after %s seconds" % (disappearingTimeout/1000,))

This small test now runs completely through and passes as long as a progress dialog is shown and the final set of files found matches the expectation. What is not being verified though is wether the progress dialog actually shows some progress, i.e. wether the progress bar advances or not. Since failing to show progress would be making the user interface quite bad its worthwhile to see how that could be done.

Verify that actual progress is being shown

Verification of the progress bar value can be done quite easily with most toolkits as the current value is accessible as a property. We’ll check the property’s value has reached a given value within a certain amount of time using the waitFor function again. Depending on the result of that check a result entry is being logged.

In order to test for different points of progress over time the example uses a small function to do the actual verification and calls it twice for the point of 100 files and for having found 1500 files. The following two snippets show the example script code for the function and how its being used while the dialog is being shown:

def verifyProgressAdvance(progressBar, wantedValue, timeout): 
    valueReached = waitFor(lambda: progressBar.value > wantedValue, timeout)
    if valueReached:
        test.passes("Progressbar reached value %s after at most %s seconds" % (wantedValue, timeout / 1000))
    else:
        test.fail("Progressbar did not reach value %s after %s seconds" % (wantedValue, timeout / 1000))
    progressBar = waitForObjectExists("{type='QProgressBar' container=':Find Files_QProgressDialog'}")
    verifyProgressAdvance(progressBar, 100, 2000)
    verifyProgressAdvance(progressBar, 1500, 5000)

This time the example script just creates an object name for the progress bar object on the fly instead of adding it to the object map. I could’ve recorded a click on that as well to get a symbolic name, but that would likely have not been any more expressive than the realname.

This final version enforces synchronization on the file-finding task in the test script so that the result of the file lookup can be easily checked. In addition it ensures that the application provides a sensible user interface while the task is being carried out and informs the user about the progress.

The general idea of waiting for appearance/disappearance of objects with waitForObject and waitFor as well as the verification of values being reached after some time can be applied to other scenarios where ‘volatile’ or ‘transient’ objects are being shown by the application or some objects properties change over a time in a deterministic manner.

While the example scripts are using a Qt application as AUT the basic idea can be applied to other toolkits and Squish editions.

Further Development

During the preparations of the example a couple of improvements came to my mind, that are beyond the scope of this article. So I’m just going to outline how the example can be extended or improved:

  • More details in the progress dialog can be verified, for example that the filenames are being displayed.
  • The verifyProgressAdvance function does not work very well if the dialog vanishes during the waitFor invocation
  • the verification of the table cell contents should likely also include the first column for completeness.
  • Use Squish’s support for reacting to signals to make the progress-validation stricter.

0 Comments

Leave a reply

Your email address will not be published. Required fields are marked *

*