Migration of image based test suites

Migration of image based test suites

Squish 6.3 has introduced the concept of image-based testing. It allows to find and interact with the AUT controls without any knowledge about its internal structure. Unlike the object based testing – which is mostly independent of the visual appearance of the GUIs – the image based tests will fail if the rendering of the relevant AUT controls changes. Squish 6.4 extends the image search functionality with the ability to find resized occurrences of the template images and occurrences with small differences to the template image. This allows to use the same template image in case of a component that changes its size.

The differences between the toolkit version and styling, current DPI settings or platform-specific rendering differences can cause the entire GUI to change size. Test suites that uses a large number of image search operations can be difficult to migrate to a new testing environment. Adjusting options of each separate image lookup function call can be tedious and error prone. If the size of the AUT controls changes in a consistent manner, it may be helpful to adjust the default image search parameters. It can be done on the Image Search tab in the Test Suite Settings editor.

Image search default parameters editor

Image search default parameters editor

Guessing proper values for these parameters may be difficult. On one hand the parameter values should be permissive enough to allow finding all the required controls. On the other hand it should be as strict as possible in order to limit both execution time and the risk of false matches. Squish IDE can compute optimal parameters for the current test execution. Provided the TestSettings.objectNotFoundDebugging option is set for the current test suite, any image search failure in a test run with the Squish IDE will trigger the Image Not Found dialog. After clicking the Adjust Search Parameters… button, the Image Search Preview dialog should display the optimal image search parameters. While you adjust the parameters, the preview pane displays updated results of the image search. Subsequently, you can apply the selected values as the new suite-specific defaults and resume the test execution.

Image Search Preview dialog

Image Search Preview dialog

The Image Not Found dialog offers also a fully automatic image search parameter adjustment mode. In order to activate it, select the Attempt to fix image search errors automatically checkbox. It will compute the optimal parameters for each of the failing image search instruction and update the defaults. Please note that because the automatic adjustment mode runs without any user interaction, certain limits on the possible parameter values are in place. These limits are a safeguard against relaxing the image search too much and allowing false-positive matches.

In order to preserve any parameter changes required on other platforms, Squish IDE can only relax the image search parameters; it never makes it any stricter. By repeating the adjustment procedure for each test platform that causes errors you can turn the image search platform-independent.

Post-mortem debugging of image search errors

If you execute a test suite without the Squish IDE and encounter an image search error, you cannot adjust the parameter values immediately. In order to enable post-mortem image search debugging, enable the ‘Log a screenshot on an error‘ option in the Test Suite settings editor. The result of the failed test should contain a screenshot attachment. After importing the test result into the IDE, the context menu of the error report contains the “Image Search Preview” entry. It opens the Image Search Preview dialog that computes the optimal values for the image search parameters and results .

Image Search Preview for test error report

Image groups

Some GUI elements can change their looks drastically between different platforms; eg. because of using native graphics or rendering style. In such case you cannot use a single template image to identify such a control. The Image Not Found dialog offers an option to add another template image to the currently used one. It converts the current image into an image group (if necessary) and adds newly selected image to it. An image group can contain any number of graphics that are considered equivalent visuals of the same GUI element. Images stored in a group do not have to be limited to platform differences. Other variations like selection and mouse-hover states may also require additional imagery.

Future extensions

The default image search parameters are set for the entire test suite. This can become problematic in case a few of the search images require different values than the rest. Every image search function call can specify custom values that override the defaults, but manual management of parameters for each separate image search  instruction is difficult and should be avoided. Future releases of Squish will support automatic adjustment of the values specific to each individual image search. This will allow automated management of the image search parameters in detailed and precise manner.


Leave a reply

Your email address will not be published. Required fields are marked *