Overview of the new file comparison functions in Squish 6.3

Overview of the new file comparison functions in Squish 6.3

With Squish 6.3 we added two new file comparison functions. The objective is to make it easier to verify not just the application state but also the output an application might generate. Squish now offers a general comparison function test.compareTextFiles. This function can be used to compare any type of file, but it is very strict. Another function that was added in Squish 6.3 is test.compareXMLFiles. Since XML is a very widely used format, Squish now offers a lot of different comparison options for these type of files. You can use both functions to compare a reference file to the file actually generated by your application. In this article I want to give some examples on how to use the new functions and highlight some of the comparison options.

General file comparisons

You can use test.compareTextFiles to verify the data stored by your application. Your application might for example save its data in a proprietary text based or binary format.

Example

To demonstrate the use of this function I picked QtCreator. QtCreator stores its general settings inside an INI configuration file. The following test script starts qtcreator and changes the language settings. It then verifies that the language change was correctly executed both on an application level, by looking at the text displayed, but also on the file level, to see if the settings have actually been written to the file correctly. To verify that the settings have been written correctly, we first need to get a reference file. To get a reference file you can just copy the file from the application and verify it manually once. In this example I created two reference files “germanReference.ini” when the configuration is set to german and “englishReference.ini”.

import os

def main():
    configFilePath = os.path.join(os.environ["APPDATA"],"QtProject\\QtCreator.ini")
    
    #Start Application
    startApplication("qtcreator")
    
    #Verify application state, did it startup in german?
    test.compare(waitForObject(":rectangle1.Neues Projekt_Button").text, "Neues Projekt")
    
    #Verify state of configuration file. Is the language set correctly in the configuration file?
    test.compareTextFiles("germanReference.ini", configFilePath, {'strictLineEndings':False});
    
    #Change language to english and shut down application for changes to take effect
    activateItem(waitForObjectItem(":Qt Creator.QtCreator.MenuBar_QMenuBar", "Extras"))
    activateItem(waitForObjectItem(":QtCreator.Menu.Tools_QMenu", "Einstellungen..."))
    clickTab(waitForObject(":Einstellungen.Umgebung_QTabWidget"), "Oberfläche")
    mouseClick(waitForObject(":interfaceBox.languageBox_QComboBox"), 54, 12, 0, Qt.LeftButton)
    mouseClick(waitForObjectItem(":interfaceBox.languageBox_QComboBox", "English"), 36, 9, 0, Qt.LeftButton)
    clickButton(waitForObject(":Einstellungen.Anwenden_QPushButton"))
    clickButton(waitForObject(":Neustart erforderlich.OK_QPushButton"))
    clickButton(waitForObject(":Einstellungen.OK_QPushButton"))
    activateItem(waitForObjectItem(":Qt Creator.QtCreator.MenuBar_QMenuBar", "Datei"))
    activateItem(waitForObjectItem(":QtCreator.Menu.File_QMenu", "Beenden"))
    snooze(5)
    
    #Restart Application
    startApplication("qtcreator")
    
    #Verify application state, did it startup in english?
    test.compare(waitForObject(":rectangle1.New Project_Button").text, "New Project")
    
    #Verify state of configuration file. Is the language set correctly in the configuration file?
    test.compareTextFiles("englishReference.ini", configFilePath, {'strictLineEndings':False});
    
    #Change language back to german and shut down application for changes to take effect
    activateItem(waitForObjectItem(":Qt Creator.QtCreator.MenuBar_QMenuBar", "Tools"))
    activateItem(waitForObjectItem(":QtCreator.Menu.Tools_QMenu", "Options..."))
    mouseClick(waitForObject(":interfaceBox.languageBox_QComboBox_2"), 63, 11, 0, Qt.LeftButton)
    mouseClick(waitForObjectItem(":interfaceBox.languageBox_QComboBox_2", "German (Germany)"), 54, 11, 0, Qt.LeftButton)
    clickButton(waitForObject(":Options.Apply_QPushButton"))
    clickButton(waitForObject(":Restart Required.OK_QPushButton"))
    clickButton(waitForObject(":Options.OK_QPushButton"))
    activateItem(waitForObjectItem(":Qt Creator.QtCreator.MenuBar_QMenuBar", "File"))
    activateItem(waitForObjectItem(":QtCreator.Menu.File_QMenu", "Exit"))

Results

When the verification succeeds both on the application and the file level the following result is displayed.

test.compareTextFiles result for qtcreator example

test.compareTextFiles result for qtcreator example

Comparison options

For text based comparisons there is a very limited set of options. The only option implemented so far is strictLineEndings which defaults to false. This option might be useful when you work with multiple operating systems that use different line endings. To make test.compareTextFiles pick up on the different line endings you’ll have to set this option to true. The example shows how to specify this option.

When the text file comparison fails the test results will also contain the differences in a a rough textual representation.

Limitations

While test.compareTextFiles can also be used for binary file formats, the difference stored in the test results will not be very helpful. Furthermore there is no way to specify features to ignore during the comparison so even the smallest changes will lead to failed comparisons.

XML file comparisons

You can use test.compareXMLFiles to compare XML files generated by your application. We chose to handle XML files separately because it is a widely used serialization format and it has very clear and distinguishable features. This gives us the option to ignore potential structural changes and only look at the actual values, or ignore certain values and structural information all together. Very common examples for features you may want to ignore in a comparison are the node order, time stamps, comments, certain attributes or maybe even entire node types. The test.compareXMLFiles function offers many options. You can find a list of the options in our documentation.

Example

Squish itself exports most of its test results as XML files and we constantly add new functions and need to make sure that old functions still produce the correct output. Therefore Squish test results are a good example to give you an overview how the test.compareXMLFiles function can be used and how to specify the different comparison options.

For this example I executed the same squish test twice to get the two test files. To make the test a little harder i also changed the XML version between test runs and did two manual changes to the files. In the following image you can see a side by side comparison of a section of the two files where the differences are highlighted in red.

Side by side comparison of two squish test results with highlighted differences

The first difference is the version attribute. We can use the “filteredAttributes” option to simply ignore the version attribute. The second difference is the time stamp which will be different for each test execution. We can use the same option to simply ignore the time attribute. The third difference is the data inside the name tag. The left side is encoded in CDATA tags while the right side isn’t. We can use the “ignoreCDATA” option to ignore this difference. The last difference is the node order between the uri and lineNo tags. To ignore the node order we can use the “ignoreNodeOrder” option.

The following source code shows how to set the options for test.compareXMLFiles since that might not be obvious. Both the “filteredAttributes” and the “filteredElements” options need to be specified as lists.

def main():
    filteredAttributes = ["version", "time"]
    options = { 'ignoreAttributeOrder': True,
                'ignoreNodeOrder': True,
                'filteredAttributes': filteredAttributes,
                'ignoreCDATA': True }
    test.compareXMLFiles("A.xml", "B.xml", options)

Results

To give you an idea how failed comparisons are denoted inside the test results, the following image shows the results when the “ignoreNodeOrder” option is not specified.

Failed test.compareXMLFiles test result

Limitations

Currently test.compareXMLFiles will stop the comparison at the first encountered difference. Therefore it will only give you the location and further information of the first encountered difference. While we developed this feature we talked about having a complete difference report, but this would require an entirely different algorithm. We might add this feature in the future though.

0 Comments

Leave a reply

Your email address will not be published. Required fields are marked *

*