To allow automating test runs, Squish provides command line tools which control the whole procedure. The tool to execute tests is squishrunner. To be able to start an AUT and communicate with it, squishserver has to be running.

A simple Unix shell script to execute the complete test suite /home/squish/suite_myapp and save its results to /home/squish/results<date>.xml would be:

#!/bin/sh

# start squishserver
squishserver &

# generate filename for logfile
LOGFILE=/home/squish/results`date +%d%m%Y`.xml

# execute the test
squishrunner --testsuite /home/squish/suite_myapp --reportgen xml,$LOGFILE

# exit squishserver again
squishserver --stop

The same as a Windows batch file wouldn't look very different:

REM start squishserver
start /B squishserver

REM generate filename for logfile
set LOGFILE c:\squishhome\results.xml

REM execute the test
squishrunner --testsuite c:\squishhome\suite_myapp --reportgen xml,%LOGFILE%

REM exit squishserver again
squishserver --stop

To avoid having to maintain a Unix shell script and a Windows batch file, another option would be to implement this script e.g. using Python which would allow a platform independent implementation.

Such a script automatically executes a test with all required initializations and cleanups. For details on how to process the resulting XML file, see Processing Test Results (Section 16.7.3).

The next step would be to have this test script run automatically e.g. once a day. On Unix this can be done by setting up a cron job which executes the script. On Windows it is possible to set up a Windows Service to run the script regularly.

For detailed documentation about cron jobs or Windows Services, search for the relevant information on the Internet or contact froglogic's commercial support to assist you.

So far everything in this manual assumed local testing. This meant that the squishserver, squishrunner, and the AUT are all running on the same machine. This chapter shows how to remotely run a test on a different machine. For example, let's assume that we work and test on computer A, and want to test the AUT located on computer B.

We begin by installing Squish and the AUT (except for Web testing, where this is not necessary), on the target computer (computer B). Now, if we are using an edition other than Squish for Web, on computer B, we must tell the squishserver where the AUT is located. This is achieved by running the following command:

squishserver --config addAppPath <path_to_addressbook>

Later we will connect from computer A to squishserver on computer B. By default the squishserver only accepts connections from the local machine. Accepting connections from elsewhere might compromise security, so if we want to connect to the squishserver from another machine we must register the machine which will try to establish a connection for executing the tests (computer A in our example) with the machine running the AUT and squishserver (computer B). This way we can ensure that only trusted machines can communicate with each other using the squishserver.

To do this, on the AUT's machine (computer B) we create a plain text file called /etc/squishserverrc (on Unix or Mac) or c:\\squishserverrc (on Windows). If you don't have write permissions to /etc or c:\, you can also put this file into SQUISH_ROOT/etc/squishserverrc on either platform. The file should have the following contents:

ALLOWED_HOSTS = <ip_addr_of_computer_A>

<ip_addr_of_computer_A> must be the IP address or host name of computer A. For example on our network the line is:

ALLOWED_HOSTS = 192.168.0.3 

but this will most certainly differ on your network.

ALLOWED_HOSTS also accepts IP addresses with wildcards. To e.g. allow all machines from the 192.168.0 range to connect to this squishserver, you can specify 192.168.0.*. To specify multiple IP addresses which should be allowed to connect to the squishserver, you can specify all IP addresses in the ALLOWED_HOSTS separated by spaces.

Once we have registered computer A, we can run the squishserver on computer B, ready to listen to connections e.g. from computer A.

We are now ready to create test cases on computer A and have them executed on computer B.

By default for convenience, the Squish IDE starts squishserver locally on startup and connects to this local squishserver to execute the tests. But it is also possible to connect to the squishserver on a remote machine, such as computer B, from withing Squish IDE. We can control this behavior through the preferences dialog. Click Edit|Preferences to invoke the preferences dialog. Uncheck the Start squishserver locally on startup checkbox, and enter the IP address of the AUT's machine (computer B) in the squishserver host line edit. The port spinbox should not need to be changed.

Now we can execute the test suite as usual. We will see that the AUT will not be started locally, but on computer B instead. After the test has finished, the results will be visible in the Squish IDE on computer A as usual.

We can also do remote testing using the command line. The command is the same as described in the previous sections, only this time we must also specify a host name using the --host option:

squishrunner --host computerB.froglogic.com --testsuite suite_addressbook

This way it is possible to create, edit and run tests on a remote machine via Squish IDE and also to control them from the command line or a shell script using squishrunner. In conjunction with Automated Test Runs (Section 16.7.1) it is possible to automate testing of an applications located on different machines and platforms using shell scripts or batch files.

In the previous section we saw how to execute an AUT and run its tests on a target machine under the direction of a master machine and also how to automatically execute test runs using scripts and batch files. In this section we will look at processing the test results from automatic test runs.

By default, squishrunner prints the test results to stdout. Although it isn't difficult to parse this output, squishrunner also includes a report generator which can output the results in XML or Excel file format. There are modules for nearly every scripting language available to parse XML, so it is quite easy to post-process the test results and convert them to the format you require.

To make squishrunner use the XML report generator, specify --reportgen xml on the command line. If you want to get the XML output written into a file instead of stdout, specify --reportgen xml,<filename>, e.g.:

squishrunner --host computerB.froglogic.com --testsuite suite_addressbook_py --reportgen xml,/tmp/results.xml

For Excel output, use the xls option for --reportgen instead. So the command then looks like:

squishrunner --host computerB.froglogic.com --testsuite suite_addressbook_py --reportgen xls,/tmp/results.xls

The XML format is very simple. The document starts with the tag SquishReport. All other tags are children of this one. The following child tags are possible:

The detailed message of the result is saved in the text node of the element.

Here is an example report of a test suite run. This test suite had just one test case, in which four comparisons passed and one failed:

<SquishReport version="1.2" >
    <summary fatals="0" testcases="1" expected_fails="0" unexpected_passes="0" warnings="0" tests="5" errors="0" fails="1" passes="4" />
    <testresult message="Start 'suite_addressbook_js'" result="START" time="2006-06-06T10:33:31" >Test 'suite_addressbook_js' started</testresult>
    <testresult fatals="0" expected_fails="0" unexpected_passes="0" warnings="0" errors="0" message="Start 'tst_add_address'       " result="START_TEST_CASE" fails="1" time="2006-06-06T10:33:32" passes="4" >Test 'tst_add_address' started</testresult>
    <testresult line="test.js:23" message="VP1: Object property comparison of ':Addressbook.ABCentralWidget1.QListView1.childCount' passed" result="PASS" time="2006-06-06T10:33:44" >'1' and '1' are equal (VP1)</testresult>
    <testresult line="test.js:23" message="VP1: Object property comparison of ':Addressbook.ABCentralWidget1.QListView1.item_1.text0' failed" result="FAIL" time="2006-06-06T10:33:44" >'Max' and 'Maxi' are not equal (VP1)</testresult>
    <testresult line="test.js:23" message="VP1: Object property comparison of ':Addressbook.ABCentralWidget1.QListView1.item_1.text3' passed" result="PASS" time="2006-06-06T10:33:44" >'max@mustermann.net' and 'max@mustermann.net' are equal (VP1)</testresult>
    <testresult line="test.js:23" message="VP1: Object property comparison of ':Addressbook.ABCentralWidget1.QListView1.item_1.text2' passed" result="PASS" time="2006-06-06T10:33:44" >'Bakerstreet 55' and 'Bakerstreet 55' are equal (VP1)</testresult>
    <testresult line="test.js:23" message="VP1: Object property comparison of ':Addressbook.ABCentralWidget1.QListView1.item_1.text1' passed" result="PASS" time="2006-06-06T10:33:44" >'Mustermann' and 'Mustermann' are equal (VP1)</testresult>
    <testresult message="End 'tst_add_address'         " result="END_TEST_CASE" time="2006-06-06T10:33:46" >End of test 'tst_add_address'</testresult>
    <testresult message="End 'suite_addressbook_js'" result="END" time="2006-06-06T10:33:46" >End of test 'suite_addressbook_js'</testresult>
</SquishReport>

In examples/regressiontesting you can find some example scripts which execute the addressbook test suite on different machines and present the daily output on a Web page by post processing the XML and generating HTML. The tutorial Tutorial: Setting Up Automated Batch Testing (Chapter 12) shows step-by-step using this example how to automate test runs and process the test results.