With the help of virtual machines one can perform parallel executions in Squish as a time saving strategy.
As an application grows, it's features increase and so do the number of testcases in the corresponding testuite. A testsuite containing a large number of testcases becomes a problem. It takes a long time to finish. The reason for this being that the testcases are run consecutively and a long running testcase then blocks other testcases, which cannot run until the slow testcase is finished. A fast running testsuite is desirable. If a testsuite takes a long time to finish, it takes a long time before one can analyse the test results. Sometimes all testcases in a testsuite are not run to avoid production delays. This can lead to bugs not being fixed immediately. Mainly because it may take days before a critical bug is even discovered. Additionally, the person responsible for introducing the bug may not be available right away to fix it. A parallel execution can significantly reduce the time taken to run all the testcases.
If a testsuite is normally run in Squish, the testcases within the testsuite are executed in sequence. However, it is possible to perform parallel execution using Squish. One of the ways this can be achieved is by using multiple virtual machines. Each virtual machine has it's own Squish installation and subsequently a running squishserver. One can split up the testcases in the testsuite into even groups. Multiple squishrunner instances on the local machine can connect to the squishserver running on each of the virtual machine. Each of these squishrunner instances run the split up testsuite groups.
One can make use of the squishrunner --tags option to execute the split up testcase groups. The --tags option tells the squishrunner to execute only those test scripts which match the given tag filter. So if you tag all your testcases in one group with 'vm1' say, then you can make use of this to tell one of the squishrunner instances to run only those testcases which are tagged 'vm1'. Another squishrunner instance shall run only those testcases that are tagged 'vm2', and so on and so forth.
For this solution to work successfully the design of a testsuite is a significant consideration.
First and foremost it becomes very important to remove dependencies between the testcases. It is a prerequisite for being able to run testcases concurrently. If there is a certain order that needs to be followed when the testcases are run, one needs to eliminate the need for such an order. There are various ways these dependencies can be introduced. For example if the first testcase performs some setup steps to initialize the application. Or if the last testcase performs a cleanup operation before the execution is finished. Another example could be when previous testcase creates a datafile which the next testcase reads from. Frerich wrote an interesting article regarding this in our blog a couple of months ago. In this article one of the things Frerich talks about is unintentional dependencies between the testcases, and ways to identify them.
It also becomes important to ensure that all testcases take relatively the same time to run. A faster testcase scheduled to run behind a long running testcase, gets blocked by the slow testcase to finish execution causing delay for the entire run.
There are mutiple advantages in using a virtual machine farm. Firstly one does not have to acquire new hardware. It is cost effective to use several virtual machines sitting on one physical machine than to buy multiple machines. This also means less physical storage space and less hardware to maintain. Another advantage is that using vmware makes this design scalable. As the number of testcases grow one can easily add more virtual machines for more parallel executions.
After a parallel run you can collect all the individual reports from and consolidate them into one.
If you are producing your results in JUnit, you can feed the individual reports to your CI server and it should nicely generate a combined result for you. This knowledgebase article shows how you can merge multiple Squish reports into one using a Python script: Merging multiple Squish reports
To be able to see a single HTML based report combining multiple runs please stay tuned for our new product Team Server to be released.
