Processing squishrunners XML Output

Processing squishrunners XML Output

One feature of our squishrunner program (which starts and controls the test execution) is that the output of the test can be given in different formats. This can be accomplished by using the ‘–reportgen’ argument and specifying an output format and an output file (i.e. ‘–reportgen xml,/tmp/results.xml’). In particular, the current version lets you generate an Excel table or a XML file with the test results instead of plain text output. The former is a simple way to make your PHB happy, the latter can be used as a hook into a whole new range of test result visualizations.

For instance, we’re using Squish internally to test Squish itself. A whole range of test suites is run every night, and the output is stored as XML files. Those XML files then get converted into HTML markup which then end up on some internal web page where every froglogic employee can see what tests failed last night. This conversion is done by a Python script, but here I’ll use an XSLT script to turn the information in squishrunner’s XML output into some web page. By the way, if you aren’t too familiar with XSLT, I can recommend the XSLT Tutorial at zvon.org for a good start.

The structure of the XML markup generated by squishrunner is very flat and roughly looks like this:

  • A ‘SquishReport’ root element which has a ‘Version’ attribute. This version number can be used to tell what structure the rest of the document has
  • A ‘summary’ child element of ‘SquishReport’ which has attributed like ‘testcases’ or ‘fatals’ or ‘passes’. You can use this to get overall result numbers (instead of iterating over all the results in the file and adding them up yourself).
  • A number of ‘testresult’ elements, each of which has a ‘result’ attribute, a ‘time’, a ‘message’ and others. These carry the actual test results.

Unfortunately, the current format (version 1.2) is not terribly object oriented. Hence, if you had e.g. a test suite with two test cases, you might end up with some markup like this (mucho simplified):


<SquishReport version="1.2">
<summary testcases="2" .. />
<testresult result="START" ..>Start of test suite 'suite_foo'</testresult>
<testresult result="START_TEST_CASE" ..>Test case 'tst_firstTest' started</testresult>
<testresult result="PASS" ../>
<testresult result="PASS" ../>
<testresult result="END_TEST_CASE" ..>Test case 'tst_firstTest' finished</testresult>
<testresult result="START_TEST_CASE" ..>Test case 'tst_secondTest' started</testresult>
<testresult result="PASS" ../>
<testresult result="FAIL" ../>
<testresult result="END_TEST_CASE" ..>Test case 'tst_secondTest' finished</testresult>
<testresult result="END" ..>End of test suite 'suite_foo'</testresuilt>
</SquishReport>

Note how there are ‘pseudo’ result types for test results, like ‘START’, ‘START_TEST_CASE’ or ‘END_TEST_CASE’ in addition to the real result types like ‘PASS’ and ‘FAIL’. These pseudo results are used to mark the start/end of a test suite or a test case. Other than this slightly awkward (in my humble opinion) structure, the XML format should be pretty self-explanatory, I suggest to just load an XML result file and look at it.

As a start, I hacked up a sample XSLT script which transforms a Squish XML report file into a nice HTML page. However, the HTML page will look rather dull by default since it expects a ‘squish-report.css’ file to exist which defines all the visual properties. Feel free to look at the generated HTML file to see what gets generated.

A few other ideas I have in my mind when it comes to doing interesting stuff with test results:

  • Maintaining a table which maps test case names to email addresses of their maintainers, and then sending the results per test case automatically to each maintainer.
  • Generating a list of SQL commands out of an XML report file, and then piping those to some local SQL server so that one can have a nice history of test results.
  • Generating DocBook out of the XML files instead of generating HTML directly. Then, one could use something like e.g. dblatex (which I can personally recommend) to generate nice PDF files out of the DocBook files.

If you have any remarks or ideas for other ways, feel free to post a comment. 🙂