We invoke the
squishrunner command to execute tests and generate JUnit and web reports. Additionally,
squishrunner is called with a
--exitCodeOnFail switch so it returns a custom exit code (nonzero) if any of the test cases have failed, and zero otherwise. Therefore, GitLab is able to set the job status accordingly.
The first step is to configure the GitLab Runner. We will configure the following environment variables:
- SQUISH_DIR – Squish installation directory
- SQUISH_LICENSEKEY_DIR – Squish license location
- SQUISH_SERVER_PORT – Port at which the squishserver process is started
- DISPLAY – Display used to display an application GUI. We use the VNC Server to provide a headless display.
The next step is to define a pipeline in the file
.gitlab-ci.yml. Pipelines are defined by specifying jobs that run in stages. In our example, we define a job named “squish-tests1” which is run in a test stage.
squish-tests1: stage: test script: - echo DISPLAY=$DISPLAY - echo "Starting VNC Server" - vncserver :$DISPLAY_NO - echo $SQUISH_DIR - echo "Starting squishserver on port=$SQUISH_SERVER_PORT..." - $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT 1>server.log 2>&1 & - sleep 5 - echo "Register AUT..." - $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT --config addAUT addressbook $SQUISH_DIR/examples/qt/addressbook - echo "Starting tests..." - $SQUISH_DIR/bin/squishrunner --port $SQUISH_SERVER_PORT --testsuite /home/tomasz/suites/suite_PageObjects --exitCodeOnFail 13 --reportgen junit,junit_report.xml --reportgen html,web_report after_script: - echo "Stopping squishserver..." - $SQUISH_DIR/bin/squishserver --stop --port $SQUISH_SERVER_PORT & - echo "Stopping VNC Server..." - vncserver -kill :$DISPLAY_NO - sleep 5 artifacts: when: always reports: junit: junit_report.xml paths: - server.log - junit_report.xml - web_report/
The job performs the following actions:
squishserverand redirect its stdout and stderr output to a
- Register the Application Under Test (AUT)
squishrunnerto run the test suite and generate a JUnit report and HTML report and set
- Collect artifacts. We use the setting
when: always, so artifacts are collected regardless of job status (by default GitLab only collects artifacts on successful job executions, which is not good in our case, as we need an HTML report to analyze the cause of failures).
When committing a change to the AUT, jobs defined in the pipeline script are started. After the application is built (not covered in our pipeline example), the test stage is executed. As part of this stage, GUI tests are executed using the Squish GUI Tester. The below screenshot shows the console output from test execution.
After test execution, artifacts, including the HTML report, are uploaded. To analyze detailed results in the HTML report, you need to select Download in the Job Artifacts view and open
web_report/index.html in a web browser.