Running GUI Tests on Each Commit in GitLab CI/CD

In a recent article, we wrote about running Squish tests on merge requests in GitLab. In this article, we present a solution to run Squish tests on each commit.

Overview

We invoke the squishrunner command to execute tests and generate JUnit and web reports. Additionally, squishrunner is called with a --exitCodeOnFail switch so it returns a custom exit code (nonzero) if any of the test cases have failed, and zero otherwise. Therefore, GitLab is able to set the job status accordingly.

Runner Settings

The first step is to configure the GitLab Runner. We will configure the following environment variables:

  • SQUISH_DIR - Squish installation directory
  • SQUISH_LICENSEKEY_DIR - Squish license location
  • SQUISH_SERVER_PORT - Port at which the squishserver process is started
  • DISPLAY - Display used to display an application GUI. We use the VNC Server to provide a headless display.

Job Configuration

The next step is to define a pipeline in the file .gitlab-ci.yml. Pipelines are defined by specifying jobs that run in stages. In our example, we define a job named "squish-tests1" which is run in a test stage.

squish-tests1:
stage: test
script:
- echo DISPLAY=$DISPLAY
- echo "Starting VNC Server"
- vncserver :$DISPLAY_NO
- echo $SQUISH_DIR
- echo "Starting squishserver on port=$SQUISH_SERVER_PORT..."
- $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT 1>server.log 2>&1 &
- sleep 5
- echo "Register AUT..."
- $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT --config addAUT addressbook $SQUISH_DIR/examples/qt/addressbook
- echo "Starting tests..."
- $SQUISH_DIR/bin/squishrunner --port $SQUISH_SERVER_PORT --testsuite /home/tomasz/suites/suite_PageObjects --exitCodeOnFail 13 --reportgen junit,junit_report.xml --reportgen html,web_report
after_script:
- echo "Stopping squishserver..."
- $SQUISH_DIR/bin/squishserver --stop --port $SQUISH_SERVER_PORT &
- echo "Stopping VNC Server..."
- vncserver -kill :$DISPLAY_NO
- sleep 5
artifacts:
when: always
reports:
junit: junit_report.xml
paths:
- server.log
- junit_report.xml
- web_report/

The job performs the following actions:

  1. Start squishserver and redirect its stdout and stderr output to a server.log file
  2. Register the Application Under Test (AUT)
  3. Call squishrunner to run the test suite and generate a JUnit report and HTML report and set --exitCodeOnFail 13 setting
  4. Stop squishserver
  5. Collect artifacts. We use the setting when: always, so artifacts are collected regardless of job status (by default GitLab only collects artifacts on successful job executions, which is not good in our case, as we need an HTML report to analyze the cause of failures).

Example

When committing a change to the AUT, jobs defined in the pipeline script are started. After the application is built (not covered in our pipeline example), the test stage is executed. As part of this stage, GUI tests are executed using the Squish GUI Tester. The below screenshot shows the console output from test execution.

After test execution, artifacts, including the HTML report, are uploaded. To analyze detailed results in the HTML report, you need to select Download in the Job Artifacts view and open web_report/index.html in a web browser.

Comments

    The Qt Company acquired froglogic GmbH in order to bring the functionality of their market-leading automated testing suite of tools to our comprehensive quality assurance offering.