In a recent article, we wrote about running Squish tests on merge requests in GitLab. In this article, we present a solution to run Squish tests on each commit.


We invoke the squishrunner command to execute tests and generate JUnit and web reports. Additionally, squishrunner is called with a --exitCodeOnFail switch so it returns a custom exit code (nonzero) if any of the test cases have failed, and zero otherwise. Therefore, GitLab is able to set the job status accordingly.

Runner Settings

The first step is to configure the GitLab Runner. We will configure the following environment variables:

  • SQUISH_DIR – Squish installation directory
  • SQUISH_LICENSEKEY_DIR – Squish license location
  • SQUISH_SERVER_PORT – Port at which the squishserver process is started
  • DISPLAY – Display used to display an application GUI. We use the VNC Server to provide a headless display.

Job Configuration

The next step is to define a pipeline in the file .gitlab-ci.yml. Pipelines are defined by specifying jobs that run in stages. In our example, we define a job named “squish-tests1” which is run in a test stage.

    stage: test
    - echo "Starting VNC Server"
    - vncserver :$DISPLAY_NO
    - echo $SQUISH_DIR
    - echo "Starting squishserver on port=$SQUISH_SERVER_PORT..."
    - $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT 1>server.log 2>&1 &
    - sleep 5
    - echo "Register AUT..."
    - $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT --config addAUT addressbook $SQUISH_DIR/examples/qt/addressbook
    - echo "Starting tests..."
    - $SQUISH_DIR/bin/squishrunner --port $SQUISH_SERVER_PORT --testsuite /home/tomasz/suites/suite_PageObjects --exitCodeOnFail 13 --reportgen junit,junit_report.xml --reportgen html,web_report
    - echo "Stopping squishserver..."
    - $SQUISH_DIR/bin/squishserver --stop --port $SQUISH_SERVER_PORT &
    - echo "Stopping VNC Server..."
    - vncserver -kill :$DISPLAY_NO
    - sleep 5
       when: always    
          junit: junit_report.xml
         - server.log
         - junit_report.xml
         - web_report/

The job performs the following actions:

  1. Start squishserver and redirect its stdout and stderr output to a server.log file
  2. Register the Application Under Test (AUT)
  3. Call squishrunner to run the test suite and generate a JUnit report and HTML report and set --exitCodeOnFail 13 setting
  4. Stop squishserver
  5. Collect artifacts. We use the setting when: always, so artifacts are collected regardless of job status (by default GitLab only collects artifacts on successful job executions, which is not good in our case, as we need an HTML report to analyze the cause of failures).


When committing a change to the AUT, jobs defined in the pipeline script are started. After the application is built (not covered in our pipeline example), the test stage is executed. As part of this stage, GUI tests are executed using the Squish GUI Tester. The below screenshot shows the console output from test execution.

After test execution, artifacts, including the HTML report, are uploaded. To analyze detailed results in the HTML report, you need to select Download in the Job Artifacts view and open web_report/index.html in a web browser.

1 Comment


I tried to use gitlab with squishrunner and I get an issue without using “13” in the “–exitCodeOnFail 13”.
My pipeline was stuck with “1”, What is the mandatory of “13” ? and why it’s not explained in your documentation ? (–exitCodeOnFail)

Thank a lot.

Leave a Reply

Your email address will not be published. Required fields are marked *

Copy link
Powered by Social Snap