Multi-touch gesture support in Squish 5.1

Multi-touch gesture support in Squish 5.1

Until Squish version 5.0, Squish supported recording and replaying of keyboard, mouse and single-touch events only. With the rise of smartphones, tablets and more sophisticated embedded user interfaces, the demand to also allow recording and emulation of multi-touch gestures as part of automated GUI tests steadily increased.

With the upcoming Squish 5.1 release we enhanced Squish to support multi-touch gestures.

More specifically, until recently, only two specific gestures were recorded by Squish, such as touchAndDrag and longPressAndDrag in Squish for Android and similar commands in other Squish editions. While this suffices for most interactions, it lacked the curved path support as well as multi-finger and touch-size pressure variations. Support for these more complex gestures will become available for Squish for Android, Qt and iOS in the upcoming Squish 5.1 release.

Here’s how:

Gesture Recording

A gesture will be recorded as gesture(waitForObject(“:SomeObject”), readGesture(“Gesture_1”)), where Gesture_1 is a file saved in the test suite directory.

This file contains all information needed to replay the gesture. Gesture files are listed in the Test Case Resources or Test Suite Resources areas of the IDE just like scripts, test data and verification points.
gesture-editor

Gesture Interpretation and Editing

A user friendly Gesture Editor is available in the Squish IDE to get a WYSIWYG view of the gestures. In addition script APIs have been implemented to programmatically create, alter and execute gestures.

The object returned by readGesture is of type GestureBuilder. An accelerate and a few geometric transformation methods were also added, giving scripts the ability to adjust its geometry and speed.

Geometry and Speed Methods

transformations

Object GestureBuilder.accelerate(factor)
Object GestureBuilder.rotate(degrees)
Object GestureBuilder.rotate(degrees, originX, originY)
Object GestureBuilder.scale(scale)
Object GestureBuilder.scale(scaleX, scaleY)
Object GestureBuilder.scale(scaleX, scaleY, originX, originY)
Object GestureBuilder.translate(x, y)


For example, to re-size gesture to 75% and move it a bit right and upwards:

gesture( waitForObject(":some_object")
       , readGesture("Gesture_1")
         .scale(0.75)
         .translate(50,-10))


GestureBuilder Creation Methods

Methods to create new GestureBuilders from scratch, add strokes and define their movements were also added.

GestureBuilder(width, height, unit)
Object GestureBuilder.addStroke(x, y)
Object GestureBuilder.addStroke(startTime, x, y)
Object GestureBuilder.addStroke(startTime, x, y, pressure)
Object GestureBuilder.addStroke(startTime, x, y, pressure, size)
Object GestureBuilder.curveTo(duration, controlX, controlY, endX, endy)
Object GestureBuilder.curveTo(duration, control1X, control1Y, control2X, control2Y, endX, endy)
Object GestureBuilder.lineTo(duration, endX, endy)
Object GestureBuilder.build()

Gesture Measurements

Units are defined in either pixels or millimeters. Gesture files are saved using millimeter units, but when creating gestures in a script, users may find it easier to use pixels (thus the flexibility/option).

The entire movement of one finger or pen from touch down to releasing from the screen is called a stroke. During the entire gesture, at least one finger or pen has to be down, with five maximum.

For example a zoom gesture where two fingers move away from each other is represented as:

gesture(waitForObject(":some_object"),
        new GestureBuilder(800, 1280, GestureBuilder.Pixel)
           .addStroke( 500, 400 )
           .lineTo(1000, 700, 100 )
           .addStroke( 300, 700 )
           .lineTo(1000, 100, 1000)
           .build())


A more complete example for rolling your own gesture for Android may be a gesture defined within the boundary of x-axis [-0.5,0.5] and y-axis [-0.5,0.5].
s-shape

  • First we get the device metrics using the Android API and the target widget bounds.
  • Then we translate the gesture to the center of that widget and scale it so that it just fits in the widget, using the widget center as scale origin.

The result of which may appear similar to the following:

var activity = findObject(":Your_Activity").nativeObject;
var metrics = activity.getClass().forName("android.util.DisplayMetrics").newInstance();
activity.getWindowManager().getDefaultDisplay().getMetrics(metrics);
var tb = new GestureBuilder(metrics.widthPixels, metrics.heightPixels, GestureBuilder.Pixel)
             .addStroke(0, 0.5)
             .curveTo(500, -0.5, 0.5, -0.5, 0, 0, 0)
             .curveTo(500, 0.5, 0, 0.5, -0.5, 0, -0.5)
             .build();

var widget = findObject(":Some widget");
var scale = widget.width > widget.height ? widget.height : widget.width;
var centerX = widget.screenX + widget.width/2;
var centerY = widget.screenY + widget.height/2;
gesture(widget,
        tb.translate(centerX, centerY)
          .scale(scale, -scale, centerX, centerY)
          .accelerate(1/2))

The same is also available for Qt and iOS.

Look forward to your comments and use of these newly supported complex gestures!