Squish tip of the Week: Producing Valuable Test Results

Squish tip of the Week: Producing Valuable Test Results

Knowing a test passed is clear, but understanding why a test failed can be more complicated.

Start by outputting relevant and valuable data to your test results. The key is identifying what provides value, and what creates noise.

Value-added items can:

  1. Help identify the reason for the failure more quickly
  2. Assist reproducing the issue
  3. Provide useful information for logging a defect
  4. Illustrate areas of possible concern or application inconsistencies
  5. Produce meaningful data for overall testing metrics

Often when optimizing a test suite, exception handlers and/or event handlers are incorporated. Logging relevant information to the test results upon handling an event or exception, can identify unexpected dialogs, text, or delays in application reaction time.

Simply logging a failure or handled exception without detail greatly limits the value provided by the reports, and can cause delays or oversights identifying potential defects.

Let’s examine the following simple JavaScript try/catch statement:


...
testSettings.logScreenshotOnFail = True
testSettings.logScreenshotOnError = True
...

try {
  activateItem(waitForObjectItem(":MenuBar", "File"));
}
catch(err) {
  test.warning("Warning on initial attempt: " + err);
  activateItem(waitForObjectItem(":MenuBar", "File"));
}

...
testSettings.logScreenshotOnFail = False
testSettings.logScreenshotOnError = False
...

Notice the test.warning message, logging a warning, but not failure to your test results in the event the exception was handled. You can identify a potential issue, or intermittent problem, which might otherwise cause a unnecessary test script failure (if the the catch was not incorporated), or go unnoticed if the exception was handled but not properly logged.

The testSettings.logScreenshotOnFail or testSettings.logScreenshotOnError functions, set to True, capture a screenshot when either a verification point failure or a script error occurr.

Finally, capturing object-level properties and values may pinpoint discrepancies between expected and actual results.

Your report now contains data pertaining to handled exceptions, screenshot evidence of the potential issue as well as the object-level properties and values at the exact moment the issue occurred. Compare that to a simple Object Not Found error, and you’ve already taken strides to increase your test results value with a few simple adjustments.

For more information on the above topics, see:

Log Screenshot on Failure or Error

Error & Exception Handling

Handling Unexpected Events

Logging Object Properties