Automatic and manual tests
Whether we like it or not, some tests will not be automated.
Instead of separating manual from automatic test specifications – the latter in a well specified machine readable format, the former in specification documents – let all tests be specified in the machine readable form (as talked about here).
Scenarios for manual tests can be thought as checklists, where the tester is prompted by the tool to perform an action and record it’s result. In 99% of the cases a question answered with a yes/no suffices to evaluate test success.
In our contrived example (first talked about here), say that a test asks for a measurement using an oscilloscope, and that it has been deemed impractical (or too expensive) to have a programmable oscilloscope. Instead, a snapshot of the expected curve is taken and the tester should perform the measurement and make sure it matches. It could go something like:
<scenario attended=“true”>
<echo text=“Please perform oscilloscope measurement.”/>
<display img=“expected.jpg”/>
<yesno>Does the measurement match the expected result?</yesno>
</scenario>
The advantages of such an approach is that the tester focuses only on the relevant test action that cannot be automated. The rest (test preparation etc.) is done automatically by the tool. The tool also takes care of validating the tester’s input, logging etc. This consistency reduces the possibility of errors.
Ofcourse it also simplifies management of test specifications using the same facilities used for automated tests.
Test management, test execution
Up to now we have been discussing the need for machine readable test specifications with a sample idea using XML and including manual as well as automated tests.
So there is a pile of test specifications in the desired format. From these specifications information can be extracted to:
- Create overview documents and lists with information about the test cases (test management)
- Execute tests and store the results (test execution)
The core problem is the existence (or lack of) a system that automates test execution in the sense that it integrates various tools that automate tests. Something like this would be very useful in a CI environment.
So lets present a system to automate test execution using a text (XML) based database of test specifications.
You just have to wait for the next installment!
Continue Reading…
Posted by Vassilis Rizopoulos on Apr 16, 2007
The simplest way to define a test is as a sequence of steps which can succeed or fail.
The test is successful when all steps are successful, or it fails when as much as one step fails.
In it’s simplest form a test consists of a single step, a script or a call to an application that runs a(nother) script.
A test specification would contain a description of the test, pre- and post-conditions together with the steps required for execution plus information about the requirements leading to this test.
Pre- and post-conditions are actually satisfied through steps taken during the test execution and become an integral part of the test.
A step can be any action taken when testing but in the context of automated tests it can be defined as a single command for which success or failure can be easily determined (the easiest way being that of a meaningful exit code).
A specification can thus be expressed in two parts: human-readable text that describes the test and can be used in documents and reports and a machine-readable scenario that defines the sequence of steps/commands to execute in order to run the test.
Putting the two together in a parseable format is the next logical step:
<specification id=“id”>
<title>Super Test</title>
<description>Why does this test exist? What is it’s purpose?</description>
<scenario>
<step/>
<step/>
</scenario>
</specification>
Parsing these specifications allows for:
- Execution of the scenario elements.
- Generation of overview documents.
- Generation of reports for each execution.
To do all of the above, lets separate the three major blocks of functionality and add a manager to handle coordination:
The above XML will not work for every project. Things like units, test groups, attributes that define types, test conditions etc. are universally…different.
Establishing a format that satisfies the requirements of all projects is an effort that will lead to a too complex, all-knowing and nothing-achieving behemoth. In short it would be futile.
What can be done is define a set of conventions and boundaries within which such specifications can be managed.
This leads unavoidably to a tool-set whose architecture allows it to be customized and adapted for specialized usage within those boundaries. As much as one would like to avoid it, the term framework immediately springs to mind.
The specification parser is the part of the system that needs to be adapted every time.
The goal is to have a simple and intuitive way of defining the scenario (e.g. the sequence of execution steps) in a problem-specific way. In other words, define a testing DSL for the project.
Using as basis the contrived example project from part I, a DSL-expressed scenario could be:
<scenario>
<flash/>
<start_sniffer log=“testcase.log”/>
<guitester script=“somescript.gui”/>
<stop_sniffer/>
<analyze input=“testcase.log” script=“check_for_errors.usb”/>
</scenario>
Additionally in many cases the specification itself will have to contain more information (i.e. requirement tracing).
In a
later installment I will present a solution that provides enough flexibility for the adaptation of the parser without exponentially increasing it’s complexity.
Continue Reading…
Posted by Vassilis Rizopoulos on Apr 15, 2007
This is the first of a series of articles on test automation, test management and test automation management within the bounds of software development.
In this first part I’ll lay down the problem bothering me and in subsequent articles I’ll talk about a concept and an implementation for a solution.
Automating Tests in a Heterogeneous Environment
Test Automation is the process by which the execution of tests is undertaken by the computer without user intervention.
Lets not go into details as to
why test automation is essential under most circumstances, suffice to say that it frees resources and saves time.
There are several tools in the market that automate tests. In almost all cases the tools concentrate in solving one of the following:
- Tests for programs written in a specific language (all unit test frameworks)
- Tests for a specific plattform (be it OS, framework etc.)
- Tests for UIs (Web or GUI)
- Special tools for testing i.e. embedded platforms
Usually the solution consists of a language to write the test script (be it Python, Java, Ruby, Delphi or VBA) and an environment that will run one or more scripts, log the output and return a success or failure result.
Let’s state the problem studying a contrived example, where we are testing a product that controls an embedded device. Development artifacts consist of a Windows-based application which presents the GUI we want to test and the embedded device’s firmware.
To make things a bit more complex the application and the device can communicate over different mediums, say a USB/serial interface and a TCP/IP-based network interface.
We have several tools for testing the different aspects of the system:
- Unit tests for the application that run on windows
- Unit tests for the embedded device functionality that run on windows and on the hardware itself.
- A GUI test automation tool
- A USB/serial traffic analyzer
- A simulator against which functional tests are written
- etc.
Testing the product would include running unit tests, functional tests in simulation mode, using a test rig that runs integration tests for all combinations of communication mediums, using a multiple test server setup for system tests in real-world conditions. There will also be stress testing, and long running tests.
Going past the unit test stage, tests are usually specified giving a short description, stating the preconditions for the test and the expected results. Usually most test tools do not explicitly provide for the setup of the test environment (i.e. the USB analyzer script will assume the correct firmware to be present, the GUI tester will expect the application to be running etc.), although most test automation tools will provide ways of running external scripts, where one can implement such preparation steps.
So extra scripts are needed, for tasks like starting and ending the application, flashing the device, generating test data etc.
In our highly heterogeneous project environment we end up with at least five different ways of implementing test cases. Managing these test cases becomes a case for “Excel Magic”, where the appointed test manager tries to keep tabs on all scripts, descriptions and test conditions through the ingenious use of spreadsheets and hard work. Possibly (and with luck) a good test management tool can help ease the complexity, but experience points to the Excel solution most of the times.
But there is no single entry point to all the tests! In the best of cases elaborate batch/shell scripts will be written to provide the glue integrating the tools together.
Integrating the tools
In our scenario Excel becomes the integrator on the test management side, while on the test execution side, batch/shell scripts take over that role.
It is a race against time that becomes very difficult to win the closer a team comes to the deadline: Every change in the testing environment (test case versions, descriptions, software version numbers) needs to be documented and tracked. Broken shell scripts must be found and corrected.
It becomes obvious that we need a unifying layer for all the different tools. A system that will allow the integration of several tools and provide that single entry point.
Ideally it will also provide us with:
- A way to define test cases whose execution requires several different tools in a consistent and simple way.
- A way to automate the execution of said test cases.
- History and reporting on test execution which can be directly traced to specified builds of the tested system using known versions of the test case set and the tools.
It turns out that if you try to build software that does all of the above, you end up creating a framework for building such a system, which is then customized according to each project’s requirements.
I’ll talk about one such solution in the next installment of the series.
Continue Reading…
Posted by Vassilis Rizopoulos on Apr 13, 2007