How to introduce a Test Automation Tool

Each organization has a testing process that needs to be understood before introducing Test automation. Some companies have processes based on strict standards because they work with systems that deal with critical information (such as banks or other financial institutions). Other companies could work on projects where time for testing is very limited, where it is not convenient to spend time in automation scripts (in this case, manual testing of critical areas would be the best approach). Each company is different, so even though Test Automation can bring benefits, we need to analyze how Test Automation will affect our current test process including the benefits and costs of introducing the tool.

Any test tool is intended to make the testing process more effective and efficient. A test tool can provide many benefits, but there are risks that need to be considered before introducing a tool into an organization. “Most of the risks associated with the use of test tools are concerned with over-optimistic expectations of what the tool can do and lack of appreciation of the effort required to implement and obtain the benefits that the tool can bring.”(Williams, 2010)

A blog about testing published this simple and funny cartoon that shows a scenario where Test automation is not justified. See Figure 1.

Figure 1 (The Cost of Automation, 2011)

In order to avoid this scenario; before introducing a Test Automation tool, we need to define the goals that we want to achieve with Test automation.  Then, assess the cost and benefits of the tools that meet our requirements and later decide if it is worth to use any of the tools.

To start, define the objectives for using Test Automation with the interested parties, usually Software Testing Team and managers; it is also very important to know the budget otherwise we can select inappropriate tools for our company.  For each requirement, we can assign a priority or ranking that will help to evaluate which tool will meet our needs. The result of this initial evaluation could be one to three tools that appear to meet the requirements (Williams, 2010).

Once we have done research on the market and have selected a set of tools that meet our needs, we need to conduct a Proof of Concept to evaluate them and define if the tools can be used in our test process.

Proof of Concept

During the Proof of Concept, we will evaluate the selected tools based on how they meet the requirements defined by the stakeholders. When conducting this assessment we need to check these considerations:

• Mandatory features that are satisfied by the tool and extra features
• The environment: some tools may only work in certain environments and using them might require to change our test environment and infrastructure
• Technical Skills:  technical skills needed to use the tool

“Once all proofs of concept have been carried out it may be necessary to amend the requirements as a result of what was found during the tool selection process. Any amendments should be agreed with stakeholders.” (Williams, 2010).

There are three possible outcomes from the Proof of Concept:

• None of the tools meet the requirements sufficiently.
• One of the tools meets the requirement better that the others, so we can select it to be used in our projects.
• The situation is unclear, more information is needed. In this case requirements need to be revised. (Williams, 2010)

Once we have selected a tool, we need to check the cost of implementing it. If the tool will be provided by a vendor, negotiate:  purchase price, annual license fee, consultancy costs, training costs and implementation costs (Williams, 2010).

If we have selected an open source solution, we need to analyze how much time it will take for us to introduce the tool, the maintenance time, and if we have support available or not.

For paid and open source solutions, we should consider the cost of introducing the test automation tool as well as the cost of maintaining it. “Test tools need to be thought of as long-term investments that need maintenance to provide long-term benefits.”(Williams, 2010)

Figure 2 Test tool payback model (Williams, 2010)

Figure 2 demonstrates the costs of Test Automation. “Note that there is an ongoing maintenance of cost of using the tool, but that this ongoing maintenance cost needs to be less than the cost of performing testing activities without the tool if the investment is to be worthwhile.” (Williams 2010)

If we have found a tool that meets our needs, the next step is a pilot project, where we will see how the tool can affect our current test process.

Pilot Project

In the pilot project we will define a business case and have measurable factors to determine if we should use the tool or not.

The objectives of the pilot project are:

• Discover any changes that need to be made in processes or practices

• Determine templates, naming standards and other guidelines for using the tool

• Estimate and quantify financial and other benefits of using the tool

• Learn more about the tool, what can be done and what cannot be done and possible workarounds

(Williams, 2010)

Once the Pilot project is completed, results need to be analyzed by stakeholders to define if the tool will be introduced or not.

This is part 3 of a 3-part series on testing automation.

• Part 1: Is Test Automation Right for Your Project?
• Part 2: Knowing When to Automate Your Testing

References

Peter Williams, (2010). 'Tool Support for Testing'. In: Brian Hambling (ed),Software Testing: An ISTQB-ISEB Foundation Guide. 2nd ed. United Kingdom: British Informatics Society Ltd. pp. 70% 3963, 82% 4670, 83% 4714-4731, 69%-70% 3933-3949.

Andy Glover, (2011). The Cost of Automation.