The pyramid approach to testtool selection

.......MAGAZINE

August 2013 $10.95

The Testing Tsunami

Rethinking the Role of AutomAted softwARe

testing

good things Come: Syncronization in HP Functional Tester (QTP)

Software teStingAutomAtedAutomAtedAn AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.com

Continuous Load Testing

Agility Applied to AutomAted peRfoRmAnCe

testing

PlannedAgility

Selecting the Right Tool

the pyRAmid AppRoACh to seleCting A tool to meet

youR needs

22 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2013

Selecting the Right Tool

The Pyramid Approach to Test Tool

Selection

By Bernd Beersma

www.automatedtestinginstitute.comAugust 2013 23Automated Software Testing Magazine

A test tool is one of the best tools

to help us throughout the test cycle

because test tools can support almost

all activities we do in testing: test

management; test design; etc. But if we

do a Google search, we get hundreds of

thousands of hits for test tools. So how

do you as a tester, test manager or test

specialist know how to select the best

tool for your day-to-day business – one

that fits your purpose?

“

“

Selecting a tool, that eventually fails to meet your requirements or expectations, can lead to:

• Poor automation

• Frustration

• Lack of commitment

• Bad business case

• Shelfware

The tool selection pyramid is a structured selection approach that will help to avoid these pitfalls.

The Tool Selection PyramidThe Tool Selection Pyramid is composed of four different phases:

1. Long List

2. Short List

3. Proof of Concept

4. Pilot

Phase 1: Long ListThe outcome of the long-list phase is, as the name states, a long list of test tools. But getting there requires a few steps:

1. Identify your stakeholders.

If we want to implement a test tool in our organizations, this will influence the test team and other departments, IT and the maintenance departments, for example. We want to identify all those who are involved

2. Define measurable, feasible goals for Test Automation (TA).

24 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2013

Be sure to involve all the stakeholders. This again creates commitment, but the other departments can also add goals of their own. For instance, business wants to decrease the time to market when you select a tool for automated regression testing.

3. Create a business case for TA. We need to set up a business case to be sure the project has enough support in the organization, and can be brought to life.

4. Set up a project team. The next thing we need to do is set up a project team for the tool selection project. Again, be sure that people from the different identified departments are on this project team, although the majority of the team can be from the test department. Someone from IT and maintenance – but also a developer – can be of real value to your team not only to create the support within the organization, but they need to help identifying requirements that are not test-related but are an essential part of the selection process.

5. Define general requirements. In this step, focus on defining general requirements like the need to automate

regression test or need to support a defect-management process. These are on a high level, and mind mapping is a good way to gather these requirements. This approach helps to get the high-level requirements for the tool in a fast and efficient way.

6. Gather general tool information. After defining the general requirements, explore the tool market. Look at tool-related websites and magazines, and events.

7. Create a long list. This list contains the tools and vendors we think should enter the next phase in our tool selection pyramid, the short-list phase.

Before entering the next phase, there is always a moment of consideration about whether or not we are ready to enter the next phase. This go/no-go moment gives us the possibility to restart the previous phase or start the selection process over.

Phase 2: Short ListThis phase also consists of several steps:

1. Define detailed requirements.

In the long-list phase we came up with

general requirements. In this phase, we need to go a little deeper into these requirements. For example, we needed a tool for automated regression testing. Now, with a mind map, we find the need for tools to support the use of external data, or tools to support cross-browser testing.

2. Gather the detailed requirements.

Add them to a scoring card and determine weight and priority. The scoring card is the basis for the next step, the Request

for Information (RFI).

3. An RFI goes to all the vendors on the long list.

Sometimes this is not possible if there is no vendor or party to whom to send the RFI. But if you still want to score, you can always see if you know an independent consultant who has experience with that tool to help you to fill out the form, or find a third party experienced with the tool and ask them to fill in the RFI. In the worst case, the project team has to fill it out. After getting back the RFI, you need to determine overall score. As you can see, this is only an example, but already 20 different main categories are identified here. So this shows tool

After drafting the requirements we invite

the vendor(s) to set up a plan for the pilot

together with members of the project

team. This needs to be a solid plan, with

fixed timelines and dates, because this

pilot phase needs to have an end, after

which the project team can start the

evaluation.

www.automatedtestinginstitute.comAugust 2013 25Automated Software Testing Magazine

selection is not that easy.

4. A demo.

Invite at max the top five scored tools to do the demo. The demo is like first contact, and you never get a second chance to make a first impression. Be sure to invite all the project members, stakeholders to let them see the demo. They can help you to decide afterwards which tools to put on your shortlist. Preferably three vendors will end up on the short list and this completes our second phase.

Again, before entering the next phase, there always is a moment of consideration if we are really ready to enter the next phase. This go/no-go moment will always give us the possibility to restart the previous phase or even start all over again with the selection process.

Phase 3: Proof of ConceptNow we entered our Proof of Concept or POC phase, during which several vendors or consultants are invited to do a proof of concept. This is like a test drive for the tool, to see how the tool performs in our situation and environment, and to see if the tool meets our requirements. Again, this phase is completed by executing several steps.

First of all, we draft the requirements for the successful completion of the POC. These requirements are the same for all participants of course, to create a fair competition. The tool must support all technologies that are part of a certain business process chain. After completion, the requirements drafted will be also part of the acceptance criteria for the POC.

Second we invite the vendors to do the POC, but this is a combined effort between us and the vendor. We need to be clear on what we expect of the vendor. This not only counts for the availability of the Software Under Test (SUT) but also of internal resources who will be executing the POC with the vendor. We need to setup a plan so we are sure the POC will be executed and finished on

time. Be sure that all systems are ready when starting with the next step in this phase, the actual execution of the POC.

Keep in mind that user accounts have to be set up, as does a test date, etc. If you are sure that everything is good to go, start executing the POC. This is also a combined effort. The more you involve the team by the selection process, the better the results of the selection will be.

After you and the vendors finished all the POCs, an evaluation of the POCs needs to be conducted. We will do a session with the project team and will see if the POC meets our requirements. The goal of this step is to choose the best tool, preferably one tool; you have selected two tools that meet all the requirements. If this is the case, no worries, we will come up with the definitive selection in the pilot phase.

But, we haven’t completed this phase yet, the next thing to do is to send a Request for Proposal. Even though I strongly believe that money is a bad driver, we still want to know what licenses, trainings, maintenance cost. We need this input to validate our business case. In the case of an open-source tool, the same procedure as for an RFI is started. Now with all this information gathered we are ready to update the business case and confirm we are still on track with our tool selection.

This completes this phase and we will be entering the pilot phase, but not before we do a go/no-go, but again, we are always allowed to restart the phase or even the whole selection project.

Phase 4: The PilotDuring this phase, one or two tools will be used in the pilot project for a longer period of time. It’s best to setup a separate project for this phase in a test environment.

Like in all of the other phases, we have to follow a few steps to complete this phase. But, after completing these final steps we have selected the best tool to help us with our testing, so completing this phase is worth the wait.

Again we start with the requirements for completion of this phase. We want to evaluate the outcome of course against our acceptance requirements. An example of a pilot requirement can be, “testers must be able to automate the regression test by themselves.”

After drafting the requirements we invite the vendor(s) to set up a plan for the pilot together with members of the project team. This needs to be a solid plan, with fixed timelines and dates, because this pilot phase needs to have an end, after which the project team can start the evaluation. After setting up this plan, we start the actual pilot with a kick off, so all participants in the pilot know what is expected of them, but also to create a feeling of collaboration between the involved parties.

No we are ready to actually start with the execution of the pilot. This is a combined effort with shared responsibility between the vendor and the project team. The goal is to prove that the tool can be implemented within the project and the organization.

Now that we have executed the pilot, final step is evaluation of the pilot. Gather the project team and see if the selected tool meets the requirements and acceptance criteria. And if all requirements are met and all systems are go, we have completed our selection project and the best tool for the job is ready to be implemented.

That leaves me with some final words of advice. Whenever you start selecting a tool, be sure to create awareness within your organization. Make it a combined effort and a shared responsibility between you and all the possible tool vendors or consultants, and always keep in mind that if you don’t feel comfortable after completing a phase, start again.

Learn more at TestKIT OnDemand:

http://www.ondemand.testkitconference.com

Wait! there’s more!

selection pyramid