ForrTel: Selecting An Automated Functional Testing Tool Carey Schwaber Analyst Forrester Research February 8, 2004. Call in at 12:55 p.m. Eastern Time.

ForrTel:Selecting An Automated Functional Testing ToolCarey Schwaber

Analyst

Forrester Research

February 8, 2004. Call in at 12:55 p.m. Eastern Time

Theme

There is no single best functional testing tool;

rather, different tools are more or less appropriate in

different environments.

Agenda

• Test automation in context

• Features and functions — today and tomorrow

• How seven leading tools stack up

• How to pick the right tool for the job

Major categories of application quality tools

Construction

DesignTesting

Deployment Analysis

• Functional testing• Performance testing

• Developer testing

• Application monitoring

Definitions

• Functional tests:

► Tests to verify that application functionality conforms to predefined specifications.

• Automated functional testing tools:

► Tools to build and automatically execute suites of functional tests scripts.

How to calculate the cost of test automation:

A simplified view of the economics of test automation

If a test script will be run every week for the next two years, automate the test if thecost of automation is less than the cost of manually executing the test 104 times.

Cost of automation Cost of manually executing the test as manytimes as the automated test will be executed

Cost of test automation Cost of tool(s)Labor costs ofscript creation

Labor costs ofscript maintenance

Automate if:

Strategies for effective test automation

• Automation as part of a well-defined testing process

• Different processes for different products and teams

• Construction of test scripts out of modular, reusable script components

• Division of labor between testing specialists and automation specialists

• Highly selective automation of test scripts

Where is automation the most valuable?

Start out by automating test scripts that:

• Verify the most critical functionality

• Are the most likely to expose defects

• Are expensive or impossible to perform manually

Use the first suites you build for:

• Smoke testing

• Regression testing

Agenda

• Test automation in context

• Features and functions — today and tomorrow

• How seven leading tools stack up

• How to pick the right tool for the job

Tool features and functions: Today

• Script creation and enhancement

» Record and playback

» Hand coding

» Data-driven

» Keyword-driven

• Script maintenance

» Object property management

» Support for custom objects

• Integration

» Integration with other testing tools

» Integration with other life-cycle tools

Highlight: Keyword-driven testing

• How it works:

» Automation engineers build modular test components.

» Test designers assemble these components into test scripts.

• What it means:

» Test design and test automation become discrete activities.

» Team members’ respective strengths are better leveraged.

• How to get it

» Build it

» Buy it (LogiGear, Mercury, SDT, or Worksoft)

» Download it (Emos, FIT/FitNesse, SAFS)

Tool features and functions: Tomorrow

• Adopt and leverage open standards

• Improve integration with other life-cycle tools

• Reach out to new user populations

• Improve support for distributed teams and organizations

Expect testing tool vendors to:

Agenda

• Test automation in context

• Features and functions — today and tomorrow

• How seven leading tools stack up

• How to pick the right tool for the job

The Wave process

• User survey:

» Online survey of 34 testing tool users to find out what they like and don’t like about their primary testing tools

• Open process to select 80+ evaluation criteria:

» Interviewed vendors, experts, and SIs

• Vendor self-evaluations and vendor interviews

• Extensive fact-checking

» Spoke with 14 customer references to ensure accuracy of vendor claims

Vendors and tools we evaluated

ToolVendor Name of testing suite or companion tools

TestPartnerCompuware QACenter Enterprise Edition+

QuickTest Professional

Mercury Quality Center

e-TesterEmpirix e-TEST suite

Functional TesterIBM RationalTest Manager, Manual Tester,

Performance Tester

WebFTRadView TestView Suite

SilkTestSegue SilkCentral, SilkPerformer

QA WizardSeapine TestTrack Pro

Vendor evaluation — Forrester Wave™Risky bets Contenders

Currentofferings

StrategyWeak

Weak

Strong

StrongLeaders

Strong performers

IBM

Empirix

Seapine

Compuware

Mercury

RadView

Segue

What’s the story behind this graphic?

• Mercury is currently the most capable overall.

• But IBM Rational has the best strategy.

• Compuware comes in third; it is many things to many audiences, but not everything to any single audience.

• Segue and Empirix, which have very different strengths, average out to about the same position.

• RadView did well in spite of the firm’s financial troubles.

• Seapine, an emerging challenger, isn’t ready for the big leagues yet.

Evaluation criteria: Strategy• Corporate strategy

» R&D spending

» Size of developer staff

» Size of support staff and quality of support

• Product strategy» Product road map

» Breadth of target audience

• Price» License cost of product as evaluated

» Licensing flexibility

• Sales strategy» Global reach

» Channel partnerships

Evaluation criteria: Market presence

• Installed base

» Number of users of this product

» Number of customers of all products

• Services

• Employees

• Financials

» Annual revenue

» Year-over-year revenue growth

How you can use the Wave spreadsheet

• Determine how much each of the evaluation criteria matters to you.

• Set your own weights.

• Read the score explanation text to familiarize yourself with these tools and vendors.

• Follow up with demos, trials, and pilots.

How to get to the Wave spreadsheet

Clickhere

How to weight the criteria

• Enter your weightings on the “Scores and weightings” tab of the Wave spreadsheet.

• View the resultant custom Wave on the “Custom Forrester Wave” tab.

Agenda

• Test automation in context

• Features and functions — today and tomorrow

• How seven leading tools stack up

• How to pick the right tool for the job

The three most important decision drivers

1. Support for the applications under test

2. Suitability for your team’s skill level(s)

» Technical skill level

» Skill homogeneity

3. Integration with other in-house tools

» Development tools

» Requirements management tools

» SCM tools

» Performance testing and monitoring tools

Different tools suit teams testing different kinds of apps

• Centralized testing groups often need to test a wide range of apps.

• Testers working with development teams often need to test only a few kinds of apps.

• Firms dedicated to a particular technology (e.g., J2EE or .NET) won’t require a tool with very broad range.

Mercury QuickTest Pro

Segue SilkTest

Compuware TestPartner

IBM Rational Functional Tester

Seapine QA Wizard

Empirix e-Tester

RadView WebFT

Broader range of supportedapplication technologies

Narrower range of supportedapplication technologies

Web only

Different tools suit users with different skill levels

Technical users Nontechnical users

Technical and nontechnical users

MercuryQuickTest Pro

CompuwareTestPartner

Empirix e-Tester

Seapine QA Wizard

IBM RationalFunctional Tester

Segue SilkTest

RadView WebFT

Different tools suit firms with other in-house tools

» Compuware TestPartner integrates with Compuware’s developer testing tools and application monitoring tools.

» Empirix e-Tester integrates with Empirix’s performance testing and monitoring tools.

» IBM Rational Functional Tester integrates with IBM’s development tools — and Microsoft’s.

» Mercury QuickTest Pro integrates with Mercury’s performance testing and monitoring tools.

Each tool integrates with some third-party tools, but a few integrate very well with other tools in the vendor’s lineup and are more attractive to firms that use these other tools:

Takeaways

• Firms should determine their approach to test automation before they begin shopping for tools.

• Three factors contribute to the cost of test automation; tool choice impacts all three.

• While there is no “best” functional testing tool, there are tools that are more or less appropriate given:

» The range of applications under test.

» The level and mix of user skill levels.

» Other in-house life-cycle tools.

Carey Schwaber

[email protected]

www.forrester.com

Thank you!

Entire contents © 2005 Forrester Research, Inc. All rights reserved.

Selected bibliography

• February 3, 2005, Tech Choices “Evaluating Automated Functional Testing Tools”

• February 3, 2005, Tech Choices “Automated Functional Testing Vendor Profile: Mercury Interactive”

• February 3, 2005, Tech Choices “Automated Functional Testing Vendor Profile: IBM Rational Functional Tester”

• February 3, 2005, Tech Choices “Automated Functional Testing Vendor Profile: Compuware TestPartner”

• October 8, 2004, Quick Take “What Hyades Means For IBM, Its Customers, And Its Competitors”

Client Choice ballots for Application Development and Infrastructure are open

through February 11.

Make your opinion count and vote today.

To view Client Choice current ballots, click here: http://www.forrester.com