AMR Research Report April 2005 by Lance Travis Remote application testing provides companies with a valuable, lower risk way to establish a relation- ship with an offshore outsourcing service provider. Benefits include lower testing costs and improved quality. More important, testing services provide a way for companies to mature their offshore outsourc- ing skills in preparation for a long-term development relationship. Offshore Remote Application T esting Can Cut Cost of Quality by Up to 75%
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
A M R R e s e a r c h R e p o r tA p r i l 2 0 0 5
by Lance Travis
Remote application testing provides companies witha valuable, lower risk way to establish a relation-ship with an offshore outsourcing service provider.Benets include lower testing costs and improvedquality. More important, testing services provide a way for companies to mature their offshore outsourc
ing skills in preparation for a long-term developmenrelationship.
Offshore Remote Application Testing Can Cut Cost ofQuality by Up to 75%
No portion of this report may be reproduced in whole or in part without the prior written permission of AMR Research. Any writtenmaterials are protected by United States copyright laws and international treaty provisions.
AMR Research offers no specific guarantee regarding the accuracy or completeness of the information presented, but the professional staffof AMR Research makes every reasonable effort to present the most reliable information available to it and to meet or exceed anyapplicable industry standards.
AMR Research is not a registered investment advisor, and it is not the intent of this document to recommend specific companies forinvestment, acquisition, or other financial considerations.
Offshore Remote Application Testing CanCut Cost of Quality by Up to 75%by Lance Travis
Offshore software testing saves companies up to 75% over in-house
costs, improves quality, and is a lower risk way to develop mature offshore
outsourcing skills.
Software defects continue to proliferate, but the cost of ensuring quality is high. Software testing can account for 30% of develop-ment time and many companies’ staff testing positions withhigh-cost, on-site independent contractors. Companies must
decide if offshore testing provides a solution to their cost of quality problems.
After interviewing 14 service providers that offer offshore testing services and reviewingthe results of 16 customer engagements, AMR Research concludes the following:
• Low cost, faster test cycles, and improved quality are benefits from offshore testing.
• Segment testing reduces the risk of offshore outsourcing.
• Engagement size and add-on services distinguish the service providers.
Software quality issues continue to plague CIOs andsoftware product development organizations
Academics report that in 2002, software defects cost $60B to x, and 50% of softwaredevelopment costs were related to nding and xing defects. Obviously, many com-panies aren’t getting the job done with in-house resources or on-site contractors. eymust decide whether the benets of using offshore software testing services outweigh
the risks.
Low cost, faster test cycles, and improved quality arebenefits from offshore testing
Companies interviewed for this Report consistently reported cost savings, faster testcycles, and improved quality as the benets they received from offshore testing.
Many companies report 60% to 75% savings by moving testing offshore
More than half of the companies that we talked to realized savings between 60% and75% over in-house costs. Rather than letting people go, one manufacturing company was able to move the replaced in-house staff to other projects and stil l save $300K
per year. Some companies report that rather than realizing savings they maintainedspending at roughly the same levels, they increased the level of testing because of thelower costs associated with offshore resources.
Savings depend on the percentage of offshore resources and on the amount of overheadassociated with managing the offshore relationship. Some projects, such as quality con-sulting projects, require a much higher on-site presence and result in smaller savings.
Leveraging staffing exibility is another way companies saved money using offshoretesting providers. Rather than keeping underutilized in-house resources, companiesramped up and down outsourced resources, depending on the recurring demands of
their development cycle.
Faster test cycle times result from offshore testing
Companies are able to leverage time zone differences to accelerate test cycles in a follow-the-sun philosophy. Daytime testing occurs in India and is then handed over to the United Statesat the end of the day in order to leverage the 9.5- to 12.5-hour time difference between thetwo countries. One company reduced its test cycle by half, and then improved quality byadding more test scenarios.
When leveraging the 24-hour clock, companies had to make arrangements for off-hours support of their test environment. Before it made arrangements, one companylost 48 hours’ worth of testing because a server failed on a Sunday evening (U.S.time) of a three-day weekend. e Indian test team was unable to work Monday andTuesday in India because no one was available to x the problem in the United Statesuntil Tuesday morning U.S. time (Tuesday night India time).
All of the service providers interviewed for this Report offer services to automate tests. After using the offshore service provider to automate its 3,500 tests, one companyreduced its test cycle from 90 days to 29. e cost of automating the tests was muchless than the cost of continually running them manually.
Quality improvement results from additional testing, and eventuallyroot-cause analysis
Companies reported immediate quality improvements as a result of test automationand reduced cycle times. Test automation allows them to run more tests during a test
cycle. Because they are able to run more tests, they are able to nd more problems.One manufacturing company reported that because it never allocated enough resourcesto in-house testing, it never would have been able to automate and test as thoroughlyas the offshore provider. A nancial services company reported a dramatic decrease inthe number of post-go-live reported defects after the offshore provider automated thecompany’s test scripts.
As the offshore testers become more familiar with the applications being tested, theybegin to help their customers nd the root cause of defects and eliminate future prob-lems. However, this ability doesn’t happen right away. Companies reported that ittook between 14 months and 3 years before the offshore testers had sufficient familiar-ity to be effective in nding the root cause of problems.
Segment testing reduces the risk of offshoreoutsourcing
Testing work can be outsourced offshore with less risk than application development ormaintenance because of the following:
• The work can be easily segmented into process consulting, test execution, test
automation, test development, and quality improvement consulting. Processconsulting allows companies to leverage the service provider’s best practices toimprove how they test. Process consulting, test execution, and test automationrequire minimal interaction with business users, allowing an IT organization toestablish a relationship with an offshore provider without involving the business users.Test development requires working with the business user to understand the purposeof the software to be tested and their acceptance criteria to be understood.
• Testing doesn’t require the offshore resources to touch the application sourcecode. The offshore resources can gain experience with the customer’s developmentprocess and its applications without running the risk of breaking or corrupting anapplication.
• Most companies underinvest in testing. An effective offshore testing program canconvince outsourcing naysayers that offshore work can be effective.
• Testing engagements can slowly ramp up. It is possible to start small and workup to a larger engagement. The number of resources supplied by the outsourcer forthe customer engagements studied in for this Report ranged from 6 to 350 people.It is also possible to start with lower value activities such as test execution, and rampup to higher value activities such as root-cause analysis (see Figure 1).
Not all companies that we talked with were convinced that testing is a good candidatefor a company’s rst foray into offshore outsourcing. One company had difficultysending test execution offshore because its development process and documentation were nonexistent. It was only after the offshore company was able to drive processimprovements into the development organization that the company was able to out-
Separate testing and development organizations importantto customers
Many customers selected their testing service provider because it offered a testing serviceseparate (reporting to different management) from its development organization. Two of
the companies selected one offshore company for development and another for testing.e customers felt that having separate organizations ensured that testing and quality
concerns would not get subordinated to development schedule concerns. Having mul-tiple companies involved created a healthy competition that beneted the customer.
One company using separate organizations for testing and development felt that it wasessential to use in-house resources as the quality advocate. e person in charge ofthe project had trouble getting the Indian resources to push hard enough for processimprovements.
Despite the lower risk, strong on-site leadership is still required
All but one company interviewed stated that having the outsourcing company sup-ply on-site resources was essential. e on-site resources are necessary to manage theow of work to and from the offshore locations. In addition to having on-site person-nel, companies recommend having strong in-house managers handle the relationship.Companies must establish ground rules for the relationship and cannot just throwthings over the wall to the outsourcer and expect success. For test execution projects,companies typically are able to move 70% to 80% of the test resources offshore, butnd that keeping the remaining 20% to 30% in-house is essential.
Offshore testing does require some infrastructure investment
e offshore resources need access to different systems than offshore developers require.For example, one nancial services company provided access to the systems that areintegrated with the to-be-tested systems to ensure a complete systems test. In this case,the company extended its private network to India.
Some companies had problems with sending test data to India. Either governmentregulations or contracts with their customers prevented them from moving test dataoffshore. In cases such as these, the offshore providers offered tools to facilitate the cre-
ation of unrestricted test data based on live data.
Engagement size and add-on services distinguish theservice providers
AMR Research interviewed a broad cross-section of service providers for this Report: Accenture , AppLabs, Cognizant , HCL Technologies , Headstrong , Infosys, ITCInfotech , Keane, Patni , Rapidigm , Sapient , Satyam, Tata Consultancy Services (TCS), and Wipro . All the service providers interviewed offer a suite of test consult-ing and software lifecycle testing services (e.g., manual testing, functional testing,performance testing, acceptance testing, and test automation). All have provided testservices for multiple clients, with AppsLab, Cognizant, Infosys, Keane, Patni, Rapidigm,Satyam, TCS, and Wipro having more than 50 customers.
e service providers support multiple engagement models: time and materials, xed-priced projects, and dedicated test centers. However, a majority of the offshore testingengagements are time and materials contracts.
e service providers examined fall into three categories: large offshore rms, smalleroffshore rms, and traditional consulting rms.
The larger rms support the largest engagements
Cognizant, Infosys, TCS, and Wipro are primarily focused on large engagements.eir average engagements number over 50 test resources and last a year or more, and
their largest engagements are double to triple the size of the smaller rms’. Considerthese rms for long-term, large test engagements (see Table 1). You can start small with these rms, but unless you show solid plans to scale up the relationship, you maynd that your requirements are mismatched with their delivery capabilities.
Firms supporting smaller engagements are ne for short-term staffaugmentation and midsize, longer term engagements
e average deals of smaller rms—AppLabs, HCL Technologies, Headstrong, ITCInfotech, Patni, Rapidigm, and Satyam—involve 5 to 10 test resources and last 6to 9 months. eir largest deals range from a low of 15 resources (Headstrong andRapidigm) to over 60 (Patni and Satyam). Consider these companies for small staffaugmentation projects. Consider the high end of this category (AppLabs, HCL, Patni,and Satyam) for larger, longer term projects.
Note that project size does not always determine project complexity. Smaller projectscan be quite complex, involving multiple applications, geographies, and quality con-sulting. Despite the relatively smaller size of their engagements, Patni and Satyamhave some of the more complex engagements.
Traditional consulting rms enter the offshore testing market
Accenture, Keane, and Sapient fall into this category. ey have recently supple-mented their implementation services with stand-alone offshore testing capabilities. Accenture created a dedicated test center of excellence in India in 2004 and now has
over 500 test resources. Keane also created a dedicated test center in India in 2004and has more than 300 test resources. Sapient provides testing services through itsdevelopment resources. To date, its testing engagements have been small compared tothe large offshore rms. Accenture's and Keane’s largest engagements involve 30 and26 test resources, respectively. Use a testing engagement with these providers primarilyas an entry point into their application implementation and consulting services.
In addition to the standard test consulting and quality offerings, some of the providersoffer unique test services that may be of specic interest:
• Usability testing —Cognizant, Patni (in 2005), Rapidigm, Satyam
• Packaged application testing —All of the service providers offer some level of pack-
aged application testing, but those planningSAP and Oracle upgrade testing shouldconsider the service providers with the deepest SAP and Oracle implementationskills: Accenture, Cognizant, HCL, Infosys, Patni, Rapidigm, Satyam, TCS, and Wipro. Note that Headstrong only offersSiebel testing services.
• Business strategy and test strategy alignment offered by Accenture, Infosys,Keane, Satyam, TCS, and Wipro —In addition to consulting about test technologyand test processes, they can help you prioritize your test strategy based on the impor-tance of specif ic applications and application functionality to your business strategy.
Source: AMR Research, 2005
Table 1: Attributes that indicate service maturity
Offshore testing services have lower-than-expected attrition rates
e service providers have focused on making testing an attractive career path andminimized attrition within their organizations. e companies offer extensive techni-cal and business training and career advancement opportunities in order to attract and
retain quality resources. As a result, attrition is often lower in the test organizationsthan in the rest of the Indian companies. Infosys, ITC Infotech, Rapidigm, Satyam,and TCS reported the lowest attrition rates within their testing organizations. Long-term customers benet because of consistency of staff on their projects.
Common tools facilitate workow between on-shore and offshore
All of the service providers interviewed support standard third-party test tools fromcompanies such asCompuware , Empirix , IBM Rational , Mercury , and Segue, andopen source tools such as JUnit . e providers use standard tools such as Mercury
TestDirector, IBM Rational’s ClearQuest, and Excel spreadsheets sent via e-mail to facili-tate the ow of information between locations. Whenever possible, companies shouldleverage the relationships the service providers have with tool vendors when establishingthe tools to be used in the engagement.
Closing Comments
Testing is a low-risk, high-reward entry into offshore services. Companies that havebeen struggling to develop an offshore outsourcing strategy should look to a limitedtest engagement as their rst step. Outsourcing test execution and test automationlimit dependence on the business units for support, offer signicant cost savings andimproved quality, and give companies a chance to hone outsourcing skills before mov-ing on to more extensive offshore engagements.