Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD

1

Strategic Direction for Strategic Direction for Functional Test AutomationFunctional Test Automation

SoftSoftTest 2005Test 2005

Susan Windsor Susan Windsor

Insight Through IntelligenceInsight Through Intelligence

WMHL Consulting Limited, MDWMHL Consulting Limited, MD

2

AGENDA

Future for Functional Testing

History of Functional Automation

Strategic Direction

Automation Frameworks in Action

Impact upon You?

3

Future for Functional

Testing

4

Increased Demand………..?

Global development, with large projects having multi site, multi geography and multi suppliers to contend with.

Corporate and regulatory requirements growing all the time

Business demands new applications, faster and cheaper to obtain competitive advantage

Closer alignment of IT and business to repair lack of business confidence

5

..Or, increased pressures?

Automation frameworks and greater business knowledge requirements, mean reduced numbers of traditional functional testers

Agile development methods mean developers undertake more unit and component testing

Growth of outsourced testing to different geographies, leading to greater competition for the roles

Is the role of the functional tester (who is neither technical or business specialist) dead?

6

History of Functional Automation

7

Record and Playback

“all you need to do…….…!” Easy to record the scripts Extremely fragile Expensive to maintain On average, each test run requires at least 50% of

the scripts to be recorded again

Please tell me no one does this anymore!!!

8

Scripting

Use the scripting language to write scripts that do what you want

Build in as much robustness as possible But, you’re building an application to test an application,

which also needs testing! Maintenance costs can be very high Needs programmer skills Cost of implementation not affordable at project level

9

Table Driven

Like scripting but more flexible and greater re-use Remove items from the script that change, e.g. data Reduces maintenance costs a bit Implementation costs about the same Still requires specialist skills to implement

10

Functional Test Automation is Broken!

Focus on technology rather than business needs 80% of functional testing still manual 60% to 70% of automation tools used for non-

functional testing Typically, traditional functional automation stops at

100 scripts, regardless of test coverage requirement Critical factors; cost of implementation and

maintenance, skills required and asset sharing over different technologies

11

Strategic Direction

12

Automation Frameworks

Wouldn’t it be good if……..– Tests could be documented in common format, regardless of

whether they are manual or automated– The format for the tests resulted in quicker preparation time than

traditional manual tests– Both developers and business testers could understand and use

the same test format– Script maintenance was no longer a requirement– Tests could use any of the test execution tools required by the

underlying technology

Faster planning, faster execution, and far less technical skill required

13

Business Analysts already using them, and use will grow

Home grown frameworks built within organisations to meet business demands

Niche suppliers providing frameworks; try a Google search – 512,000 hits this week

– Seen a handful that appear mature– Latest review by Paul Herzlich (OVUM analyst)

Market Leaders such as Mercury developing Business Process Tester (BPT)

This is the industry direction now

14

Automation Frameworks

in Action

15

The project

Lloyds TSB Offshore in Channel Islands Migration from current banking platform IBIS to Temenos Globus

G13 Tight deadlines due to EU Savings Tax Directive coming into force

on 1st July 2005 Business processes required additional Globus modules Multiple releases leading up to deadline required substantial testing SQS-UK contacted to provide consultancy

16

The Team

SQS-UK recognised the requirement for test automation Joint team from Odin Technology and SQS-UK assembled Working alongside Lloyds TSB staff Odin Technology provided testing technology SQS-UK provided implementation resource and testing know-how

17

Project Approach

Split project roles for Automation Technicians and Testers– Three phase technical approach

Tool Evaluation & Technical Feasibility Technical Implementation Support

– Test Analysis and Design throughout

Tester

Technician

ToolEvaluation

TechnicalImplementation Support

Test Analysis & Design

Project Time

18

Project Phase 1:Tool Evaluation and Technical

Feasibility

19

Tool Evaluation

Automation isn’t always straightforward! GUI Tools don’t work “out of the box” for all environments They will almost certainly require some custom functionality to

interact with the SUT The Evaluation process:

– Assess what custom functionality is required – Assess what involvement is required from an Automation

Technician to provide the custom functionality– Assess if there are any other efficiency improvements that can be

made through application of technology This provides the scope of technical implementation phase

20

Project Phase 2: What would normally happen...

21

Typical Automation Project

Take existing Manual Scripts Rework by hand into automated scripts & function libraries

The bulk of the work…

Building the Object Map– Providing the Mapping between the business terms used to

describe the SUT Interface and the test tools requirements Coding Test Scripts

– Providing the instructions and associated data for the test tool to interact with the SUT and perform testing

22

Object Mapping

Providing the Mapping between the business terms used to describe the SUT Interface and the test tools’ requirements

Application

23

Example

Manual Tester

Test Tool

“The Login Window”

class: MSWDialoglabel: “Login.*”

24

Example 2

Manual Tester

Test Tool

“Username”

class: HTMLEditid: Lgn_Uid

25

Object Mapping in a typical Automation Tool

Usually implemented under different names in different tools– GUI Map– Object Repository– Object Map

Mapping an Object – Start the Object recognition tool– Point at the Object with the Mouse Pointer– Select the Object– Give the Object a meaningful business name– Make manual amendments to the properties if required

26

Test Script Creation

Providing the instructions and associated data for the test tool to interact with the SUT and perform testing

AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then

ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then

rc=AxeMainAPI.TestAbort

Script 1.1




Script 1.1

27

Scripting Approach

Use the Tools Scripting Language to construct business processes

Function Login(ByVal UID As String, ByVal PWD As String)

Dim rc rc = wait_window(“Login”,30) if rc <> 0 Then

log_msg(“Window Not Found”, FALSE)Return

End if focus_window(“Login”) click_on_text (“UserName”) type(UID & “{Return}”) click_on_text (“UserName”) type(PWD & “{Return}”)

End Function

28

System Under Test

Test Automation Tool




Script 1.1




Script 1.1

1. Open Application2. Enter Userid for Authorised User3. Enter Password for User4. Click Login5. Select Create new user from menu6. Enter postcode and house number7. Tick “Secure” user authority8. Enter unique password9. Hit Generate userid10. Validate userid is 8chars long11. Write down userid for later use

Script 1.1


Script 1.1


Script 1.1

Reworking of ManualScripts

Application

ObjectMaps

29

Project Phase 2: What was done differently…

Taught an old dog some new tricks…

30

Test Design Model

A simple structured way of defining tests that is independent of the interface and the means of execution

31

Features of the Test Design Model

Easy to design tests without technical knowledge Well structured Can be used to describe tests for any interface

– Graphical User Interfaces (GUI)– Non-UI Components (WebServices, APIs etc.)

Independent of Execution Mechanism– Manual testing– Automated testing

Not theory - proven 7 year history of implementations In use in UK and Europe at many organisations for defining

testing

32

Test Design Model

Test

Test steps

A single test step

Object

Action

Data

Sub-tests

33

Test Model – GUI Example

TestLogin – enter user credentialsProduct search – enter product IDProduct details – validate product infoLogout

Sub-testLogin as user 1

StepEnter Username “jsmith”

34

A test step

Object + Action + Data Action

– SET enter data, navigation– GET retrieve data– VAL compare expected/actual data

35

How tests are designed

Using Microsoft Excel Business processes are modelled Sequences of steps and data are built up in Sub-Test Tables Sequences of Sub-Tests model business processes

[]Login SignInaction action data action data actionset set jsmith set testing1 set

Username Password

36

Interpreting the Model

There are products that can process the model– To generate Test Automation Scripts

For UI Testing using GUI Automation tools For Non-UI Testing using Harnesses

– To generate documentation for manual testing

37

Interpreting the Model

Manual Scripts Non-UI AutomatedScripts

AutomatedScripts

38

Globus A Self Descriptive Application

39

What is a Self-Descriptive Application?

An application that, through an automated process, can provide a description of its interfaces

Application

40

The Globus User Interface How it works

GlobusDesktop

Application

Application

A description of the interface for an application is

stored in the database

The Globus Desktop interprets the description and

dynamically creates the user

interface

Windows and objects are

presented to the user

41

How Globus Describes its Interface

CustomInterpreter

GlobusDesktop

Application

API

API can be invoked to provide the

description without presenting it

42

A Sample of Globus Information

APP: CUSTOMER_INST

FIELD CUST_SNAMETYPE INPUTLABEL SurnameLENGTH 20SEQ 1

FIELD CUST_GENTYPE OPTIONLABEL GenderLENGTH 1SEQ 7

43

How this Information was used

The Description gives us 3 things:

– The business names used for the objects– The properties the test tool required for object identification– The sequence of the objects in the window

This could provide us with two things automatically:

– The Object Map– The sequence of steps for our TDM sub-tests

44

More on Self-Descriptive Apps

Requirement on development to “name” objects in business terms

Microsoft .NET Applications Windows Forms and Web Applications can provide us

with– Object Maps– Sub-tests for the Test Design Model

This can drastically reduce the cost of automation

45

Axe

Microsoft Excel

Globus




Script 1.1




Script 1.1AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then



Script 1.1

Test DesignModel & Object Maps

Automation Scripts

Overall - The Test System Design

46

Project Phase 2:Implementation

47

The Implementation

Technical Implementation– 9 days of Test Tool Technician

Using Automatic Interface Description in Globus– 1800+ Objects Mapped automatically– 90 Sub-test components generated

30-75 steps per Sub-test– Time to Generate: 42 seconds

48

What was left for the Testers?

Test Design!– Designing scenarios for testing the application– Providing Test Data– Sequencing Sub-Tests to form business processes

49

Designing the Tests

4 Testers – Using the Test Design Model No previous automation experience

40 Days Elapsed time

1012 Complex Banking Scenarios Automated

50

Raw Statistics – Test Execution

Single Script– Automated = 80secs– Manual = 480secs– Automated is 6x Faster

Full Regression Pack– Automated = 22 Hrs (1 Workstation)– = 5.5 Hrs (4 Workstations)– Manual = 132 Hrs– = 22 Man/Days (using 6hrs

productivity per day)

22 Man/Days Testing in 5.5 Hrs.

51

Impact Upon You

52

Recommendations

Find out your organisations direction Decide upon your own future

1. Increased management skills

2. Increased non-functional skills

3. Increased business skills

Ensure you understand how Automation Frameworks support testing so you have a view when asked, because you will be….!

53

Thank YouThank You

Susan Windsor Susan Windsor WMHL Consulting WMHL Consulting Limited, MDLimited, [email protected]

Case Study provided by Odin Technology Ltd Case Study provided by Odin Technology Ltd www.odin.co.uk www.odin.co.uk

Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD

Documents

business testers

business needs80

test formatscript maintenance

test run

business specialist

test execution tools

lack of business confidence

business process tester