Automation from start to finish

Testing Automation From Start to Finish

Liang Gao

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

2

Testing Automation Attribute:

贵expensive


3

Testing Automation Attribute:

冰山iceberg


Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

4

Agenda


Frequently functionality regression testing Sample based performance testing Repetitive work for system and solution

testing

5

Automation is useful for….


Faster turnaround Reduce human errors Cost saving

6

Frequently functionality regression testing


7

Sample based performance testing


Semi-automation Configure 1000 VPN tunnels Create 10000 routing table entries …..

8

system and solution testing



9

Agenda


No matter manual or auto

10

The Goal

Accurate

Testing Report



11

Agenda


12

Remote Control Your Products

Refrigerator

Scanner

Tablet

Telecommuter House with Router

Television

Television - Closed-circuit

Truck

Video Camera

Video Phone

Tunnel

Set Top Box (STB)

Printer

Personal Computer Web Browser

Weather Station

Web Server

Woman - End User

Woman - Sitting


2 ways of software manual testing (or Device Control) Mouse and Eye (expectation) Keyboard and Eye (expectation)

Automation is to repeat Mouse and Eye – GUI automation Keyboard and Eye – CLI automation

13

Remote Control Your Testing


Windows based GUI automation AutoIT

Browser based GUI automation QTP WinRunner Silktest AutoIT Selenium

14

GUI Automation


Expect TCL Expect Perl Expect Python Expect

15

CLI Automation


Most network devices (network equipment) Management interface: Browser or console

Mobile devices Keyboard simulation Serial port, USB port or Bluetooth

16

Embedded devices State of the Union


17

A Sigma Approach (in progress)

Replay keyboard simulation, Screen capture and Image

comparison

Result Verification (manual, only once)

Keyboard simulation, predefined key sequence (XML), render engine and

Screen capture


18

Pick the Tool is just beginning万里长征第一步



19

Agenda


Lessons learned: API (Library) design is key for Productivity Modularity Maintainability Expansion

Don’t start with programming, start with API design

磨刀不误砍柴工 – It is true…

20

API Design


Design pattern or other OO technique apply Need senior software architect blessing More design brain storming sessions, more

reviews, the better. Spend more time on the API design, it will

pay off in the long run

21

API Design


22

API Principle

Control Library

API

Scripts


Script only contains testing actions, no software independent info

23

API Design - Principle

•Connect to Device

•Issue XXX command to send SMS messages

•Verify messages have been sent successfully

•Verify messages saved in the SMS sent folder

•Delete message

•Verify message has been deleted

Device_id = connect_to_device();

Status = Send_SMS(device_id, $string);

Assert($status);

Status = Get_SMS(sent, $string)

Assert($status)

Status = delete_SMS(sent,$string);

Assert($status)


API User Manual

Most important document for script development

API 名称

描述

输入参数参数 1参数 2参数 3

返回值

使用举例

24



25

Agenda


26

Script Development Process

×Ô¶¯»¯½Å±¾¿ª·¢Á÷³Ì²Î¿¼


All common development coding standard applies Version control Standard template Standard header 20% of the code is comments Well indented Use API and functions Enough error checking

27

Define Script Quality



28

Agenda


Testing script has it own quality too Exactly match the manual testing steps Just by reading logs, we know exactly what

scripts are trying to do Device and hardware/software environment

independent Enough debugging message when test case

failed Clean out environment before exit or before

execution 29

Define Script Quality


30

Script Quality Checklist

Others Script Quality ChecklistScript Quality

ChecklistSigma



31

Agenda


Always remember our goal You need at least an execution engine for

Error resuming batch run Report generation

32

Automation Framework ？


Want to design and implement an Automation Framework?

Checkout

<<Automation Framework: Design and Implementation>>

33

Automation Framework



34

Agenda


35

Execution Process

Always remember the goal


Image upgrade process Parallel execution Total run time Debugging time Report collection and generation

36

Execution Process


Where to get the image Can you automatically get the image and

upgrade to your product Call this a “clean” procedure

37

Image upgrade process


To save total run time – sometimes even automation is too long

This is where the script need to be environment independent

Be aware on the equipment cost issue How many parallel scripts can an engineer

run and manage (also equipments)

38

Parallel execution


Total run time per engineer Total Debugging time per engineer Total reporting time per engineer

39

Run Time


If it is parallel, need a process to parallel report collection and generation

Testing Information Database? Web Interface?

40

Report collection and generation



41

Agenda


Who write the scripts? Who write the framework Who execute the scripts Are they happy

42

Team Management


Whoever write the test plan, write the test scripts Knows test case the best Not professional coder Hard for project management

Professional scripter Need to understand the test case first High productivity Good Quality

43

Who Write Scripts


Or any other testing related tools If a tester develop a great testing tool, who

maintains it? Who to coordinate the common share library

development Who to develop testing tools if there is a

request Most of the time developer does this. If have resources, think about a Tools Team

44

Who Write Framework


Whoever write the scripts, execute the scripts Low productivity Bad script portability Unpredictable schedule (bad for scheduled release)

Dedicated resources (professional executer) Streamline resources High productivity Fixed total run time Most modern companies do it this way

45

Who Execute the Scripts


Test cases writer and professional scripter Scripter and executer

46

Interaction process


Repeatable test cases

47

Case Writer and Scripter


Script integration process Newly develop scripts submit to runner Use “State” to manage

Script maintenance process New image change Use “State” to manage

48

Scripter and Runner


Manual tester, scripter are doing creative things

Script runner may be board Role rotation Clear role for a clear career path

49

Team Building


50

Streamlining the team

脚本开发

脚本执行

脚本开发脚本开发脚本开发

脚本执行脚本执行


No matter manual or auto

51

The Goal

Accurate

Testing Report

Automation from start to finish

Documents

consent of sigma

training purpose

automation framework

testing automation attribute

sigma approach

imagecomparison17this

expensive2this material

api design design pattern