Visualization and Statistical Methods in High Volume Test Automation of Embedded Devices

7/31/2019 Visualization and Statistical Methods in High Volume Test Automation of Embedded Devices

1/46

July 15, 2008

C. Martin Taylor

Visualization & Statistical Methods

in High Volume Test Automationof Embedded Devices

CAST 2008 Toronto, Ont. Canada


2/46

2 | Copyright 2008 Texas Instruments, Inc.

Abstract

An industrial experience report from Texas InstrumentsEducation Technology (TI Calculators) division.

Describes how we combined

High Volume Test Automation techniques

Data visualization

Statistical regression methods

to detect

memory leaks

device crashes

performance problems

in the new TI-Nspire math and science learning handhelds. Describes how these techniques have been used at TI to compare

results between various TI-Nspire software builds.


3/46


Presentation Outline

Background Concepts & Tools

High Volume Test Automation

TI-Nspire Devices the System Under Test

TI-CAT Test Automation Framework & TI-CAT for TI-Nspire Specific Applications of HVTA to Embedded Device Testing

Detecting Large Memory Leaks with RRR

Detecting Device Crashes with RRR

Measuring Small Systemic Memory Leaks with RRR Detecting Performance Problems

New Developments (since publication deadline)

Improved RAM Available Metric

Automated result graphing and trend slope computation

Detailed memory leak reports with instrumented code


4/46


High-Volume Test Automation & RRR

High-Volume Test Automation (HVTA):

The execution of large numbers of automated tests that alreadypass in order to expose longevity errors that are generally hard

to find. [Cem Kaner & Pat McGee, 2004]

Extended Random Regression (ERR)

An HVTA technique of repeatedly executing a set of already

passing automated tests in a randomized fashion. Renamed Repeated Random Regression (RRR)

by McGee & Taylor in 2004.

Used at Texas Instruments Ed. Tech. division since mid-2004.


5/46


TI-Nspire Devices

TI-Nspire and TI-Nspire CAS(Computer Algebra System)

Embedded devices with:

Custom ASIC

Integrated ARM 926 CPU

32 Mb of RAM

32 Mb of Flash

USB port

Download new S/W via USB

Nucleus real-time OS

TI-Nspire TI-Nspire CAS


6/46


TI-Nspire Device Software

TI-Nspire device software has a Document model

Each Document can have 1 or more Problems

Each Problem can have 1 or more Pages

Each Page can have 1-4 Applications

Each Application can be one of:

Calculator

Graphs & Geometry (G&G)

Lists & Spreadsheet (L&S)

Notes

Data & Statistics (D&S)

Documents can be saved

Saved documents can be transferred between Device

PC

Mac


7/46


TI-CAT Test Automation Framework

TI-CAT = Texas Instuments Calculator Automated Testing

Developed at TI since 2003

Enables automated testing of TI calculators

Supports:

System testing at the user interface level

Software library testing at the Application ProgrammingInterface (API) level

RRR techniques are applied at the user interface level


8/46


TI-CAT for TI-Nspire

TI-CAT Test Engine runs on a PC

Sends Events and Commands to device via USB

Test Engine Service runs on the device

Injects Events into TI-Nspire event queue Keystrokes

Mouse actions

Special events

Executes commands like: GetScreen

CursorBlink ON/OFF

Standard device services provide:

InstallOS GetDeviceInfo

Includes RAM Available


9/46


TI-CAT for TI-Nspire - Simplified Test Architecture


10/46


Memory Leaks in Automation or Comms Software?

Are there any memory leaks in these tools?

Try a simple automated test

Send keys to display the TI-Nspire About dialog box

Capture this screen

Send keys to remove the dialog box

Request RAM Available from the device

Exercises relevant parts of the test & communications software only involves a minimum of functionality

in the TI-Nspire application software

Run it 1000 times and track RAM Available


11/46


No Obvious Memory Leaks in Tools


12/46


TI Nspire Add each app 5 times, each in a new doc


13/46


Detecting Large Memory Leaks with RRR


14/46


RRR Test Set for CAS Device 1.2 Development Cycle

About 100 automated tests available

Mixture of BAT tests and L&S regression tests

Ran all 100 as an RRR run, recorded RAM Available


15/46



16/46


What happens in BAT_032?

The BAT_032 test does a number of things that could becausing a memory leak:

Puts 3 apps in a single-page layout

Runs Graphs & Geometry (G&G)

Runs Calculator

Runs Lists & Spreadsheets (L&S)

Defines functions in G&G & Calculator

Puts defined functions into a Function Table in L&S

To determine specific memory leak source:

Get & report RAM Available after each step in this test

Run the test a few times and plot the RAM Available


17/46


Each steep slope isfrom the sameFunction Table step

Same data with lossfrom Function Tablesteps eliminated.


18/46


What happens in Function Table?

Function Table appears to be the source of the memory leak

What part of Function Table?

Entering/Leaving this feature?

Putting functions into a Function Table?

Write automated exploratory tests to isolate these features

One to just enter and leave Function Table feature

Another to create 2 simple functions and add them to aFunction Table

Run each test many times and plot the RAM Available & trend


19/46


RAM Available data isnoisy but has a flattrend over time.


20/46


When a Function Table isused, it allocates, anddoes not free, about 500K!This memory leak was

quickly found and fixed.


21/46




22/46



When a software error causes a TI-Nspire device to crash

device automatically re-boots itself

RAM Available is reset to the maximum allowed by

the available physical memory & software build constraints Test Automation engines re-establish communications

after the reboot

next automated test proceeds normally

Device crash and reboot can be identified on a graph ofRAM Availalble by a local maximum


23/46




24/46



Logs showed each device crash occurred during the same test

Used exploratory testing techniques to try various sub-setsof the test steps until we found a minimal set of steps that

caused a device crash These minimal steps were added to a bug report which

assisted the developers in finding and fixing this device-crashing bug


25/46


Measuring Small Systemic Memory Leaks with RRR


26/46


Measuring Small Systemic Memory Leaks with RRR

Tests showed that the TI-Nspire Lists & Spreadsheet (L&S)application had some memory leaks

Wanted to track these during the 1.4 development cycle

First establish a baseline metric for comparison

Ran RRR using a set of L&S-only tests on the TI-Nspire 1.3

Plotted graph of RAM Available and trend during that run

This became the baseline used for comparison during the1.4 development cycle


27/46


TI-Nspire 1.3 L&S Baseline Memory Leak Metrics


28/46


TI-Nspire 1.3 L&S Baseline Memory Leak Metrics

Two statements can be made about L&S memory usage from thebaseline 1.3 graph:

There is a small systemic memory leak of about 255 bytes per

test, as illustrated by the linear regression trend line Many of the tested L&S features use a large amount of memory,

as illustrated by the deep spikes in RAM Available

Used this baseline to visually compare the L&S memory leak metrics

throughout the 1.4 development cycle These results are shown on the next several slides


29/46


TI-Nspire 1.4 L&S Memory Leak Metrics

Bad News: Trendslope is steeper.

Good News: Spike depth(overall memory usage) issmaller.


30/46



Trend slope continuesto get worse as new

functionality is added.


31/46



Trend slope starts toimprove in build

1.4.8748.


32/46



Trend slope almostback to 1.3 baseline

by build 1.4.10784!


33/46


Detecting Performance Problems


34/46


Detecting Performance Problems

Test Scenario

Create a simple document and save it with the default name

System suggests Document1 the first time

If this is chosen, system suggests Document2 the next time

And so on

Successive executions of this test seemed to take longer

To measure this we used a different HVTA technique

Set up repeated executions of the same test

Measured and recorded time to execute each

Graphed successive execution times


35/46


Performance degradesexponentially!


36/46


Comparing Performance Metrics

Original Save File performance test was in TI-Nspire 1.2

Repeated the same repeated test execution in TI-Nspire 1.3

Comparison of graphs shows a slight deterioration

No root cause determined yet

Developers supect a problem in the underlying file system

Further work is needed to determine and fix the root cause


37/46



38/46


New Developments


39/46


Improved RAM Available Metric

Original technique

Capture and save RAM Available at the start of each test

i.e. with memory still allocated from the previous test

Resulted in spikey graphs since some actions uselarge amounts of memory

New technique

Capture and save RAM Available at the same relativepoint in each test

Each test starts with:

Delete any previous document

Create a new empty document

Capture and same RAM Available at this point If all allocated memory from the previous document is freed

then RAM Available in a new empty document should beconstant


40/46



41/46


Automated Result Graphing & Trend Slope Computation

Original manual process All graphs shown in the published paper and in this presentation

so far were produced manually with MS Excel

Automated tests produce CSV files of result data

CSV files read into Excel, graphs generated manually,trend lines fitted and slopes computed using Excel

Current automated process

An automated Data Driven Test (DDT) calls other tests

DDT spreadsheet or CSV file specifies Which tests to run

In what order to run them

How many times to run each

RAM Available data accumulated until the DDT ends

At the end of the DDT, graphs & tables of results automaticallygenerated using the Confluence Wiki tool

This technique is

NOT using RRR


42/46



43/46


Auto-generated Results from TI Wiki


44/46


Detailed Memory Leak Reports

Instrumented code in DEBUG builds

Most TI-Nspire code uses a MemAlloc wrapper for malloc

In a DEBUG build this wrapper keeps track of

Where MemAlloc was called (file name & line number) Sequential block number of allocated memory

DEBUG builds also provide a routine to write current memoryallocation data to the devices serial port

Added a TI-CAT Test Engine interface to these DEBUGmemory reporting routines

Modified the new DDT to use these DEBUG routines

Also uses pre-existing SerialPortIO Test Engine to read and

filter devices serial port output Automatically produces detailed memory leak reports to the Wiki


45/46



46/46


Questionsand / or

Discussion