Emulation as Test Infrastructure Dan Voss, Varolii Corp. SASQAG May 15, 2008.

Emulation as Test Infrastructure

Dan Voss, Varolii Corp.

SASQAG

May 15, 2008

SASQAG 5/15/08 Dan Voss 2

Outline

Emulation past, present, and future How a software emulator works Emulation benefits Demonstration vs. Hardware Target Automated Test in Emulation Automated Test demonstration Q & A

SASQAG 5/15/08 Dan Voss 3

Emulation is the ability of one system to operate like another system.

Then came microprocessors.

(def.) Computers. Imitation (of a particular computer system) by using a software system, often including a microprogram or other computer that enables it to do the same work, run the same programs, etc., as the first. [1]

SASQAG 5/15/08 Dan Voss 4

Emulation Past

• Emulates hardware

1970s 2000s1980s 1990s

SASQAG 5/15/08 Dan Voss 5

An early hardware emulator [2]

SASQAG 5/15/08 Dan Voss 6

Emulation Past

• Emulates hardware

• Primitive but expensive

1970s 2000s1980s 1990s

SASQAG 5/15/08 Dan Voss 7

•

“brain” behind the emulator [3]

SASQAG 5/15/08 Dan Voss 8

Emulation Past

• Emulates hardware

• Primitive but expensive

• Few alternatives to chip makers’ systems

1970s 1980s 1990s 2000s

SASQAG 5/15/08 Dan Voss 9

Emulation Past

Problems mounted

• Faster speed vs. native mode emulation

• Primitive tools vs. larger code size

• Single-user systems vs. larger teams

• No role for Test

1970s 1980s 1990s 2000s

SASQAG 5/15/08 Dan Voss 10

Emulation Present

Breakthrough at 32-bit target system

• Lower cost tools on PC platform

• Standard desktop OS (Linux)

• Widely supported environment

1970s 1980s 1990s 2000s

SASQAG 5/15/08 Dan Voss 11

Emulation Present (cont’d)

Emulate software source files

• Tool environment is like the target

• Everybody can have one

1980s 1990s 2000s1970s

SASQAG 5/15/08 Dan Voss 12

Emulation Future

Emulate the system

• Niche – Windows apps on Mac

• Broad – system virtualization

1980s 1990s 2000s1970s

SASQAG 5/15/08 Dan Voss 13

software emulation details

if emulator buildemulator code for this operation

elsetarget code for this operation

endif

Sprinkled throughout the code!

SASQAG 5/15/08 Dan Voss 14

software emulator architecture

app code

app libraries

OS kernel drivers

OS kernel

OS libraries

devices

SASQAG 5/15/08 Dan Voss 15

Project Charter

• Contract on interface code design

• Aim for “perfect” emulation

• SDET commitment

SASQAG 5/15/08 Dan Voss 16

Perfect emulation benefits

All Everybody can have one

Team groups work independently

Test Immediate access to app code

Lower integration risk

Test is a full partner

SASQAG 5/15/08 Dan Voss 17

cell handset (8/04)

Emulator implemented:keypad, extra keysdisplaybrowsermessagingskin interfacebattery monitorRF module signaling

Missing: voice channelringtones

Curitel G400B

GSM 4 band

SASQAG 5/15/08 Dan Voss 18

WiFi Media Player (11/07)Emulator implemented:

keypad, extra keys

display

browser

WiFi connection

audio (limited formats )

video (limited formats)

battery monitor

Missing: Bluetooth, FM radioHaier “ibiza” player

30 GB hard drive

SASQAG 5/15/08 Dan Voss 19

Demo emulator vs. target

Play same audio on both

Play same video on both

Display same picture on both

SASQAG 5/15/08 Dan Voss 20

Emulation for automated test

• Measurement needs app state

• Examples key up/down WiFi connection

screen location audio playback

backlight video playback

memory used input buffer

• Keep it lightweight

SASQAG 5/15/08 Dan Voss 21

Varia Automated Test Artifacts

• Smoke test for continuous integration

• Functional test for stability

• Automated test cases (e.g., critical, tedious)

SASQAG 5/15/08 Dan Voss 22

Automated Test Demonstration

Stress test example

SASQAG 5/15/08 Dan Voss 23

Conclusions

• Emulation has come a long way

• Perfect emulation is a productive infrastructure to get a head start on challenging embedded applications

• Perfect emulation enables early testing

• Perfect emulation encourages automated test

SASQAG 5/15/08 Dan Voss 24

References

[1] Webster’s Encyclopedic Dictionary of the English Language, 1996

[2] Intel MDS in-circuit emulator photo

www.classiccmp.org/hp/mds-800/ice-80.jpg

[3] Intel MDS main unit photowww.google.com/imgres?imgurl=http://www.classiccmp.org/hp/mds-800/ice-80.jpg&imgrefurl=http://www.classiccmp.org/hp/mds-800/mds-800.htm&h=100&w=133&sz=60&tbnid=-6NLxJ5U9ocJ:&tbnh=100&tbnw=133&prev=/images%3Fq%3Dintel&sa=X&oi=image_result&resnum=1&ct=image&cd=2

SASQAG 5/15/08 Dan Voss 25

Q & A

SASQAG 5/15/08 Dan Voss 26

Emulation as test infrastructureDan Voss, Varolii Corp

SASQAG 5/15/2008

Abstract

Emulation tools are particularly suited to software for embedded applications. Early emulators were based on hardware, limited in resources, and proprietary.They were poor tools to develop code, and just about useless for testing.Processor performance has improved tremendously, however, and the situationis much better today. A prime reason is targets can now run a conventional desktopoperating system. I will describe several important benefits from this for Developmentas well as Test, and what it takes to make infrastructure based on emulation workableThere will be a demonstration comparing emulator operation with the real target for aportable entertainment product. Emulation encourages automated test, and anexample from the same project will be demonstrated.