Top Banner
MI AM Tutorial 4/29/13 8:30AM How to Break Software: Embedded Edition Presented by: Jon Hagar Independent Test Consultant Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] www.sqe.com
37

How to Break Software: Embedded Edition

Jan 15, 2015

Download

Technology

In the tradition of James Whittaker’s book series How to Break … Software, Jon Hagar applies the testing “attack” concept to the domain of embedded software systems. Jon defines the sub-domain of embedded software and examines the issues of product failure caused by defects in that software. Next, Jon shares a set of attacks against embedded software based on common modes of failure that testers can direct against their own software. For specific attacks, Jon explains when and how to conduct the attack, as well as why the attack works to find bugs. In addition to learning these testing skills, practice the attacks on a device—a robot that Jon will bring to the tutorial—containing embedded software. Specific attack methods considered include data issues, computation and control structures, hardware-software interfaces, and communications.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: How to Break Software: Embedded Edition

MI AM Tutorial

4/29/13 8:30AM

How to Break Software: Embedded

Edition

Presented by:

Jon Hagar

Independent Test Consultant

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Page 2: How to Break Software: Embedded Edition

Jon Hagar

Jon Hagar is a systems-software engineer and tester consultant supporting software product integrity and verification and validation, with a specialization in embedded and mobile software systems. For more than thirty years Jon has worked in software engineering, particularly testing, supporting projects including control system (avionics and auto), spacecraft, mobile-smart devices, IT, and attack testing of smart phones. Jon has built and managed embedded test labs with test automation; publishes and speaks regularly with more than fifty presentations and papers; and authored a new book on mobile/embedded software, scheduled for publication in 2013, and parts of three other books.

Page 3: How to Break Software: Embedded Edition

4/11/2013

1

Jon Hagar Copy right 2010 How to Attack Embedded Software 1

How to Break Software:

The Embedded Edition

Jon Hagar

[email protected]

[email protected]

Jon Hagar Copyright 2010 How to Attack Embedded Software 2

Some Thoughts

• Are you in the right class?

– Definitions and introductions will help here

• Why do we test and what do we need to test?

– Risks

– Information

– Users

– Attacks

Page 4: How to Break Software: Embedded Edition

4/11/2013

2

Jon Hagar Copyright 2010 How to Attack Embedded Software 3

This is a Workshop Tutorial

• A bit of a talking head (with charts)

– Based on my book

• Attendees should be prepared to:

– Do some reading & thinking

– Use the reference material

– Talk & ask questions

– Share (lessons learned and retrospectives)

• LET’S PLAY TEST… and try out some new

things

Jon Hagar Copyright 2010 How to Attack Embedded Software 4

Agenda

• Definitions and introductions

• Risk based concepts

• Exploratory approaches

• Attacking the scenario(s)

• Attacking the hardware-software interface

• Wrap up and references

• Exercises and testing

Page 5: How to Break Software: Embedded Edition

4/11/2013

3

Jon Hagar Copyright 2010 How to Attack Embedded Software 5

DefinitionsEmbedded Software Systems . . .

• Interact with unique hardware/systems to solve specialized problems in the “real world”

– IT software runs with largely “generic” hardware

– Users are barely aware the device uses or has software

• Usually have significant hardware interface issues and concerns

– Initialization, noise, power up/down, timers, sensors, etc.

• Often are resource constrained

– RAM, ROM, stack, power, speed, time, etc.

• Typically has a restricted or no Human User/Computer Interface (HCI) but is evolving rapidly

• Often no way (or only a risky way) to update and/or change the software

• Involves risks, hazards, safety, and/or some specialized domain knowledge and logic/algorithms usually controlling hardware

Jon Hagar Copyright 2010 How to Attack Embedded Software 6

Close Cousins:

Mobile, Smart, and Handheld• As the names imply, these are devices—small, held in the hand, often

connected to communication networks, including

– Cell and smart phones – apps (not covered today)

– Tablets

– Medical devices

• Typically have:

– Many of the problems of classic “embedded” systems

– The power of PCs/IT

– More user interface (UI) than classic embedded systems

– Fast updates

• Are getting more power, memory, and features (software, e.g., apps)

• The “hot” area of computers/software

– Testing rules are “evolving”

Page 6: How to Break Software: Embedded Edition

4/11/2013

4

Jon Hagar Copyright 2010 How to Attack Embedded Software 7

What do these look like?Examples

– Avionics systems: planes, cars, rockets, military,…..

– Telecom: switch, routers, phones, cell devices,….

– Transportation: traffic control, railroad, trucking, ….

– Industrial control: lighting, machines, HVAC, nuclear/power,…

– Medical: pacemaker, dispensers, …….

– Home and office systems: control, entertainment (TV box), …

– And the list goes on

• Now starting to include PDA’s and other items that “blur” the lines

Jon Hagar Copyright 2010 How to Attack Embedded Software 8

Fundamental Software Capabilities

Dr. James Whittaker lists four capabilities:• Software accepts inputs from its environment

• Software produces output and transmits it to its environment

• Software stores data internally in one or more data structures

• Software performs computations using input or stored data

Embedded devices can be refined with1. Function in/with Time

2. Use/control of unique hardware, OR

3. Refinement of items 1 and 2

Page 7: How to Break Software: Embedded Edition

4/11/2013

5

Jon Hagar Copyright 2010 How to Attack Embedded Software 9

• Handheld/Embedded software has

similar defects to other software

• Requirements & Design

• Logic & Math

• Control Flow

• Data

• Initialization & Mode changes

• Interfaces

• Security

• Gaming

• etc. . .

Embedded: Knowing the Bug (error)

But adds context defects/issues

• Software and hardware

development cycles done in

parallel, where aspects of the

hardware may be unknown to the

software development effort

• Hardware problems which are

often fixed with software late in

the project

• Small amounts of dense complex

functions often in the control

theory or safety/hazard domains

• (a big one) Very tight real-time

performance issues (often in milli-

or micro-second ranges)

Jon Hagar Copyright 2010 How to Attack Embedded Software 10

The “World” of Mobile-Smart/Embedded Software

• Inputs and outputs involve hardware, software, and humans

• Time dependent

– NOTE: most software has “time” (performance) issues but here things are often “hard real time”

– Embedded and real-time “time” may be a requirement

Software

Stimulus-Inputs Response-Outputs

Expected

Unexpected

Wanted

UnwantedHardware

Page 8: How to Break Software: Embedded Edition

4/11/2013

6

Jon Hagar Copyright 2010 How to Attack Embedded Software 11

Exercise: Why do we test?

• Handheld Mobile/Embedded Software

Jon Hagar Copyright 2010 How to Attack Embedded Software 12

YAM* Lifecycle Embedded (*yet another model)

• Software - Many builds, iterations and increments

– Test “circles” around schedule milestones

…………

…Prototype………

………Prototype n…..

But what about the hardware lifecycle?

start Lab drop end

Build 1

start Eng drop lab drop end

Build 2

Page 9: How to Break Software: Embedded Edition

4/11/2013

7

Jon Hagar Copyright 2010 How to Attack Embedded Software 13

Hardware Build

Hardware Build

Example High Level Embedded Lifecycle

System Creation

Hardware Build

Software Build

Software Build

Software Build

Software Build

Hw

Issue

Results: Software is “late”

Software Build

Jon Hagar Copyright 2010 How to Attack Embedded Software 14

My Assumptions…

• This is not a “general” class on systems, software, and/or testing and I assume the following knowledge:– Test plans and planning

– Requirements testing

– Test labs and building labs

– Standards you operate under (yes, there are many)

– Tools you use

– Testing experience (a software system)

– Embedded design for testability is an accepted practice

• That you want something more . . .

Page 10: How to Break Software: Embedded Edition

4/11/2013

8

Jon Hagar Copyright 2010 How to Attack Embedded Software 15

If What I Assume is False(when you get home)

• Reference list is available to do some reading

• Other full classes are available

• You are reading books

• You will ask questions

• Looking to have an epiphany

• You are ready to learn

Keep in mind that I I I I do not have all the answersdo not have all the answersdo not have all the answersdo not have all the answers

Jon Hagar Copy right 2010 How to Attack Embedded Software 16

Section 1: Testing Preliminary

Page 11: How to Break Software: Embedded Edition

4/11/2013

9

Jon Hagar Copyright 2010 How to Attack Embedded Software 17

Exercise: Test the Embedded Game

• Break into teams

• Define a test

• Define some rules: No destructive testing please

• List of requirements– This is a handheld game

– You think of something (say spinach) and it figures out what you are thinking by proposing

20 questions to you

– Questions begin with animal, vegetable, mineral and go from there

– Game has non-standard input keys, display screen, and embedded software

– Game knows things and will figure out what you are thinking of

• Now . . . build a test for this device

Jon Hagar Copyright 2010 How to Attack Embedded Software 18

What do you mean you cannot test?

• What is wrong?

• What do you need to do testing?

• Is this not the world many testers live

in?

• We should start simple in testing, but

maybe this is not simple enough?

Page 12: How to Break Software: Embedded Edition

4/11/2013

10

Jon Hagar Copyright 2010 How to Attack Embedded Software 19

So, Let’s Back up a Little

• Let me give you some attack support concepts

& techniques (in case you don’t know these)

• You can apply these if you are a staff tester or

a “crowd source” contractor

• This is a simulation, but in the real world,

often you will just be given the software or a

device to test ---- You CAN test…….

Jon Hagar Copyright 2010 How to Attack Embedded Software 20

Risk and Exploratory-Attack Testing

• You cannot test everything

• Risk(s) based testing helps bound the test

scope problem

• Testing is about providing information and

understanding

• Exploration gets you started with whatever

you have (or don’t have)

Page 13: How to Break Software: Embedded Edition

4/11/2013

11

Jon Hagar Copyright 2010 How to Attack Embedded Software 21

Risk-Based Testing• Address, mitigate, attack and retire product risks

• Do you remember what a risk is?– Potential problem - Consequence and effect

– Occurrence – likelihood or chance of happening

– Impact – what happens

• Do this from the beginning (proposal) to the end (retirement) of the product (Hw-Sw) lifecycle

• Risks should feed the Attacks (more on that later)

Jon Hagar Copyright 2010 How to Attack Embedded Software 22

Sample Product Risks Testers

Should Consider

• Safety

• Security

• Hazard

• Business impacts

• Control (loss of)

• Computation

• Functional elements

• Non-functional

• Data

• Regulation (s) and legal factors

• Output noise

• Environment and input factors

• System factors – complexity, interfaces,

human/non-human

?

Page 14: How to Break Software: Embedded Edition

4/11/2013

12

Jon Hagar Copyright 2010 How to Attack Embedded Software 23

How to Use Risk Analysis in Testing

• Goal oriented testing (where to focus)

• Priority of attack and scenario– Never enough time to test everything

– Can define the “un attacked” (risks)

• Minimization of risks by focusing on the scary/critical first

• Provide information back to the team sooner

Jon Hagar Copyright 2010 How to Attack Embedded Software 24

Risk-Based Testing Process (simple)

• Identify the product

• Find product supporting information

• Identify risks associated with the product

• Risk priority (what you will test first?)

• The resulting risks by priority define the attacks

Page 15: How to Break Software: Embedded Edition

4/11/2013

13

Jon Hagar Copyright 2010 How to Attack Embedded Software 25

Risk Analysis Throughout the Test Process

• Many testers just think “requirements” in embedded, but…

• Always be “thinking” risks, since it can drive and control your

testing

• Do this by team brainstorming (make lists)

• Tests and analysis provide learning/data points/information

– Errors in an area of code?

– Hardware that doesn’t work?

– Piece of code from a vendor is more complex?

– Operations the system will/can do?

• Particularly off nominal and unusual (where bugs hide)

Jon Hagar Copyright 2010 How to Attack Embedded Software 26

Exercise: Redeaux

• Back into your teams

• Conduct a risk exercise for your device

Risk Statement ( If x, then y happens) Priority

Page 16: How to Break Software: Embedded Edition

4/11/2013

14

Jon Hagar Copy right 2010 How to Attack Embedded Software 27

Risks Should Define Exploration

For mobile-embedded,

exploratory testing can be important

Jon Hagar Copyright 2010 How to Attack Embedded Software 28

Exploratory –Attack Testing

• What is it? – Scientific “methods”

– Engineering understanding

– May call it something else, but most of us do it

– Attacks “target” specific bugs using test techniques

• How and when to apply? – As early in a lifecycle as possible (with prototypes, models, etc.)

– When you want to “learn” and test at the same time

– When being a little “informal” is OK

– All the time?

Page 17: How to Break Software: Embedded Edition

4/11/2013

15

Jon Hagar Copyright 2010 How to Attack Embedded Software 29

Bach/Kaner:

“Exploratory testing is simultaneous learning,

test design and test execution.”

• Exploratory testing has rules and concepts

• Underlying it is a “model” of human understanding of

software and knowing how it fails

• NOT AD HOC: Ad hoc has all too often been associated with

sloppiness, carelessness, no documentation, non-repeatable,

and so froth—but may have a place at times too

Exploratory–Attack Testing Definition

Jon Hagar Copyright 2010 How to Attack Embedded Software 30

• Exploratory testing is situational - Use it when…

• Rapid feedback

• Learning

• Upfront rapid learning

• Attacking

• Risk

• Independent assessment

• Target a defect

• Prototyping

• Need info

• Test beyond the requirements

In Embedded

Page 18: How to Break Software: Embedded Edition

4/11/2013

16

Jon Hagar Copyright 2010 How to Attack Embedded Software 31

Many authors define it as:

1. Time/Schedule (limited)

2. The Tester (your team)

3. A Testing Mission (also called “Charter”)

4. Results• Usually in the form of opened Defects

• Sometimes an annotated Mission statement

and opened Defects list

• Maybe a “report”

• Retrospective (more on that in a minute)

General Concept of Exploratory

Jon Hagar Copyright 2010 How to Attack Embedded Software 32

� Test Design

� Critical Thinking

� Diverse Ideas

� Rich Resources

� Careful Observation

Exploratory Critical Components

Page 19: How to Break Software: Embedded Edition

4/11/2013

17

Jon Hagar Copyright 2010 How to Attack Embedded Software 33

Process for This Class (one of many)

• Have an outline (top level plan and/or risk list)

• Create a flip chart, notecard, state model, or some

representation of each test task

– No “heavy” weight documentation of the “test case”

– See Exploratory Charter (test objective)

• Have a Target concept - charter (Risk, Attack, Bug, Learning, …)

• Have a schedule/time box (hours — not more than 1-2 days)

• Do the test

– Design test

– Execute test

– Learn about the product: change the risk list, modify/add

tests, and so on

• Repeat the process as needed

Jon Hagar Copyright 2010 How to Attack Embedded Software 34

Exploratory Test Card (Charter)

Name of Test: Who is testing (test team)

• What to Test: Success Criteria:

– Risk (s): 1.

– Attack 2.

– Other (requirements, …..) 3.

• Support items needed:

• Role (User you play during the test): Others Steps

• Actions:

– 1.

– 2.

– 3.

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)

Document all of these.

Page 20: How to Break Software: Embedded Edition

4/11/2013

18

Jon Hagar Copyright 2010 How to Attack Embedded Software 35

Exercise: So Let’s Go Back to

the Embedded Game Device to Do Testing

• Test of the Game App

• Use the risks (chart 25) for the device to define test objectives

• Apply Exploratory-Attack

– Do a Charter

• Learn – do one cycle of exploration

Jon Hagar Copyright 2010 How to Attack Embedded Software 36

Group Flip Chart

Feedback - Retrospective

• What did you accomplish?

– Find any bugs? If so, how many?

• What did you think of?

• What would you do differently?

Page 21: How to Break Software: Embedded Edition

4/11/2013

19

Jon Hagar Copy right 2010 How to Attack Embedded Software 37

Section 2:

So we have Risk Analysis &

We have done a first Exploratory Test

• Basic and addressed in many books

• What’s next?

• Lets get to a real embedded device

Jon Hagar Copyright 2010 How to Attack Embedded Software 38

What is an Attack

• Attacking your software–In part, the process of attempting to demonstrate that a system (hardware, software, and operations) does not meet requirements, functional and

non-functional objectives

– Embedded/handheld software testing must include “the system” (hardware, software, operations, users, etc.)

• Attacks go after common modes of failure and bugs to demonstrate that “does not meet” exists

• We go after our enemy with many approaches– Tools

– Levels

– Attacks

– Techniques

– Etc.

Page 22: How to Break Software: Embedded Edition

4/11/2013

20

Jon Hagar Copyright 2010 How to Attack Embedded Software 39

An Attack Is…

• Based on a common mode of failure seen over and over– Maybe seen as a negative, when it is really a positive

– Goes after the “bugs” that may be in the software

– Based on or using classic test techniques and test concepts• Lee Copeland’s book on test design

• Many other good books

• A Pattern (more than a process) which must be modified for the context at hand to do the testing

• Testers learn these in a domain after years and form a mental model (most good testers attack)

Jon Hagar Copyright 2010 How to Attack Embedded Software 40

Kinds of Attacks

• Whittaker offers a good starting point for software

attacks in general that can be applied to embedded:

– User Interface Attacks

– Data and Computation

– File System Interface

– Software/OS Interface

• Whittaker’s “How to Break Software” lists 23 attacks

• “Software Test Attacks to Break Mobile and

Embedded Devices” lists 32 attacks and 8 sub attacks

Page 23: How to Break Software: Embedded Edition

4/11/2013

21

Jon Hagar Copyright 2010 How to Attack Embedded Software 41

Embedded Attack Classification

• Developer Attacks (unit/code testing)

• Control System Attacks

• Hardware-Software Attacks

• Mobile and Embedded Software Domain Attacks

• Time Attacks (Performance)

• Human User Interface Attacks

• Smart and/or Mobile Phone Functional App Attacks

• Mobile/Embedded Security Attacks

• Generic Attacks

– Functional, mind mapping, and combinatorial tests

Jon Hagar Copyright 2010 How to Attack Embedded Software 42

The Software You Test

• Do you know how it fails?

• Do you test for success or failure?

– Both?

• Will this workshop give you all the answers and

all possible attacks?

– No, but you can start asking questions and thinking

Page 24: How to Break Software: Embedded Edition

4/11/2013

22

Jon Hagar Copyright 2010 How to Attack Embedded Software 43

Introducing the Robots

• Requirements – in the class hand out for each robot grouping

• Rules (this exercise takes some thinking and reading)

– NO destructive testing (Please BE CAREFUL with the robots)

– There are bugs to be found (record and report them)

– Each group defines an attack and gets “time” on the devices (but time in our

environment is limited—just as it is in the real world)

• Environment

– This room, but we have some “test tools”

– This is software testing, but within the hardware it is embedded in

• This will be a simple testing process, but use the attack concepts and report

experiences back in the debrief

1. Define risks based on hardware, software, requirements, and bugs (risk list)

2. Conduct each attack session using a charter

3. Define a test attack using provided attack pattern (handout)

4. Will do a debrief after each test session

5. Group will rotate different robot configurations

Jon Hagar Copyright 2010 How to Attack Embedded Software 44

Additions Considerations

• We will try to get at least 2 different attacks

• Use the concepts we used on games

• Ask questions (of each other & me)

• No destructive testing and I am the “tester”

(you must tell me what to do)

• Use the tools at hand

• But first we need to think about embedded

users – next exercise

Page 25: How to Break Software: Embedded Edition

4/11/2013

23

Jon Hagar Copyright 2010 How to Attack Embedded Software 45

More Test Time

• We are going to do another attack on a different embedded

device

• Each group will be tasked with one of the two attacks

• Each group should complete a charter and see the “test

master” for access to the device

• You’ll have a robot with software loaded

• Practice identifying: Risks, Users, Exploration, and Attacks

• Follow the “suggested patterns” of the attacks

• We’ll go until no more time left

Jon Hagar Copyright 2010 How to Attack Embedded Software 46

Understanding Users for

Embedded Attacks

• Let’s list some of my Game App users because

• Users play into risks, attacks, bugs, what to look for

• You should be able to do the same for the robots (or any software you test)

Page 26: How to Break Software: Embedded Edition

4/11/2013

24

Jon Hagar Copy right 2010 How to Attack Embedded Software 47

So you have a few basics:

Risk thinking

Exploration

Software’s users

Attack patterns are provided next

Jon Hagar Copyright 2010 How to Attack Embedded Software 48

Attack Group 1

Stories, Tours, and Scenarios

• Call them what you will

• There are subtle differences depending on

whose material you have read

• They are how the system gets used end-to-end

• They combine use, users, information,

techniques, tools, and (maybe) attacks

Page 27: How to Break Software: Embedded Edition

4/11/2013

25

Jon Hagar Copyright 2010 How to Attack Embedded Software 49

Apply This Attack When…

• Time interacts with the software, events, inputs, and

outputs

• Checklist of things to look for & consider (possible

bugs) – Order problems

– Too Long

– Too Fast

– Not at Right Time mark or point

– Late

– Late or early

– Early

– Deadlocked caused by a race condition(hard to find)

– Extra input or output events

– Missing events

– Wrong input/output within events

Jon Hagar Copyright 2010 How to Attack Embedded Software 50

Attack Factors

• What - Look for things not in the right order

• Who – Test team

• Where – Lab and/or field testing where hardware

and software interact

– Tools may be important here

Page 28: How to Break Software: Embedded Edition

4/11/2013

26

Jon Hagar Copyright 2010 How to Attack Embedded Software 51

How?• Understand what the system does or is supposed do

– a sequence of events or functions

– Look in: concepts of operations, user guides, use cases, models, & any other

information that will detail functions and usage over time

– From these, organize a sequence or set of sequences

• First attack case: Focus on a typical situation based on requirements and/or use cases

• Second attack case: Consider the off–normal , non–failure modes

– Look for the failure modes and effects—does the software recover well?

– Review and understand system errors and failure history from the field.

• Build up histories of attacks based on outputs and log files

– Warning: log files can contain large amounts of detailed data and this can also

adversely affect the performance (especially timing) of the software

• Conduct risk analysis as the effort progresses

• Final Attack cases: Build Extreme cases

– such as “Soap Opera” Tests

• Warning: Watch becoming “script” bound

Jon Hagar Copyright 2010 How to Attack Embedded Software 52

Attack Group 2

Refer to second attack handout

Here we are attacking the hardware-to-software

interface

Page 29: How to Break Software: Embedded Edition

4/11/2013

27

Jon Hagar Copyright 2010 How to Attack Embedded Software 53

Attack: Analog-Digital Hw-Sw Interfaces

• When - The software is “controlling” the unique hardware

• What – Look at the interface, hardware (as a user), and what

the software is controlling

• Who – Test team (independent)

• Where – Lab where the hardware and software are both

present

• Bugs to look for (next page)

Jon Hagar Copyright 2010 How to Attack Embedded Software 54

Taxonomy: A2D and D2A Bug Possibilities

Type Situation Impact Notes

A2D A2D representation information is lost

because measurement is not precise

Software

computation is

based on incorrect

data

Number of bits used to store the analog

converted data is not large enough or

sampling rate to get bits is not correct.

A2D A2D information is contaminated with

noise

Software

computation use

noise when it

should not

The noise term may not be known,

accounted for, or misrepresented.

A2D A2D information is calculated

correctly

Computation has

unknown error

Sources of error can come from:

calibrations used on variables, variables

lacking initialization, or calculations are not

done with enough accuracy (single versus

double floating point

D2A D2A conversion losses “least

significant bits” (LSB) in conversions,

but bits are, in fact, important

because computer word sizes are too

small

Output to analog

device is wrong

Number of bits stored from the digital

world to the analog world do not have

enough precision, so analog data is

incorrect.

D2A D2A information does not account for

noise of the real world

Software

computation does

not include a factor

for noise

The analog values are not correct given the

noise of the real world (output data may be

lost in the noise).

D2A D2A information is calculated

correctly because of internal factors

Computation has

unknown error

Sources of error can come from:

calibrations used on variables, variables

lacking initialization, or calculations are not

done with enough accuracy (single versus

double floating point

Page 30: How to Break Software: Embedded Edition

4/11/2013

28

Jon Hagar Copyright 2010 How to Attack Embedded Software 55

How?• Upfront data gathering and analysis are important beginnings – what do we know

(or can ask about)

• Identify input devices

• Identify output devices

• Define the input disturbances (unexpected system inputs)

• Define possible output disturbances (unexpected system outputs)

• Determine what is or is not possible in the test environment

• Conduct a risk analysis (see likely bugs table)

• Identify the users of the device and software

- Testers should be aware that embedded systems have resource constraints in

memory, CPU usage, and time

• Use the above information to define an exploratory chart attack

• Go run that attack

• Learn

• Design

• Repeat (until time out)

Jon Hagar Copyright 2010 How to Attack Embedded Software 56

Questions to Ask with This Attack

• If the hardware is a prototype (not like what will be in the field), will that

impact testing or test results?

• If a simulation is used, what bugs might be missed because actual

hardware or software is not used?

• If the test inputs and environment are not representative of the real world

both in terms of expected and unexpected values, what risks will be

acceptable?

• If the hardware is not understood, will testing be weak?

• If the major sources of “noise” are not defined, will the system be

susceptible to impacts from unexpected inputs or outputs?

• All of these questions will involve test tradeoffs, accepted risk, and

compromise

– 40% of this kind of attack should be “normal” situations

– Start normal and move to off normal and stress cases

Page 31: How to Break Software: Embedded Edition

4/11/2013

29

Jon Hagar Copyright 2010 How to Attack Embedded Software 57

Group Flip Chart

Feedback – Retrospective Session

• What did you accomplish?

– Bugs?

– Tests?

• What things did you think of?

– Wish I had a ???? I need more time???

• What favors or opposes an attack?

• What would you do differently next cycle?

Jon Hagar Copy right 2010 How to Attack Embedded Software 58

Final ThoughtsFinal ThoughtsFinal ThoughtsFinal Thoughts

Page 32: How to Break Software: Embedded Edition

4/11/2013

30

Jon Hagar Copyright 2010 How to Attack Embedded Software 59

Wrap Up

• This tutorial only covered some basic

introduction (key attacks) and sampling

– There are many more

• Understanding your local context and error

patterns is important (one size does NOT fit all)

• Attacks are patterns…you still must THINK

• These attacks target Embedded and Mobile

Jon Hagar Copy right 2010 How to Attack Embedded Software 60

More Attacks (from my book and others)

• Attack 18: Bugs in Timing Interrupts and Priority Inversion

• Attack 19: Finding Time Related Bugs

• Attack 20: Time Related Scenarios, Stories and Tours

• Attack 21: Performance Testing Introduction

• Attack 22: Finding Supporting (User) Documentation Problems

• Sub–Attack 22.1: Confirming Install–ability

• Attack 23: Finding Missing or Wrong Alarms

• Attack 24: Finding Bugs in Help Files

• Attack 25: Finding Bugs in Apps

• Attack 26: Testing Mobile and Embedded Games

• Attack 27: Attacking App–Cloud Dependencies

• Attack 28 Penetration Attack Test

• Attack 28.1 Penetration Sub–Attacks: Authentication —

Password Attack

• Attack 28.2 Sub–Attack Fuzz Test

• Attack 29: Information Theft—Stealing Device Data

• Attack 29.1 Sub Attack –Identity Social Engineering

• Attack 30: Spoofing Attacks

• Attack 30.1 Location and/or User Profile Spoof Sub–Attack

• Attack 30.2 GPS Spoof Sub–Attack

• Attack 31: Attacking Viruses on the Run in Factories or PLCs

• Attack 32: Using Combinatorial Tests

• Attack 33: Attacking Functional Bugs

• Attack 1: Static Code Analysis

• Attack 2: Finding White–Box Data Computation Bugs

• Attack 3: White–Box Structural Logic Flow Coverage

• Attack 4: Finding Hardware–System Unhandled Uses in

Software

• Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs

• Attack 6: Long Duration Control Attack Runs

• Attack 7: Breaking Software Logic and/or Control Laws

• Attack 8: Forcing the Unusual Bug Cases

• Attack 9 Breaking Software with Hardware and System

Operations

• 9.1 Sub–Attack: Breaking Battery Power

• Attack 10: Finding Bugs in Hardware–Software

Communications

• Attack 11: Breaking Software Error Recovery

• Attack 12: Interface and Integration Testing

• 12.1 Sub–Attack: Configuration Integration Evaluation

• Attack 13: Finding Problems in Software–System Fault

Tolerance

• Attack 14: Breaking Digital Software Communications

• Attack 15: Finding Bugs in the Data

• Attack 16: Bugs in System–Software Computation

• Attack 17: Using Simulation and Stimulation to Drive

Software Attacks

Page 33: How to Break Software: Embedded Edition

4/11/2013

31

Jon Hagar Copyright 2010 How to Attack Embedded Software 61

Summary: Thank You (ideas used from)

• James Whittaker (attacks)

• Elisabeth Hendrickson (simulations)

• Lee Copeland (techniques)

• Brian Merrick (testing)

• James Bach (exploratory & tours)

• Cem Kaner (test thinking)

• Many teachers

• Generations past and future

• Books, references, etc.

Jon Hagar Copyright 2010 How to Attack Embedded Software 62

Book List (my favorites)

• “Software Test Attacks to Break Mobile and Embedded Devices” – Jon Hagar, to be published in 2013

• “How to Break Software” James Whittaker, 2003– And his other “How To Break…” books

• “Testing Embedded Software” Broeckman and Notenboom, 2003

• “A Practitioner’s Guide to Software Test Design” Copeland, 2004

• “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993

• “Computer Related Risks”, Neumann, 1995

• “Safeware: System Safety and Computers”, Leveson, 1995

• Honorable mentions:– “Embedded System and Software Validation” Roychoudhury, 2009

– “Systems Testing with an Attitude” Petschenik 2005

– “Software System Testing and Quality Assurance” Beizer, 1987

– “Testing Computer Software” Kaner et. al., 1988

– “Systematic Software Testing” Craig & Jaskiel, 2001

– “Managing the Testing Process” Black, 2002

Page 34: How to Break Software: Embedded Edition

4/11/2013

32

Jon Hagar Copyright 2010 How to Attack Embedded Software 63

More Resources

• www.stickyminds.com – Collection of test info

• www.embedded.com – info on attacks

• Association of Software Testing

– BBST Classes

http://www.testingeducation.org/BBST/

• Your favorite search engine

Jon Hagar Copyright 2010 How to Attack Embedded Software 64

Definitions (for this class)

• Taxonomy - the practice and science of classification.

• Test – the act of conducting experiments on something to determine the quality and provide information

• Test case – One set of inputs, environmental set up, and results (expected and un)

• Attack – to set up, forcefully, and attempt to “damage” the system or software, using tools, methods, and techniques

• Bug (error) – Results that depart from the expected (from requirements, design, standards, user, etc.)

• Lifecycle – From beginning-to-end, the steps, stages, and activities to create (birth-to-death)

• Procedure – a particular way of accomplishing tests, usually written (one or more test cases)

• Tour – a journey to find information (tests) with a focus/direction (story)

• Scenario – a sequence of events with a test plot or story

• Script – see procedure, but normally uses automation

• Users – someone/something that interacts with the system/software (can be human or machine, or?)

• Quality – Value to someone and that they will pay for

Page 35: How to Break Software: Embedded Edition

4/11/2013

33

Jon Hagar Copyright 2010 How to Attack Embedded Software 65

Exploratory Test Card (Charter)

Name of Test: Who is testing (test team)

• What to Test: Success Criteria:

– Risk (s): 1.

– Attack 2.

– Other (requirements, …..) 3.

• Support items needed:

• Role (User you play during the test): Others Steps

• Actions:

– 1.

– 2.

– 3.

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)

Jon Hagar Copyright 2010 How to Attack Embedded Software 66

Exploratory Test Card (Charter)

Name of Test: Who is testing (test team)

• What to Test: Success Criteria:

– Risk (s): 1.

– Attack 2.

– Other (requirements, …..) 3.

• Support items needed:

• Role (User you play during the test): Others Steps

• Actions:

– 1.

– 2.

– 3.

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)

Page 36: How to Break Software: Embedded Edition

4/11/2013

34

Jon Hagar Copyright 2010 How to Attack Embedded Software 67

Exploratory Test Card (Charter)

Name of Test: Who is testing (test team)

• What to Test: Success Criteria:

– Risk (s): 1.

– Attack 2.

– Other (requirements, …..) 3.

• Support items needed:

• Role (User you play during the test): Others Steps

• Actions:

– 1.

– 2.

– 3.

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)

Jon Hagar Copyright 2010 How to Attack Embedded Software 68

Exploratory Test Card (Charter)

Name of Test: Who is testing (test team)

• What to Test: Success Criteria:

– Risk (s): 1.

– Attack 2.

– Other (requirements, …..) 3.

• Support items needed:

• Role (User you play during the test): Others Steps

• Actions:

– 1.

– 2.

– 3.

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)

Page 37: How to Break Software: Embedded Edition

4/11/2013

35

Jon Hagar Copyright 2010 How to Attack Embedded Software 69

Exploratory Test Card (Charter)

Name of Test: Who is testing (test team)

• What to Test: Success Criteria:

– Risk (s): 1.

– Attack 2.

– Other (requirements, …..) 3.

• Support items needed:

• Role (User you play during the test): Others Steps

• Actions:

– 1.

– 2.

– 3.

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)