Agile Testing for Large Embedded Systems

AGILE TESTING FOR LARGE EMBEDDED SYSTEMS

RICH DAMMKOEHLER

HT

TP

://S

D.K

EE

PC

AL

M-O

-MA

TIC

.CO

.UK

/I/K

EE

P-C

AL

M-I

TS

-JU

ST

-A-T

HE

SIS

.PN

GH

TT

P://M

ED

IA.T

UM

BL

R.C

OM

/99

B6

A6

61

8F

EC

0B

2B

21

E3

5E

FD

AD

6A

5E

51

/TU

MB

LR

_IN

LIN

E_

MO

IBO

6X

VC

R1

QZ

4R

GP.JP

G

HTTP : / / 2 3 AXY 6 K2 Q8 VAFA J U .Z I PPY K I D .NETDNA-CDN .COM/WP-CONT ENT /UPLOA DS /2 0 1 4 /0 6 /TES T I NG -DARTH-VADER. J PG

• ATDD

H TT P : / / A PL E A S U R E T OT E AC H. F I L E S . W OR D PR E S S . C O M/ 2013 / 06 / S T O PC O L L A BORAT E . J PG

• TDD

HTTP : / / W W W.SC ION .COM / SC IO NRAC I NG / W P-CONTENT / UPLOADS / 2 01 3 / 06 / F LOR IDA-F R IDAY-3 8 1 . J PG

• Integration

H TT P : / / A D M I N . U T E P. E D U / P O RTAL S / 1 87 8 / I N T E G RAT I O N % 2 0 T E S T I N G % 2 0 P I C . J P G

TEST ING THE AGGREGATE

• Bench Testing

HTTP : / / P ES W I K I . C OM / I M AG ES /3 /30 /M P I _T ES T_BEN C H_ 600 . J P G

BENCH TEST ING

• NRL/RL Testing

H TT P : / / ME D I A 4 . S - N BC N E W S .C OM/ I / N E W S C MS / 2 0 1 4 _ 1 2 / 2 5 7 9 8 6 / 1 4 0 3 1 7 -S I M UL AT OR- J H C - 1 3 4 5 _ 0 B AC D 5 0 4 D 6 B A 0 2 9 3 2 3 E A A B5 5 4 A 1 8 3 5 C 6 . J PG

H TT P : / / W WW. A L A S KA- I N - P I C T U R E S . C O M/ DATA / ME D I A / 20 / M I L I TA RY- J E T- C O C KP I T _ 4483 . J PG

SIMULATION & REAL WORLD TEST ING

• Load and Performance

H TT P : / / T OOLS .U T E ST.C OM / W P- C ON T E N T / U PLOA D S / S I T E S / 3 / 2 0 1 4 / 0 7 / W T 4 -LOA D -T E ST-A N D - R E C OR D -W E BS I T E -PE R F OR MA N C E - OV E R-T I ME . J PG

PERFORMANCE TEST ING

• Gotchas and Remedies

H TT P : / / I M G . W O N K E TT E . C OM / W P- C O N T E N T / U P LOA D S / 2 0 1 4 / 0 8 / S N AK E - O I L . J P G

• Hardware isn’t available/scarce/expensive

• Dual targeting

• Virtualization

• Bench Testing

• 5 Stage Dev Cycle

H TT P : / /WWW.OXFAMBLOGS .OR G/FP2 P /WP- CONT E NT /U PLOAD S /WAT E R- SCARC I T Y.BMP

GOTCHA : HARDWARE TARGET SCARCITY

• Dual Targeting

H TTP : / / I M G 1 . W I K I A . N OCOOK I E . N ET / _ _ CB 2 0 1 3 0 3 1 1 0 5 1 3 3 8 /STA RWA R S / I M AG ES /B /BC /T I E _ TA R G ET_ F F7 . PN G

F IX : DUAL TARGETING

• Virtualization

H TT P : / / B PL R E A D I NG S U G G E S T I O NS . F I L E S . W O R D PR E S S . C O M/ 2013 / 09 / V I RT U A L- R E A L I T Y. J PG

F IX : V IRTUALIZAT ION

• Compiler Differences

• Software Design Modularity

• Continuous Integration

• 5 Stage Dev Cycle

H TT P : / / M I C H A E L H YATT. C O M / WP-C O NT E NT / U P LOA DS / 2012 / 01 / I S T O C K _ 000003501504S M A L L . J P G

GOTCHA : COMPILER D IFFERENCES

• (Dual Targeting) Library Differences/Bugs

• Software Design Modularity

• TDD / Contract Testing

• Continuous Integration

• 5 Stage Dev Cycle

H TTP : / / I M G 1 . W I K I A . N OCOOK I E . N ET / _ _ CB 2 0 1 3 0 3 1 1 0 5 1 3 3 8 /STA RWA R S / I M AG ES /B /BC /T I E _ TA R G ET_ F F7 . PN G

F IX : DUAL TARGETING

• Software Design Modularity

HTTP : / / W W W. IBMSYSTEM SMAG.COM / G ETATTACHM ENT / 5 B3 EF 7 14 -F F E9 -4 6 9 C -8 7 CF -5 C9F 9 B1 F EA5 7 /

F IX : DESIGN MODULARITY

• Continuous Integration

H TT P : / / N E B U L A. W S I MG . C OM / E 6 D 2 E 4 7 7 1 5 8 6 1 3 8 D F D 9 1 E 1 1 7 2 0 3 4 F D C 6 ? AC C E SS KE Y I D = 5 3 9 0 C B 7 3 FA AC 4 5 B 7 A 2 3 C & D I SPOS I T I ON = 0 & A LLOW OR I G I N = 1

F IX : CONTINUOUS INTEGRATION

• 5 stage test cycle

HTTP : / / 2 .BP.BLOG S POT.COM/ -EK4 MHZALRZE /UHKM7 I 3 YPT I /A A A A A A A A NKO/KANTTQOM4 UM/S 1 6 0 0 /M I CRO_CYCLE .PNG

EMBEDDED TDD CYCLE

• Library’s have bugs

H TTP : / /W W W. M YC L E A N I N G P R O D U C TS . C O M /B LO G /W P-C O N TE N T / U P LOA D S /2 013 /08 /B E D -B U G S - I N -A-BO O K 2 . J P G

GOTCHA : L IBRARIES HAVE BUGS

• Software Design Modularity

HTTP : / / W W W. IBMSYSTEM SMAG.COM / G ETATTACHM ENT / 5 B3 EF 7 14 -F F E9 -4 6 9 C -8 7 CF -5 C9F 9 B1 F EA5 7 /

F IX : DESIGN MODULARITY

• Continuous Integration

H TT P : / / N E B U L A. W S I MG . C OM / E 6 D 2 E 4 7 7 1 5 8 6 1 3 8 D F D 9 1 E 1 1 7 2 0 3 4 F D C 6 ? AC C E SS KE Y I D = 5 3 9 0 C B 7 3 FA AC 4 5 B 7 A 2 3 C & D I SPOS I T I ON = 0 & A LLOW OR I G I N = 1

F IX : CONTINUOUS INTEGRATION

HTTP S : / / I G OR. I O / I M G /G I T-BRA N C H I N G /V ER S I ON . P N G

F IX : SEMANTIC VERSIONING

• 5 stage test cycle

HTTP : / / 2 .BP.BLOG S POT.COM/ -EK4 MHZALRZE /UHKM7 I 3 YPT I /A A A A A A A A NKO/KANTTQOM4 UM/S 1 6 0 0 /M I CRO_CYCLE .PNG

F IX : EMBEDDED TDD CYCLE

• Memory/Performance Constraints on Target

• Software Design Modularity

• Better Emulation

• Use budgets

• 5 Stage Dev Cycle

HTTP : / /WEB .S TA NFORD.EDU/GROUP /S TVP /CG I -B I N /BLOG/WP- CONTENT /UPLOADS /2 0 1 1 /0 2 /CONS TRA I NT S -4 6 0 X2 5 0 . J PG

GOTCHA : PLATFORM CONSTRAINTS

• Software Design Modularity

HTTP : / / W W W. IBMSYSTEM SMAG.COM / G ETATTACHM ENT / 5 B3 EF 7 14 -F F E9 -4 6 9 C -8 7 CF -5 C9F 9 B1 F EA5 7 /

F IX : DESIGN MODULARITY

• Better Emulation

H TT P : / / A S S E T S . V I C E . C O M/ C O NT E NT- I MAG E S / C O NT E N T I MAG E / 151824 / MAT R I XC O D E . J PG

F IX : CONSTRAINT EMULATION

• Budgeting

H TT P : / / W W W. R E N A I S S AN C E S O FT WAR E . N E T / F I L E S / I M G / F L AS H U S AG E G RA P H. J P G

F IX : BUDGETS

• Header Incompatibilities

• Software Design Modularity

• Continuous Integration

• TDD

HTTPS : / /WWW.MODMY P I .COM/ I MAG E /DATA /TUT OR I ALS /MODEL-B -PLUS -COMPAT I B I L I TY /RA S PBERRY-P I -COMPAT I BLE - 5 . J PG

GOTCHA : HEADER INCOMPATIB IL IT IES

• Software Design Modularity

HTTP : / / W W W. IBMSYSTEM SMAG.COM / G ETATTACHM ENT / 5 B3 EF 7 14 -F F E9 -4 6 9 C -8 7 CF -5 C9F 9 B1 F EA5 7 /

F IX : DESIGN MODULARITY

• Continuous Integration

H TT P : / / N E B U L A. W S I MG . C OM / E 6 D 2 E 4 7 7 1 5 8 6 1 3 8 D F D 9 1 E 1 1 7 2 0 3 4 F D C 6 ? AC C E SS KE Y I D = 5 3 9 0 C B 7 3 FA AC 4 5 B 7 A 2 3 C & D I SPOS I T I ON = 0 & A LLOW OR I G I N = 1

F IX : CONTINUOUS INTEGRATION

• Long Build Times

• Extreme Automation

• Software Design Modularity

• Continuous Integration

• Semantic Versioning

• 5 Stage Dev Cycle

H TT P : / / SK I NNY GL I TT E R.COM/WP- CONT E NT /U PLOAD S /2 0 1 3 /0 6 /BOR E D -WOMAN- 2 . J PG

GOTCHA : LONG BUILDS

• Extreme Automation

H TTP : / /UPLOA D . W I K I M E D I A . OR G /W I K I PE D I A /COM M ON S/8 /8 A /A UTOM AT I ON _ OF_ F OUN DRY_ W I TH _ R OBOT. J PG

F IX : HEAVY AUTOMATION

• Software Design Modularity

HTTP : / / W W W. IBMSYSTEM SMAG.COM / G ETATTACHM ENT / 5 B3 EF 7 14 -F F E9 -4 6 9 C -8 7 CF -5 C9F 9 B1 F EA5 7 /

F IX : DESIGN MODULARITY

• Continuous Integration

H TT P : / / N E B U L A. W S I MG . C OM / E 6 D 2 E 4 7 7 1 5 8 6 1 3 8 D F D 9 1 E 1 1 7 2 0 3 4 F D C 6 ? AC C E SS KE Y I D = 5 3 9 0 C B 7 3 FA AC 4 5 B 7 A 2 3 C & D I SPOS I T I ON = 0 & A LLOW OR I G I N = 1

F IX : CONTINUOUS INTEGRATION

HTTP S : / / I G OR. I O / I M G /G I T-BRA N C H I N G /V ER S I ON . P N G

F IX : SEMANTIC VERSIONING

• How do I apply it?

• 5 Cycle Approach (Embedded TDD Cycle)

HTTP : / / 2 .BP.BLOG S POT.COM/ -EK4 MHZALRZE /UHKM7 I 3 YPT I /A A A A A A A A NKO/KANTTQOM4 UM/S 1 6 0 0 /M I CRO_CYCLE .PNG

F IX : EMBEDDED TDD CYCLE

• In support of Testing

• Configuration Management

• Source Control Management

• Collective Ownership

H TTP : / / S1 . CDN . A UTOE VOLUT I ON . COM / I M AG E S /N EW S/H OW- P I T-CR EW S-W OR K- I N - M OTOR SPORT-1 4 2 2 5 _ 2 . J PG

SUPPORTING ACT IV IT IES

• Configuration Management

H TTP : / /W W W. TR OL L . M E / I M AG ES /TH E -BOBS /B UT-DO -W E- H AV E - EN OUG H -CON F I G URAT I ON -M A N AG E M E N T. J PG

• Source Control Management

H TTP : / /ATOM A N T I C . G I TH UB . I O /SL I DES -V ER S I ON CON TR OL / I M G /M EM E-V ER S I ON CON TR OL A L LTH I N G S . J PG

• Collective Ownership

H TT P : / /D I YCONFE SS I ONS .COM/WP-CONT E NT /U PLOADS /2 0 1 3 /0 5 /NOT-SH AR I NG . J PG

COLLECT IVE CODE OWNERSHIP

THE F INAL PRODUCT

http://fc05.deviantart.net/fs71/i/2014/055/5/f/cyborg_with_human_camo_by_jessedr1-d77td5g.jpg

Rich Dammkoehler, Pillar [email protected]

HT

TP

://S

D.K

EE

PC

AL

M-O

-MA

TIC

.CO

.UK

/I/K

EE

P-C

AL

M-A

ND

-AS

K-M

E-Q

UE

ST

ION

S.P

NG