Cmp Esd0312

E M B E D D E D S Y S T E M S D E S I G N

MARCH 2012

The Official Publication of The Embedded Systems Conferences and Embedded.com

VOLUME 25,NUMBER 2

Discriminated unions

9

Debug real-time SW with logic analyzer

19

Managing multipleprocesses in multicore

27

Probing pointers34

VERIFYINGREQUIREMENTSAND OBJECTCODE 12

SAFE RELIABLE

SECURE

For nearly 30 years the world’s leading industrial companies have trusted Green Hills Software’s secure and reliable high performance software for safety-critical applications.

From manufacturing automation, to telemetry and control systems, Green Hills Software has been delivering proven and secure underpinning technology.

To fi nd out how the world’s most secure and reliable operating system and software can take the risk out of your industrial project, visit www.ghs.com/s4i

Copyright © 2012 Green Hills Software. Green Hills Software and the Green Hills logo are registered trademarks of Green Hills Software. All other product names are trademarks of their respective holders.

TRUSTED SOFTWARE FOR INDUSTRIAL AUTOMATION AND CONTROL

Mouser delivers the components you need, on-time. And with local Technical Support and Customer Service Experts in 19 locations around the world, you’ll fi nd the newest components to launch your new design seamlessly.

The Newest Products for Your Newest Designs®

Mouser and Mouser Electronics are registered trademarks of Mouser Electronics, Inc. The Newest Products for Your Newest Designs is a registered trademark of Mouser Electronics, Inc.

mouser.com Semiconductors and electroniccomponents for design engineers.

Scan Here

mouser.com

Authorized Distributor

We deliverASAP, PDQand JIT . So,you’re neverSOL.

AS9120ACertified Distributor


THE OFF IC IAL PUBLICATION OF THE EMBEDDED SYSTEMS CONFERENCES AND EMBEDDED.COM

VOLUME 25, NUMBER 2MARCH 2012

COLUMNSprogrammingpointers 9Discriminated unionsBY DAN SAKS

Discriminated unions can be useful insome applications, but they also pro-vide insight into the advantages ofusing virtual functions.

break points 34Probing pointersBY JACK G. GANSSLE

The wrong probe can either cause yourcircuit to fail or even to physicallydestroy components. Here are some ofthe issues.

DEPARTMENTS#include 5Safety threats: from satellites topods in you pocketBY COLIN HOLLAND

DARPA's High-Assurance CyberMilitary Systems program is look-ing for a synthesizer capable ofproducing a machine-checkableproof that generated code satisfiesfunctional specifications as well assecurity and safety policies.

parity bit 7Open to interpretation

new products 33M2M and security

IN PERSONDesignWest/ESC Silicon Valley March 16–29, 2012http://esc.eetimes.com/siliconvalley/

ONLINEwww.embedded.com

12

EMBEDDED SYSTEMS DESIGN (ISSN 1558-2493) print; (ISSN 1558-2507 PDF-electronic) is published 10 times a year as follows: Jan/Feb, March, April, May, June,July/August, Sept., Oct., Nov., Dec. by the EE Times Group, 600 Harrison Street, 5th floor, San Francisco, CA 94107, (415) 947-6000. Please direct advertising and editorialinquiries to this address. SUBSCRIPTION RATE for the United States is $55 for 10 issues. Canadian/Mexican orders must be accompanied by payment in U.S. funds with addi-tional postage of $6 per year. All other foreign subscriptions must be prepaid in U.S. funds with additional postage of $15 per year for surface mail and $40 per year for airmail. POSTMASTER: Send all changes to EMBEDDED SYSTEMS DESIGN, EE Times/ESD, PO Box #3609, Northbrook, IL 60065-3257, [email protected]. For cus-tomer service, telephone toll-free (847) 559-7597. Please allow four to six weeks for change of address to take effect. Periodicals postage paid at San Francisco, CA and additionalmailing offices. EMBEDDED SYSTEMS DESIGN is a registered trademark owned by the parent company, EE Times Group. All material published in EMBEDDED SYSTEMSDESIGN is copyright © 2010 by EE Times Group. All rights reserved. Reproduction of material appearing in EMBEDDED SYSTEMS DESIGN is forbidden without permission.

Cover Feature:Tracing requirementsthrough to object code verificationBY MARK PITCHFORDVerifying object code can mean the difference between successand failure, quality and crap. But skipping the step because astandard doesn’t require it or because it theoretically eats intoprofits is a surprisingly common practice. The author postu-lates that this practice is not only shortsighted but no longervalid.

19 Troubleshooting real-time softwareissues using a logic analyzerBY DAVID B. STEWARTThis logic analyzer technique is a power-tool for theembedded software engineer’s toolbox.

28 Manage multiple processes and processorsin a deterministic multicore designBY CHRIS MAINHow should processes running on different RTOSes communi-cate in multi-OS systems? The author suggests you can manageinter-process communications with global-object networking.

INDUSTRIAL

MEDICAL

AEROSPACE

AVIATION

SYSTEM ON A CHIP

CONSUMER

Express Logic has completed 14 years of successful business operation, and our fl agship product, ThreadX, has been used in over 1 billion electronic devices and systems, ranging from printers to smartphones, from single-chip SoCs to multiprocessors. Time and time again, when leading manufacturers put their company on the line, when their engineering team chooses an RTOS for their next critical product, they choose ThreadX.

Our ThreadX RTOS is rock-solid, thoroughly fi eld-proven, and represents not only the safe choice, but the most cost-effective choice when your company’s product

ThreadX, FileX, and TraceX are registered trademarks, and NetX, USBX, PEGX, StackX, and Certifi cation Pack are trademarks of Express Logic, Inc. All other trademarks are the property of their respective owners.

Copyright © 2010, Express Logic, Inc.

Express Logic has completed 14 years

When Your Company’s Success, And Your Job, Are On The Line - You Can Count On Express Logic’s ThreadX® RTOS

REALLY COUNTSTHREADX: WHEN IT

simply must succeed. Its royalty-free licensing model helps keep your BOM low, and its proven dependability helps keep your support costs down as well. ThreadX repeatedly tops the time-to-market results

reported by embedded developers like you. All the while, Express Logic is there to assist you with enhancements, training, and responsive telephone support.

Join leading organizations like HP, Apple, Marvell, Philips, NASA, and many more who have chosen ThreadX for use in over 800 million of their products – because their products are too important to rely on anything but the best. Rely on ThreadX, when it really counts!

Contact Express Logic to fi nd out more about our ThreadX RTOS, FileX® fi le system, NetX™ Dual IPv4/IPv6 TCP/IP stack, USBX™ USB Host/Device/OTG stack, and our PEGX™ graphics toolkit for embedded GUI development. Also ask about our TraceX® real-time event trace and analysis tool, and StackX™, our patent-pending stack size analysis tool that makes stack overfl ows a thing of the past. And if you’re developing safety-critical products for aviation, industrial or medical applications, ask about our new Certifi cation Pack™ for ThreadX.

For a free evaluation copy, visit www.rtos.com • 1-888-THREADX

ThreadX, FileX, and TraceX are registered trademarks, and NetX, USBX, PEGX, StackX, and Certifi cation Pack are trademarks of Express Logic, Inc.

, our patent-pending stack size analysis tool that makes stack overfl ows a

Edward L. Lamie

With ThreadX

Second Edition

Now with appendices for ARM, Coldfi re,

MIPS and PowerPC architectures

Newnes

INCLUDEDINCLUDEDINCLUDEDCD-ROM

REAL-TIME

EMBEDDED

MULTITHREADING

Express Logic has completed 14 years of successful business operation, and our fl agship product, ThreadX, has been used in over 800 million electronic devices and systems,

Express Logic has completed 14 years simply simply licensing model helps keep your BOM low, and its proven dependability helps keep your support costs down as well. ThreadX repeatedly tops the time-to-market results

T H R E A D

BY Colin Holland #includeE M B E D D E D S Y S T E M S D E S I G NE M B E D D E D S Y S T E M S D E S I G N BY Colin Holland #includeE M B E D D E D S Y S T E M S D E S I G N

Director of ContentColin [email protected]

Managing EditorSusan Rambo(415) [email protected]

Acquisitions/Newsletter Editor, ESD and Embedded.comBernard Cole(928) [email protected]

Contributing EditorsMichael BarrJack W. CrenshawJack G. GanssleDan Saks

Art DirectorDebee [email protected]

Production DirectorDonna [email protected]

Article submissionsAfter reading our writer’s guidelines, sendarticle submissions to Bernard Cole [email protected]

Subscriptions/RSS Feeds/Newsletterswww.eetimes.com/electronics-subscriptions

Subscriptions Customer Service (Print)Embedded Systems DesignPO Box # 3609Northbrook, IL 60065- [email protected](847) 559-7597

Article Reprints, E-prints, andPermissionsMike LanderWright’s Reprints(877) 652-5295 (toll free)(281) 419-5725 ext.105Fax: (281) 419-5712www.wrightsreprints.com/reprints/index.cfm?magid=2210

PublisherDavid Blaza(415) [email protected]

Associate Publisher, ESD and EE TimesBob Dumas(516) [email protected]

Corporate—UBM ElectronicsPaul Miller Chief Executive OfficerDavid Blaza Vice PresidentKaren Field Senior Vice President, ContentFelicia Hamerman Vice President, MarketingBrent Pearson Chief Information OfficerJean-Marie Enjuto Vice President, Finance Amandeep Sandhu Director of Audience Engagement &

AnalyticsBarbara Couchois Vice President, Partner Services &

Operations

Corporate—UBM LLCMarie Myers Senior Vice President,

ManufacturingPat Nohilly Senior Vice President, Strategic

Development and BusinessAdministration


Safety threats: from satellitesto pods in your pocket

www.embedded.com | embedded systems design | MARCH 2012 5

The spotlight is shining again onthe security of embedded com-puter systems as the Defense Ad-

vanced Research Projects Agency(DARPA) sets up its High-AssuranceCyber Military Systems (HACMS)program.

“Our vision for HACMS is toadopt a clean-slate, formal method-based approach to enable semi-auto-mated code synthesis from executable,formal specifications,” said DARPAprogram manager, Kathleen Fisher.

Embedded computer systems playa part in every aspect of DoD’s tech-nology. The software in these systemsdoes everything from manage largephysical infrastructures and run pe-ripherals such as printers and routers,to control medical devices such aspacemakers and insulin pumps. Net-working these embedded computersystems enables remote retrieval of di-agnostic information, software up-dates, and access to innovative features,but it also introduces vulnerabilities tothe system via remote attack.

In addition to generating code,HACMS is looking for a synthesizercapable of producing a machine-checkable proof that the generatedcode satisfies functional specifica-tions as well as security and safetypolicies. A key technical challenge isthe development of techniques to en-sure that such proofs are composable,allowing the construction of high-as-surance systems out of high-assur-ance components.

Technologies cited as key to

HACMS include semi-automated soft-ware synthesis systems, verificationtools such as theorem provers andmodel checkers, and specification lan-guages. And opportunities will aboundfor industry at large to take advantageas HACMS aims to produce a set ofpublicly available tools integrated into ahigh-assurance software workbench,widely distributed to both defense andcommercial sectors. In the defense sec-tor, HACMS plans to enable high-as-surance military systems ranging fromunmanned ground, air and underwatervehicles, to weapons systems, satellites,and command and control devices.

Of course, security for embeddedsystems is not a new concern. Securityis a major focus throughout this year’sEmbedded Systems Conference, nowpart of DESIGN West. ESC offers twotracks directly on security and anotheron embedded systems disasters, whichwill probably touch on the aftermathof security-related flaws. Also, for thefirst time along side ESC, Black Hat ispresenting a one-day version of itsrenowned Black Hat Briefings, a seriesof technical security conferences thatbring together thought leaders fromthe corporate and government sectorsto academic and even underground re-searchers. On the same day (March29th) Robert Vamosi, author of WhenGadgets Betray Us: The Dark Side of ourInfatuation with New Technologies, willexamine in his keynote address how, inthe rush to embrace the “next newthing,” real concerns about informa-tion security or personal privacy fall bythe wayside.

I hope to see as many of you aspossible at ESC–DESIGN West—findout more at www.ubmdesign.com.

Colin [email protected]

Colin Holland is the directorof content for EmbeddedSystems Design magazine,Embedded.com, and theDesignWest and East (whichincludes the EmbeddedSystems Conferences). You may reach him at [email protected].

INNOVATORS CREATE INTELLIGENT ROBOTS

THAT TRANSFORM MANUFACTURING.

Image supplied by KUKA

When the innovators at KUKA decided to create the foundation for the automation of tomorrow, they decided to work with Wind River. Building upon the proven reliability and breakthrough performance of our VxWorks and multi-core platforms, the KUKA team developed their revolutionary KR C4 controller—a safer, more powerful, more flexible system that allows all integrated controllers to share a common database and infrastructure. And by working with Wind River, they did it all with far less risk.

INNOVATORS START HERE.Please visit http://www.windriver.com/customers to learn more about how Wind River customers have transformed their devices.

KUKA KR C4Control system that integrates robot, motion, sequence, and process control

Built on VxWorks and multi-core operating environments

openFlow and Software DefinedNetworking exhibit tremendouspromise. (“Controlling network

flow with the OpenFlow protocol” DanielProch, www.eetimes.com/4235488)! Ihope the open source consortium makeit easier and transparent for small com-panies and individual like us to get in-volved, participate, and contribute to itsfuture success.

—Netteligent

Many of the possibilities for networkcontrol described for OpenFlow andeven for existing technologies openup questions about the motives be-hind that control. Cable providersand other ISPs are increasingly get-ting into providing content, and it isvery easy to see them tilting the tabletowards their content and againstcompetitors (i.e. Netflix). It is alsoquestionable that centralized controlover all flows works better than a ro-bust distributed control system. Evenif you argue that centralized controlis necessary to tame rogue data (i.e.P2P), there is still dependence onper-packet inspection. Most of thesemarking schemes can be spoofed rel-atively easily. Bottom line, there isstill a lot of work to be done in thisarea.

—Larry Mittag

I am not sure if vendors like Cisco, Ju-niper, etc., are really interested inOpenFlow implementation. Notice,their routers and switches would per-form very simple tasks—no sophisti-cated routing/switching any more.Only simple “match-action” aproach.The “intelligence” is moved to the ex-ternal manager. New features wouldbe implemented externally torouter/switches. So their devices

would be easily replaced by any ven-dor boxes supporting OpenFlow.

—bgolab

Stay on targetAlthough enlightening as always, unfor-tunately, this story line appears to bebranching away from this topic prema-turely (“Quiet down out there!” JackCrenshaw, www.eetimes.com/4235490).Models for Kalman filtering are notan independent choice that can befathomed by examining input/outputdata plots. There are two special re-quirements:

First, the model must be one-steppredictive. Given past and current

data, it must provide a prediction (avery good one can’t hurt!) of what thenext output response will be, giventhe next input, but before that newoutput is observed.

Second, the model form does notuse “x-y” past history directly like theleast-squares fitting discussed so far,but instead, it must reduce past histo-ry to bare essentials in the form of aset of “state” variables (sometimeswithout any direct physical interpre-tation), with model outputs subse-quently derived from these state vari-ables, and with new inputs serving toinfluence future state variable values.The key relationships are describedby a matrix of parameters that I’llcall a “state matrix” for brevity.Without this matrix model, you can’tcalculate the variance propagationequations, nor “Kalman gain” correc-tions to keep the model on-track, andthe ordinary Kalman Filter idea is introuble.

Control theory students knowthat a state matrix is easily obtainedby looking at the last page of thehomework handout. In real world ap-plications, however, the problem ofestimating parameter values for the“state matrix” is a whole lot harder. Isuspect that most attempts to applyKalman filtering are already in troubleat this point, well before reaching theless familiar territory of the variancemodel. It might be interesting to hearabout difficulties others have hadwith this.

I hope that Jack gets back to de-scribe clear and effective ways to dealwith the tricky “model identification”fitting problem, or the rest won’t mat-ter much. Of course, this article wasstill just the beginning of the story.

—RidgeRat

Open to interpretationparity bit

Many of the possibilities fornetwork control describedfor OpenFlow and existingtechnologies open up ques-tions about the motivesbehind that control.

!!!


center of the engineering universe

designwest

It’s incoming fast! For the first time, seven summits including ESC converge at DESIGN West, creating a burst of expert training and education,

hands-on knowledge exchanges, and the latest products and techniques. Be here, at the center of the engineering universe.

March 26-29, 2012McEnery Convention Center

San Jose, CA

Sharpen the skills you need now.

designmed

android

escblackhat

multicore

sensors

center of the engineering universe

Everything you need to know in one location at one time.

g gre, at the center

MarcMcEnery C

Sa

pngineering univ

2012on CenterCA

ubmdesign.com

RegisterNow

C solution using a construct called adiscriminated union, and examine itslimitations.

AN ILLUSTRATIVE PROBLEMSuppose you have an application thatemploys two-dimensional geometricshapes, such as circles, rectangles, andtriangles. At a minimum, each shapeobject contains some linear or angulardistances sufficient to characterize thephysical extent of the shape. For ex-ample, a circle has a radius, a rectanglehas a height and a width, and a trian-gle has two sides and an angle. Theshapes may have common attributesas well, such a position (planar coordi-nates), or outline and fill colors.

A fairly traditional C implemen-tation for a shape is a structure with anested union, such as:

typedef struct shape shape;

struct shape {

coordinates position;

color outline, fill;

shape_kind kind;

union {

circle_part circle;

rectangle_part rectangle;

triangle_part triangle;

} u;

};

The union member, u, is large enough to hold the largestof its members, but it can only store the value of onemember at a time.

In this example, each union member has a differentstructure type. For a circle, you need to store only the ra-dius, as in:

typedef struct circle_part circle_part;

struct circle_part {

double radius;

};

For a rectangle, you need the height and width:

Off and on for almost two years,I’ve been writing about tech-niques for representing and

manipulating memory-mapped de-vices in C and C++. My more recentcolumns have been more about C++than C, focusing on language fea-tures such as constructors and new-expressions, which C++ doesn’tshare with C.1, 2

Some readers have suggested thatthe C code I presented is preferable tothe C++ code because the C structureimplementations for devices are gen-erally simpler than their correspon-ding C++ class implementations. Lastmonth, I argued that the C++ imple-mentations are actually better becausethey’re easier to use correctly andharder to use incorrectly.

Other readers have claimed thatmy C++ implementations are flawedbecause they’re too simple—they don’t use inheritanceand preclude the use of virtual functions.

Classes with virtual functions can be very useful, butthey aren’t the solution to every problem. Classes thatrepresent memory-mapped devices, such as the ones I’vepresented, work with real hardware specifically becausethey don’t use virtual functions. The new C++ Standardacknowledges the usefulness of such classes by definingcategories such as standard layout classes, which avoid fea-tures such as virtual functions.

In the coming months, I’ll explain what virtual func-tions are. I’ll show why they can be useful in some appli-cations, but undesirable in classes that represent memory-mapped device registers. I’ll also show you how C++typically implements virtual functions by showing howyou can emulate them in C.

I’ll begin this month by looking at the sort of problemthat virtual functions are good at solving. I’ll show a typical

By Dan Saks

Discriminated unions

Dan Saks is president of Saks & Associates, a C/C++training and consulting company. For more informa-tion about Dan Saks, visit his website atwww.dansaks.com. Dan also welcomes your feed-back: e-mail him at [email protected].

programmingpointers


Discriminated unions can be useful in some applications, but they also provide insight intothe advantages of usingvirtual functions.

!!!

typedef struct rectangle_part rectangle_part;

struct rectangle_part {

double height, width;

};

For a triangle, you need two sides and an adjacent angle:

typedef struct triangle_part triangle_part;

struct triangle_part {

double side1, side2, angle;

};

When the union members have such simple types,you might find it easier to dispense with the namedstructure types and simply define unnamed structuresinside the union, as in:

union {

struct {

double radius;

} circle;

struct {

double height, width;

} rectangle;

struct {

double side1, side2, angle;

} triangle;

} u;

The value of the shape’s kind member indicates whichunion member is currently in use. The shape_kind typeenumerates the possible values:

enum shape_kind {

sk_circle, sk_rectangle, sk_triangle

};

typedef enum shape_kind shape_kind;

For example, setting a shape’s kind member to sk_rec-tangle indicates that it’s now OK to access (write to orread from) the union’s rectangle member and itsheight and width members.

A union paired with a discrete value that indicatesthe active member of the union is called a discriminatedunion or a tagged union. Some programming languageshave a similar construct called a variant record. The dis-crete value is called a discriminator or tag. I prefer theterms discriminated union and discriminator because tagalready has another meaning in C. The discriminatortypically has an enumeration type, but could have an in-tegral type.

You can write a small assortment of initializationfunctions to properly initialize each kind of shape. Forexample:

programming pointers

Listing 1 A function that finds the largest shape in anarray of shapes, using an overt switch statement.

shape const *largest(shape const s[], size_t n) {shape const *p = NULL;double max = -1;size_t i;for (i = 0; i < n; ++i) {

double area = -1;switch (s[i].kind) {case sk_circle:

area = PI * s[i].u.circle.radius* s[i].u.circle.radius;

break;case sk_rectangle:

area = s[i].u.rectangle.height* s[i].u.rectangle.width;

break;case sk_triangle:

area = sin(s[i].u.triangle.angle)* s[i].u.triangle.side1* s[i].u.triangle.side2 / 2;

break;}if (area > max) {

max = area;p = &s[i];

}}return p;

}

ANONYMOUS UNIONSC++ and more recent dialects of C support anony-mous unions. That is, you can leave the union itselfunnamed and access the union members as if theywere members of the enclosing structure. If you de-fine the shape structure as:

struct shape {~~~union {

circle_part circle;rectangle_part rectangle;triangle_part triangle;

}; // union name omitted};

then you can simplify expressions such as:

s->u.rectangle.height = h;

to just:

s->rectangle.height = h;

My examples don’t use anonymous unions be-cause I tested this code with older C compilers thatdon’t support them.

10 MARCH 2012 | embedded systems design | www.embedded.com

void rectangle_construct

(shape *s, double h, double w) {

s->kind = sk_rectangle;

s->u.rectangle.height = h;

s->u.rectangle.width = w;

}

initializes a shape as a rectangle with a particular height andwidth. You can write similar initialization functions for circleand triangles. Using these functions, you can create an arrayof assorted shapes as follows:

shape sa[4];

~~~

circle_construct(&sa[0], 2);

triangle_construct(&sa[1], 5, 6, asin(0.8));

rectangle_construct(&sa[2], 3, 4);

circle_construct(&sa[3], 3);

Suppose your application needs to determine the shapewith the largest area in a collection of shapes. You can do thejob with the function named largest shown in Listing 1.

Most of the loop body in Listing 1 is a switch statementthat computes the area of a shape. This is arguably a funda-mental shape operation that would be better packaged as anoperation all by itself:

double shape_area(shape const *s) {

switch (s->kind) {

case sk_circle:

return PI * s->u.circle.radius

* s->u.circle.radius;

case sk_rectangle:

return s->u.rectangle.height

* s->u.rectangle.width;

case sk_triangle:

return sin(s->u.triangle.angle)

* s->u.triangle.side1

* s->u.triangle.side2 / 2;

}

return -1;

}

Using the shape_area function dramatically simplifies thelargest function, as shown in Listing 2.

CORRECTNESS AND MAINTENANCE PROBLEMSOne of the problems with discriminated unions is that it re-quires a lot of discipline to ensure that you don’t access aunion member unless it’s the currently active member asindicated by the discriminator. In most cases, this meansyou should be scrupulous about wrapping the accesses tounion members inside if- or switch-statements that test thediscriminator.

Unfortunately, nothing in C prevents accidents such as:

case sk_rectangle:

area = s->u.triangle.side1

* s->u.triangle.side2;

~~~

This code will compile and then produce unpredictable, pos-sibly erroneous, run-time results.

Some languages, such as Ada, have built-in support fordiscriminated unions intended to prevent such mishaps. An-drei Alexandescru showed how you can implement safe dis-criminated unions in C++ using template metaprogrammingtechniques.3, 4

Another problem with discriminated unions is that theycan easily lead to code that’s hard to maintain. For example,in addition to the shape_area function, your applicationmight include other shape operations such asshape_perimeter, shape_resize, or shape_put. Eachfunction has a similar, if not identical, switch-statementstructure as shape_area. If you add a new shape, or modifythe attributes of an existing shape, you probably have tochange every one of those functions.

Virtual functions provide an alternative to discriminatedunions that are safe, efficient, and often more maintainable.I’ll discuss virtual functions in my upcoming columns. ■

ENDNOTES:1. Saks, Dan. “Calling constructors with placement new,” Embedded Sys-

tems Design, September 2011, p. 9. www.eetimes.com/4219506.

2. Saks, Dan. “Using member new to map devices,” Embedded.com,November 2011. www.eetimes.com/4230743.

3. Alexandrescu, Andrei. “Discriminated Unions (I),” Dr. Dobbs Journal,April 1, 2002. http://drdobbs.com/184403821.

4. Alexandrescu, Andrei.“Discriminated Unions (II),” Dr. Dobbs Journal,June 1, 2002. http://drdobbs.com/184403828.

Listing 2 A function that finds the largest shape in anarray of shapes, using the shape_area function.

shape const *largest(shape const s[], size_t n) {shape const *p = NULL;double max = -1;size_t i;for (i = 0; i < n; ++i) {

double area = shape_area(&s[i]);if (area > max) {

max = area;p = &s[i];

}}return p;

}


o longer is it enough to write robust software. The recenttrend in standards development is to prove that a project’srequirements are fulfilled, even if those requirements havechanged during the course of a project’s lifecycle. Require-ments traceability yields a more predictable outcome atdeployment and responds to an increased demand for


sound monitoring and managementtechniques during development, par-ticularly between project phases.

Most requirements traceabilitystops short of object code, suggestingan implied reliance on the faithful ad-herence of compiled object code to theintentions expressed by the author ofthe source code. This can have criticalconsequences, such as putting people’slives risk or having a significant impacton business.

Where an industry standard is en-forced, a development team will usual-ly adhere only to the parts of standardthat are relevant to their application.

Object code verification, on the otherhand, ensures that critical parts of anapplication are not compromised bythe object code, which in principle is adesirable outcome for any software—whatever its purpose. However, can ob-ject code verification be justified as partof a test regime outside the confines ofits enforcement through the requiredadherence to a standard—particularlyin those industries where software fail-ure brings dire consequences, and yetstandards are less mature.

In this article, I explain why it’s im-portant to verify object code and howit’s possible to manage requirements so

Tracing requirementsthrough to objectcode verification

cover feature

you can trace them right through toobject-code verification (OCV).

STANDARDS & CERTIFICATIONSIrrespective of the industry and thematurity of its safety standards, thecase for software that has been provenand certified to be reliable throughstandards compliance and a require-ments-traceable process is becomingever more compelling.

According to research directed bythe U.S. National Institute of SecurityTechnology, 64% of software vulnera-bilities stem from programming errors.For example, an analysis of 3,140 med-ical device recalls conducted between1992 and 1998 by the U.S. Food andDrug Administration (FDA) revealedthat 242 of the recalls (7.7%) were at-tributable to software failures. In April2010, the FDA warned users aboutfaulty components in defibrillatorsmanufactured by Cardiac ScienceCorp. Unable to remedy the problemswith software patches, Cardiac Sciencewas forced to replace 24,000 defibrilla-tors. As a result, Cardiac Science’sshares were hit; the company reporteda net loss of $18.5 million.

The medical-equipment standardIEC 62304 is designed specifically toprovide suitable processes to minimizethe likelihood of such problems inmedical devices. Other industries havesimilar standards as shown in Table 1.

Although each is tuned to a specif-ic industry sector, these standards havemuch in common. In particular, theIEC 61508 industrial standard is some-times used as a basis for other stan-dards, including all of the othersshown except DO-178B/C.

One example of how this com-monality of purpose shows itself is inthe use of Safety Integrity Levels. Ineach case, a risk assessment is complet-ed for every software project to assignthe required assessment safety level ofeach part of the system. The more de-manding the safety level, the more rig-orous and thorough the process andtesting need to be.

REQUIREMENTS MANAGEMENTAND TRACEABILITYRequirements traceability is widely ac-cepted as a development best practiceto ensure that all requirements are im-plemented and that all development ar-tifacts can be traced back to one ormore requirements. The automotive in-dustry’s draft standard ISO 26262 re-quires bidirectional traceability and hasa constant emphasis on the need for thederivation of one development tierfrom the one above it (see sidebar).

While this is a laudable principle,last minute changes of requirements orcode made to correct problems identi-fied during test tend to put such idealsin disarray. Many projects fall into apattern of disjointed software develop-ment in which requirements, design,implementation, and testing artifactsare produced from isolated develop-ment phases. Such isolation results intenuous links between requirements,the development stages, and/or the de-velopment teams.

Leading safety critical standards and safety integrity levels.Industry Standard Safety integrity levelAvionics DO-178B (First published 1992) / DO-178C (2012) Level E to Level AIndustrial IEC 61508 (First published 1998, updated 2010) SIL Level 1 to 4Railway CENELEC EN 50128 (First published 2001) SIL Level 0 to 4 Nuclear IEC 61513 (First published 2001) SIL Level 1 to 4Automotive ISO 26262 (First published and updated 2011) ASIL A to ASIL DMedical IEC 62304 (First published 2006) Class A to Class CProcess IEC 61511 (First published 2003) SIL Level 1 to 4Table 1

MAINTAIN BIDIRECTIONAL TRACEABILITY OF REQUIREMENTSThe intent of this specific practice is to maintain the bidirectional traceability of re-quirements for each level of product decomposition. When the requirements aremanaged well, traceability can be established from the source requirement to itslower level requirements and from the lower level requirements back to theirsource. Such bidirectional traceability helps determine that all source requirementshave been completely addressed and that all lower level requirements can betraced to a valid source. Requirements traceability can also cover the relationshipsto other entities such as intermediate and final work products, changes in designdocumentation, and test plans. —Source: ISO 26262

RTM sits at the heart of the project, defining and describing the interaction between the design, code, test and verification stages of development.

Figure 1

Map requirements to architecture;

generate defects

Manage requirements;assign verification anddebug tasks; track defects

Implement requirements;map to code and verify

design; generate defects

Verifies requirementsagainst test cases;generate defects

Model ordesignspecification

Softwarerequirementsand defectreports

Code base

RequirementsTraceability

Matrix(RTM)

Test cases

Project managers

Test engineers

Software engineers

Development and build engineers


cover feature

The traditional view of software de-velopment shows each phase flowinginto the next, perhaps with feedback toearlier phases, and a surroundingframework of configuration manage-ment and process (e.g., Agile, RUP).Traceability is assumed to be part of therelationships between phases. However,the reality is that while each individualphase may be conducted efficiently, thelinks between development tiers be-come increasingly poorly maintainedover the duration of projects.

The answer to this conundrum liesin Requirements Traceability Matrix(RTM) as shown in Figure 1, which sitsat the heart of any project even if it’snot identified as such. Whether the linksare physically recorded and managed or

not, they still exist. For example, a devel-oper creates a link simply by reading adesign specification and uses that todrive the implementation.

This alternative view of the develop-ment landscape illustrates the impor-tance that should be attached to theRTM. Due to this fundamental centrali-ty, it’s vital that project managers placesufficient priority on investing in toolingfor RTM construction. The RTM mustalso be represented explicitly in any life-cycle model to emphasise its importance,as Figure 2 illustrates. With this elevatedfocus, the RTM is constructed and main-tained efficiently and accurately.

When the RTM becomes the centerof the development process, it has animpact on all stages of design from high-level requirements through to target-based deployment and OCV.

The Tier 1 high-level requirementsmight consist of a definitive statement ofthe system to be developed. This tiermay be subdivided depending on thescale and complexity of the system.

Tier 2 describes the design of thesystem level defined by Tier 1. Above all,this level must establish links or trace-ability with Level 1 and begin theprocess of constructing the RTM. It in-volves the capture of low-level require-ments specific to the design and imple-mentation and that have no impact onthe functional criteria of the system.

Tier 3’s implementation refers to thesource/assembly code developed in ac-cordance with Tier 2. Verification activi-ties include code rule checking and qual-ity analysis. Maintenance of the RTMpresents many challenges at this level, astracing requirements to source code filesmay not be specific enough and devel-opers may need to link to individualfunctions. In many cases, the system islikely to involve several functions. Trace-ability of those functions back to Tier 2requirements includes many-to-few re-lationships. It’s very easy to overlookone or more of these relationships in amanually-managed matrix.

In Tier 4 host-based verification,formal verification begins. Using a test

The requirements traceability matrix (RTM) plays a central role in a development lifecycle model. Artifacts at all stages of development are linked directly to the requirements matrix. Changes within each phase automatically update the RTM so that overall development progress is evident from design, coding, and test through to object code verification at the 6th tier.

Figure 2

Graphics to high-levelrequirements Low-level to

high-levelrequirements

Low-level requirementsto code Code and quality

review defects

Test cases tolow-level

requirements

Test cases tolow-level

requirements

Tier 1

Tier 2

Tier 3

Tier 4

Tier 5

high-levelrequirements

Implementation(source/assembly code)

Host

Target

Requirements Traceability Matrix




model/design concepts

Legacy/hand code

Softwarespecs

Static anddynamicanalysis

Automatedtest

Object code tosource codeTier 6

Object code verification

Requirements Traceability MatrixAutomated

test

Listing 1 void f_while4( int f_while4_input1, int f_while4_input2 ){

int f_while4_local1, f_while4_local2 ;

f_while4_local1 = f_while4_input1 ;f_while4_local2 = f_while4_input2 ;

while( f_while4_local1 < 1 || f_while4_local2 > 1 ){

f_while4_local1 ++ ;f_while4_local2 — ;

}}

Listing 2 1 void1 f_while4 (1 int f_while4_input1 ,1 int f_while4_input2 )1 {1 int1 f_while4_local1 ,1 f_while4_local2 ;1 f_while4_local1 = f_while4_input1 ;1 f_while4_local2 = f_while4_input2 ;————————————————————————————————————-2 while2 (2 f_while4_local1 < 12 ||————————————————————————————————————-3 f_while4_local2 > 1————————————————————————————————————-4 )————————————————————————————————————-5 {5 f_while4_local1 ++ ;5 f_while4_local2 — ;5 }————————————————————————————————————-6 }


strategy that may be top-down, bottomup, or a combination of both, softwaresimulation techniques help create auto-mated test harnesses and test-case gen-erators as necessary. Test cases should berepeatable at Tier 5 if required.

At this stage, we confirm that thesoftware is functioning as intendedwithin its development environment,even though there is no guarantee it willwork when in its target environment.However, testing in the host environ-ment first allows the time-consumingtarget test to confirm that the tests re-main sound in the target environment.

Tier 5’s target-based verificationrepresents the on-target testing elementof formal verification. This frequentlyconsists of a simple confirmation thatthe host-based verification performedpreviously can be duplicated in the tar-get environment, although some testsmay only be applicable in that environ-ment itself.

Tier 6 takes the target-based work astage further, to tie in the comparison ofthe object and source code as part of theRTM and an extension to it.

OBJECT CODE VERIFICATIONThe aerospace DO-178B standard(6.4.4.2 Structural Coverage Analysis)defines OCV as:

“The structural coverage analysis maybe performed on the source code, unlessthe software is Level A and the compilergenerates object code that is not directlytraceable to source code statements.Then, additional verification should beperformed on the object code to estab-lish the correctness of such generatedcode sequences. A compiler-generatedarray bound check in the object code isan example of object code that is not di-rectly traceable to the source code.”

Object code verification hinges onhow much the control flow structure ofthe compiler-generated object code dif-fers from that of the application sourcecode from which it was derived.

cover feature


Listing 3 39 _f_while4:40 push fp41 ldiu sp,fp42 addi 2,sp43 ldi *-fp(2),r0 ; |40|44 ldiu *-fp(3),r1 ; |41|45 sti r0,*+fp(1) ; |40|46 sti r1,*+fp(2) ; |41|47 ble L3 ; |43| New test 248 ;* Branch Occurs to L3 ; |43|49 ldiu r1,r050 cmpi 1,r0 ; |43|51 ble L5 ; |43|52 ;* Branch Occurs to L5 ; |43| New test 353i54 L3:55 ldiu 1,r0 ; |45|56 ldiu 1,r1 ; |46|57 addi *+fp(1),r0 ; |45|58 subri *+fp(2),r1 ; |46|59 sti r0,*+fp(1) ; |45|60 cmpi 0,r0 ; |43|61 sti r1,*+fp(2) ; |46|62 ble L3 ; |43| New test 163 ;* Branch Occurs to L3 ; |43|64 ldiu r1,r065 cmpi 1,r0 ; |43|66 bgt L3 ; |43|67 ;* Branch Occurs to L3 ; |43|6869 L5:70 ldiu *-fp(1),r171 bud r1

Free Evaluation Kits: www.smxrtos.com/evalFree Demos: www.smxrtos.com/demo

Save time – and money – with embedded software solutions built to run right out of the box. Get development started quickly, with no integration required and full supportfor popular tools. With Micro Digital you have low-cost, no-royalty licensing, full source code, and direct programmer support. So get your project off to a great start. Visit usat www.smxrtos.com today.

Your solutionis here.

www.smxrtos.com

R T O S I N N O V A T O R S

ARM • Cortex • ColdFire • PowerPC • x86 • IAR EWARM • CrossWorks • GCC • CodeWarrior

Object code control flow versus sourcecode control flow: Consider the verysimple source code in Listing 1. This Ccode can be demonstrated to achieve100% source code coverage by means ofa single call:

f_while4(0,3);

and can be reformatted to a single oper-ation per line (shown in Listing 2).

The prefix for each of these reformat-ted lines of code identifies a “basicblock”—that is, a sequence of straight linecode. The resulting flow chart for thefunction shows both the structure of thesource code and the coverage attained bysuch a test case with the basic blocks iden-tified on the flow chart nodes (Figure 3).

The object code generated by a com-piler will depend on the optimizationsetting, the compiler vendor, the target,and a host of other issues. Listing 3shows just one example of resulting (re-formatted) assembler code generated bya widely-used, commercially availablecompiler with optimization disabled.

The flow chart looks quite differentfor the assembler code—and using theidentical test case generates a quite differ-ent flow graph both in terms of appear-ance and in terms of coverage (Figure 4).

It’s clear from the flow chart and theassembler code that more tests are neces-sary to achieve 100% code coverage.

• New test 1. Line 62. End of block 3.Branch to L3. This ble branch al-

ways evaluates to false with the ex-isting test data because it only exer-cises the loop once, and so only oneof the two possible outcomes resultsfrom the test. Adding a new test caseto ensure a second pass around thatloop exercises both true and falsecases. A suitable example can beprovided thus:

f_while4(-1,3);

• New test 2. Line 47. End of block 1.Branch to L3. This code contains an“or” statement in the while loopconditions. Both the existing testcases result in the code:

f_while4_local1 < 1

returning a “true” value. The addi-tion of a new test case to return a“false” value will address that:

f_while4(3,3);

• New test 3. Line 52. End of block 2.Branch to L5. The remaining unex-ercised branch is the result of thefact that if neither of the initial con-ditions in the “while” statement issatisfied then the code within theloop is bypassed altogether via theble branch.

So, the final test is added willprovide such a circumstance:

f_while4(3,0);

This results in 100% statement andbranch coverage of the assemblercode, as shown in Figure 5.

• So—to achieve 100% coverage of theassembler code, four tests are required:

f_while4(0,3);

f_while4(-1,3);

f_while4(3,3);

f_while4(3,0);

WHY VERIFY OBJECT CODEFor most development teams softwaretesting traditionally has consisted of onlyfunctional system testing. The problemwas that often the test data selected exer-cised only a limited amount of the sourcecode while in-the-field data was quitedifferent in nature. As a result, much ofthe code was only exercised for the firsttime when it was actually deployed.

Even in the field, it’s highly likelythat circumstances required to exercisemuch of the code have never occurred,and so even legacy applications have sus-tained little more than an extension offunctional system testing through theirin-field use (see Figure 6).

Consider that problem in terms of thedata sets used to exercise that code. It is anatural outcome that most circumstanceswill generate data that exercises a particu-lar part of the code that becomes wellproven—but it’s the exceptional cases thatoccur rarely that cause failure. Indeed,anyone who has been involved with thecorrection of problems seen in the fieldwill confirm how frustrating the repro-duction of relevant circumstances can be.

Structural coverage (see Figure 7)addresses this problem by providing evi-dence that all of the code base has been ex-ercised. Such an approach has been provento reduce the risk of failure and conse-quently is specified in most, if not all, in-dustrial standards concerned with safety.

OBJECT CODE COVERAGEStructural code coverage offers a provenmechanism for ensuring that software isrobust and safe. But we’ve already estab-lished that merely exercising all of thesource code does not prove that all of

cover feature

A dynamic flow graph showing the assembler code exercised through a single function call.

Figure 4

Scr

eens

hot

cour

tesy

of

LDR

A.

A dynamic flowgraph showing 100% assembler code exercised through additional function calls

Figure 5

Scr

eens

hot

cour

tesy

of

LDR

A.

A dynamic flowgraph showing the source code exercised through a single function call.

Figure 3

Scr

eens

hot

cour

tesy

of

LDR

A.


the object code has been similarly exer-cised and proven. True, it’s less likelythat an unexercised route through theobject code will lead to system failure.But even lesser risks can be unacceptableif a system is sufficiently safety-, com-mercially-, or mission-critical.

In short, how big are your risks?Consider the fact that our example

mismatch between source- and object-code flow charts was generated in a com-piler with optimization disabled. Manymore differences are likely as the result ofcompiler interpretation and optimization.While traditional structural coverage tech-niques are applied at the source-code lev-el, the object code executes on the proces-sor—and that’s what really matters.

Any differences in control flow struc-ture between the two can make for signif-icant and unacceptable gaps in the testingprocess. In some industries, these gaps areacknowledged and accounted for. For ex-ample, in aerospace, the DO-178B stan-dard requires developers to implementOCV facilities for those elements of theapplication that have a Level-A (safety-critical) classification. While this is often asubset of the application as a whole, it hastraditionally represented a significantamount of testing effort and hence has al-ways required considerable resources.

Opportunities to implement auto-mated, compiler-independent processescan help to reduce overall developmentcosts by considerable margins, and con-versely make OCV commercially justifi-able in other fields.

AUTOMATED OCV Automated OCV solutions can provide acomplete structural coverage analysis so-

lution for both source and object codefrom unit to system and integration lev-els. Typical solutions combine bothhigh- and object-level (assembler) codeanalysis tools, with the object-level toolvariant being determined by the targetprocessor that the application is re-quired to run on. A typical examplemight see C/C++ and TMS320C25x As-sembler analysis tools teamed togetherto provide the required coverage met-rics, as shown in Figure 8.

OCV AT THE UNIT LEVELSome automated tools enable users tocreate test cases for structural coverageof high-level source and apply the sametest cases to the structural coverage ofthe corresponding object code. A driverprogram is generated by such a unit-testtool that encapsulates the entire test en-vironment, defining, running, and mon-itoring the test cases through initial testverification and then subsequent regres-sion analysis. When used for OCV, thisdriver may linked with either the high-level source unit or the associated objectcode. In so doing, users can apply a uni-form test process and compare code inorder to determine any discrepancies ordeficiencies, as Figure 9 demonstrates.

If structural coverage discrepanciesor deficiencies are identified at the objectlevel, users are presented an opportunityto define additional test cases to close anygaps in the test process. The obvious ad-vantage of identifying and applying cor-rective action at such an early develop-ment stage is that it’s much easier andcheaper. It also significantly increases thequality of the code and the overall testprocess with the latter reaping benefits at

Code exercised both on site and by functional testing is likely to include many unproven execution paths.

Figure 6

Applicationcode base

Code exercised

during functional test

Codeexercised

in service –initial application

Code exercisedin service –enhanced application

Ensuring all code is exercised using structural coverage.

Figure 7

Code base exercised

Data

Functional testIn service—initial application

In service—enhanced applicationStructural coverage


cover feature


the later stages of integrationand system testing and on-ward in the form of reducedfailure rates and maintenancecosts when the application isin the field.

While the code is stillunder development, devel-opers can also benefit fromthe considerable additionaltest feedback together withsatisfying the necessary OCVrequirements in a highly au-tomated and cost-effectivemanner. The results of theseanalysis facilities can be fedback to the developmentteam with the possibility thatfurther code and design defi-ciencies may be identifiedand rectified, further enhancing thequality of the application as a whole.

MAKING OCV THE NORMIt’s clear that OCV has always involvedsignificant overhead and even in theaerospace sector, it’s only enforced as arequirement for the most demandingsafety integrity levels. Even then, the el-ements nominated for OCV in theseapplications are usually a subset of theapplication as a whole—a specialistniche indeed.

Until quite recently, unit test hasbeen considered a text-book nicetyand yet the ever-increasing capabilitiesand ease of use of automated unit-testtools has introduced a commercial jus-tification of such techniques evenwhen risks are lower.

Most applications include key ele-ments in the software; a subset of codethat’s particularly critical to the successof the application and can be identifiedin the application requirements. Thesoftware requiring OCV can be identi-

fied and traced throughan extension to theRTM.

The advent of tools to automate thewhole of that process from require-ments traceability right through toOCV challenges the notion that theoverhead involved can only justify thetechnique in rare circumstances. Just asfor unit test before it, the time has comefor OCV to be commercially justifiablein a much broader range of circum-stances. ■

Mark Pitchford is a field applications en-gineer with LDRA, where he specializes insoftware test.

PASSED

The automation of object code verification can reduce the necessary amount of testing effort by a considerable margin.

Figure 8

User-configurable for different

target environments

LDRAtest

manager

Sourcecode

Assemblersourcecode

Testcase

generation

Testharness

generation

Testexecution

Test resultanalysis and

reporting

InstrumentedAssembler

source

LDRAdesignreview

LDRAtest

verification

LDRATestbeta

TBrun

Object code and source flow graphs illustrate the structure and coverage of the high- and low-level code, and hence any discrepancies between them.

Object code Source code

Scr

eens

hots

cou

rtes

y of

LD

RA

.

Figure 9

BY DAVID B. STEWART, PHD, INHAND ELECTRONICS, INC.

Wbreakpoints to monitor the interrupt will likely break

functionality.

Another example is synchronization and timing errors

in real-time systems. Developers regularly add mutexes,

semaphores, condition variables, or other mechanisms to

guard shared resources. But how do you test and debug an

issue with these mechanisms? Can you tell whether or not a

race condition or priority inversion is actually occurring?

How do you track down a glitch that only happens random-

ly? Can you identify the root cause of a particular real-time

constraint not being met?

To troubleshoot these types of real-time issues, the logic

analyzer, which traditionally is used by hardware engineers

to debug their digital circuits, can be the software engineer’s

best aide. It provides a new power-tool for your software de-

bugging toolbox. It enables tracing through a program with

microsecond resolution and can actively be used to monitor


hile many embedded software issues can be debugged us-ing either print statements or symbolic debuggers (such asJTAG tools), there are many instances where neither ofthose methods can help. For example, suppose the prob-lem is in an interrupt handler that is executing ten thou-sand times per second. It’s not practical to put print state-ments as that will ruin the real-time execution. Adding

Dave Stewart is one of the nation’s leadingexperts in architectures and implementationfor embedded real-time systems. In his manyyears of experience, previously as a consultantand now as director of systems integrationat InHand Electronics, Dave has regularly beeninvolved in the final stages of system delivery.He credits the techniques described in thisarticle as “the primary difference betweenbeing an average engineer who has difficultyfinding closure because there is always ahandful of nasty lingering bugs to fix, and themaster troubleshooter who finds the root causeof the final issues and resolves them to be ableto say ‘Done.’” This article is based on a classhe taught at ESC Boston 2011.

Embedded SystemsDesign magazinebegins a series ofarticles based on ESC classes, whichESC/DESIGN Westinstructors have adapted for our readers.

ESC classroom

This logic analyzer technique is a power-tool for the embedded software engineer’s toolbox.

Troubleshooting real-time softwareissues using alogic analyzer

drivers, interrupt handlers, operatingsystem, or real-time application codethat has critical time constraints.

The logic analyzer is not the holygrail of debug tools by a long shot. Infact, this method is significantly morechallenging than using either print state-ments or symbolic debugging. I’m stillwaiting to have a system with a buttonthat says “Debug Me”—click it, andyour problem is found and fixed. Butuntil that tool exists, you may need to godeep into the trenches, namely the low-est levels of code where print statementsand symbolic debuggers cannot do agood job, to find the root cause of aproblem. The logic analyzer allows thesoftware engineer to do that. The meth-ods I describe in this article are not theeasiest to learn and are extremely diffi-cult to master. However, if you do learnthem and effectively use them, it couldmake the difference between you foreverbeing an average embedded systems en-gineer and becoming an expert.

The techniques I present hereshould supplement your current debug-ging tool box. If you spend an unsuc-cessful day or more trying to track downa problem with other tools, don’t spendmore time using the same methods. In-stead, try the techniques in this article.

The methods can be used on almostany embedded architecture. I have usedthese techniques on small 8-bit and 16-bit microcontrollers with single-threadexecution as slow as 8MHz and usingonly 2K RAM. The same techniqueshave been used equally effectively onmulti-threaded complex ARM architec-tures such as the XScale, OMAP, andDaVinci processors with 512MB RAMand 1GHz clock speed. Although the set-

up is different in each case, the funda-mentals of the logic analyzer methodsremain the same.

The key to understanding the bene-fit of this logic-analyzer approach is torecognize that a software troubleshoot-er is a detective. A debug tool only pro-vides clues. The better the clues, thefaster the problem can be found, andthus the quicker it can be fixed. Thelogic analyzer takes more setup timethan print statements or a symbolic de-bugger, but the quality and quantity ofthe clues can be significantly better.

DEBUG MACROS FOR PRINTINGThe use of macros to standardize printstatement debugging is both useful toknow and represents the basis for logicanalyzer debugging macros.

Consider the procedure when usingprint statements to debug code. Amake-shift code segment is shown inListing 1, with print statements added

to trace the flow of the code. Thiswould produce the following output(with assumed values for x and y):

Start of myfunc

x = 10

f(x) = 24

End of myfunc

While such code will help trace theinternals of myfunc(), it’s rather ineffi-cient to type in all those print statements,and the amount of information providedin each statement is minimal. What ifthese lines of code were just a handfulout of thousands of debug statements?Tracing back to the original source codecould also be time consuming.

A simple solution is to abstract printstatement debugging into macros. Listing2 shows two macros that we can use. Withthese macros, the debug code can bechanged to that shown in Listing 3. Theresulting output would be:

[file.c/3] myfunc()

[file.c/5] x = 10

[file.c/7] y = 24

[file.c/9] myfunc()

When such code is intermixed withthousands of debug statements, it’s stillpossible to easily follow the code, as file-names, function names, line numbers,and values of variables are each clearlymarked. Because macros are used, it’s alsoeasier to use shortcuts or cut-and-paste toadd the debug statements to your code,thus also making it more efficient.

Suppose we want to print the vari-ables in hex instead of decimal. We cancreate alternate macros, as shown inListing 4. If we replace lines 5 and 7 with

ESC classroom

Listing 1 Typical ‘print statement’debugging. void myfunc(void) { int x,y; printf(“Start of myfunc\n”); x = read(); printf(“x = %d\n”,x); y = f(x); printf(“f(x) = %d\n”,y); etc. printf(“End of myfunc\n”);}

I’m still waiting to have asystem with a buttonthat says “Debug Me”,but until then . . .

!!

Listing 2 Macros to standardize print statement debugging

#define DEBUG_WHERE() printf(“[%s/%u] %s()\n”,__FILE__,__LINE__,__FUNCTION__)#define DEBUG_INT(_var) printf(“[%s/%u] %s = %d\n”,__FILE__,__LINE__,#_var,_var)

Listing 3 Usingdebug macros.void myfunc(void) { int x,y; DEBUG_WHERE(); x = read(); DEBUG_INT(x); y = f(x); DEBUG_INT(y); etc. DEBUG_WHERE();}

Listing 4 Defining additional print statement debug macros

#define DEBUG_HEX8(_var) printf(“[%s/%u] %s = 0x%02X\n”,__FILE__,__LINE__,#_var,_var)#define DEBUG_HEX16(_var) printf(“[%s/%u] %s = 0x%04X\n”,__FILE__,__LINE__,#_var,_var)


DEBUG_HEX8() and DEBUG_HEX16()macros respectively, it results in thischange to the outputs:

[file.c/5] x = 0x0A

[file.c/7] y = 0x0018

There are many different ways thatmacros can be used to standardize de-bug print statements, but I won’t coverthem here, since our focus is on debug-ging with the logic analyzer.

The constraint with the print-state-ment method is that you might be ableto add maybe 50 to 100 debug state-ments per second to your code before itbreaks functionality, or maybe just ahandful per second before it affectsreal-time performance. Using the logicanalyzer, it’s possible to get thousandsof debug statements per second withnegligible impact on the system’s real-time performance. Furthermore, thedata is time-stamped and can be visual-ized as timing diagrams, enabling rap-idly identifying anomalies in the code.

The concept of logic-analyzer de-bugging is similar to using these DEBUGmacros, except that the macros will bedefined to send codes to a logic analyzerinstead of print statements to a console.

Before going into detail on logic-analyzer macros, I’ll describe the logicanalyzer and hardware setup needed tosupport this type of debugging.

LOGIC ANALYZERA logic analyzer is a hardware test in-strument that is used to monitor digitalsignals. Typical logic analyzers haveanywhere from 32 to hundreds of digi-tal channels. For software debugging,we generally only need 16 to 24 chan-nels, and speeds under 100 MHz. Sothere is no need to get high speed ormaximum number of channels.

The lower-cost analyzers can be ob-tained for under $500; they’re simply apod that plugs into the USB port of aPC, and software on the PC is used tovisualize the captures. At the other ex-treme are some very powerful analyzersthat can cost $20,000 or more. While

the high-end analyzers have many at-tractive features that can speed up trou-bleshooting, a low-cost one is sufficient,at least for starters.

The one so-called luxury of a logicanalyzer that is highly desirable is thememory depth. Memory depth repre-sents how many samples can be storedin the logic analyzer before its memoryfills up. For example, if sampling at 1million samples per second, an analyzerwith 32-K depth will fill up in just32 ms, while an analyzer with 4-Mdepth can run for four seconds. The lat-ter is extremely beneficial, because it al-lows you to watch the system, have acouple of seconds to manually triggerafter you observe the failure, and knowthat the signals have been capturedwhen the problem occurred. If you onlyhave 32 ms, it’s not possible to manual-ly observe a failure, and thus it’s moredifficult to get the problem captured.

Other features that can be helpfulbut aren’t necessary are good search fil-ters, state machine triggers, and abilityto compress captured data so that sam-ples are only stored when the signals ac-tually change, and thus longer tracescan be obtained.

Every logic analyzer has at least twoviews for captured data, a timing viewand a state view, as Figure 1 illustrates.The timing view is used to quickly visu-alize patterns, identify anomalies, andzoom in to points of interest in a datacapture. The state view is primarily use-ful with more advanced troubleshoot-ing, in which the data is downloaded to

a PC and software is written to auto-matically parse the data and extra infor-mation such as execution time, check-ing for real-time deadline errors, andvalidating that no race conditions oc-curred. How to automatically parse alogic analyzer’s data is a more advanced


Examples of logic analyzer views. Top–timing view; bottom–state view.

Figure 1

ESC classroom

ESC 2012 CLASSESDAVE STEWART

Dave Stewart will teach fourclasses at ESC/DESIGN West inMarch, 2012.

ESC-210: Solving Real Problems thatRequired Hiring a Consultant2:00 PM–3:00 PM March 27

ESC-223: Costly Mistakes of Real-Time Software Development3:15 PM–4:15 PM, March 27

ESC-313: Remotely TroubleshootingEmbedded Systems using High Qual-ity Log Files2:00 PM–3:00 PM, March 28

ESC-316: Achieving Real-Time Per-formance with Linux/Android orWindows CE3:15 PM–4:15 PM March 28

Use promo code UBMDESIGNfor 20% off upcoming ESC class-es. For more information aboutthese and other sessions, go towww.ubmdesign.com/sessions/esc

topic that I won’t cover in this article.

HARDWARE SETUPTo use a logic analyzer to troubleshootsoftware, you must connect it to the em-bedded system under test. Unfortunately,at least for existing hardware platforms,this is sometimes the biggest challenge.

The goal is to identify eight unuseddigital outputs from the embedded sys-tem that can be used to send codes fromsoftware to the logic analyzer. Some-times this is easy. There might be an un-used 8-bit digital I/O port that can beused for this function. In other cases,there might be eight spare pins, but be-cause they’re scattered on various I/Oports, they’re not as easy to write to.Nevertheless, as long as they’re free, thatis good enough.

If there aren’t at least eight availablepins, you may need to scrounge forthem. For example, suppose a system hasan SD card, but the SD card is not usedwhen the failure to be debugged hap-pens. The command, clock, four datalines, card detect, and write protect lines

can potentially all be used to provide 8bits; they just need to be reconfigured insoftware.

If the target system has an FPGA (orCPLD), it may be possible to use eightpins from that device and send a com-mand to the FPGA using whatever com-munication means already exists be-tween those two devices, then allow theFGPA to output the digital signals.

The next challenge is to hook upthese eight outputs to one of the portson the logic analyzer. Most logic analyzerprobes can connect directly to 0.1-inchpins and include clips that allow attach-

ing onto smaller pins. If necessary, ahardware technician can add wires to theembedded system under test to bringout these eight lines in such a way thatthe logic analyzer can more easily beconnected to it.

If the problem that needs to be de-bugged includes some kind of I/O, suchas push-button keys, a controlled digitaloutput, or a serial link, each of these sig-nals should also be connected to otherchannels on the logic analyzer.

Figure 2 shows a block diagram ofthe setup. The I/O connections on chan-nel 2 are optional.

When designing hardware for a newreal-time embedded system, a design-for-test requirement should be to haveeight dedicated digital output pins avail-able specifically for software’s use, and tohave these brought out to a connectorfor easy connection to a logic analyzer.At InHand Electronics, we typicallyplace a high-density connector on theboard with all the signals that we maywant to hook up to the logic analyzer forsoftware debugging, so that board realestate used for software debugging isminimized. We have a separate break-out board populated with 0.1-inch pinsfor each signal that makes it easy to con-nect a logic analyzer. An example isshown in Figure 3; a cable is used tomake the connection from the systemunder test to the breakout board.

Once the hardware is setup, the nextstep is to define macros to output codesto the logic analyzer.

DEFINING LDEBUG MACROSThe first step is to create theLDEBUG_HEX8() macro (or alternately,an inline function) to send an 8-bit codeto the logic analyzer via the digital out-puts identified above. The actual defini-tion of the macro depends on theprocessor, hardware setup, and whichoutputs were selected. An additionalmacro or function, LDEBUG_INIT(),might need to be defined to initialize theselected pins as digital outputs.

Listing 5 shows four examples ofdefining the LDEBUG macros, extractedfrom real target platforms that I’ve usedin the past. In Listing 5a, a TI MSP430had Port 4 reserved for this LDEBUGfunction. The port is initialized to out-put, and writing to the output registersends the 8-bit value to the processor’sport four pins.

A slightly more complex macro isshown in Listing 5b. This exampleshows the macros for a Marvell PXA270,which has separate set and clear regis-ters. In this example, GPIOs 64 thru 71,


ESC classroomHardware setup for troubleshooting with a logic analyzer.

Figure 2

Logic analyzer

(Assuming8-bit

channels)

Embeddedsystem

under test

RX/TX

I2C

SPI

Keys

D0-D7

Ch.1

Ch.2

Example of a design-for-test platform. The embedded system under test (InHand’s DM3730 Fury SBC, in green) has a high-density connector to interface to a breakout board (in red), where all the logic analyzer connections can easily be made.

Figure 3

Photo courtesy InHand Electronics Inc. www.inhand.com

The goal is to identifyeight unused digital out-puts from the embeddedsystem that can be usedto send codes from soft-ware to the logic analyzer.

!!!


which are bits 0 through 7 on GPIO Port2 of the processor, are used.

A third, much more complex scenariohappens when scrounging bits, as shownin Listing 5c. This example is also for aPXA270, but in this case, the specificGPIOs that were available are GPIOs 84,85, 93, 94 on GPIO Port 2, and GPIOs116, 106, 107, and 108 on GPIO Port 3.

The fourth example (which is whatis used on Fury in Figure 3) is shown inListing 5d. This definition is clearly thesimplest but requires that hardware wasdesigned specifically to accommodatethis feature. An FPGA is connected tothe processor’s memory bus, and theFGPA’s application programming inter-face is used to abstract writes to the reg-ister. This example assumes the FPGA isconfigured to output any value writtento register 3 to the 8-bit digital output.

Regardless of the complexity of themacro, once the macro is defined, all re-maining usage is simple by calling LDE-BUG_HEX8(), and the code passed as anargument can be captured by the logicanalyzer.

VERIFYING WITH THE SETUPBefore troubleshooting a real issue, testthe macros and verify the setup usingthe test function provided in Listing 6.This is the equivalent of getting “HelloWorld” program working as a first stepon a new platform.

Execute the function on the targetplatform. It doesn’t matter if this func-tion is executed as a standalone applica-tion, or is integrated into the initializa-tion code of the full system. As long as

it gets executed and the test pattern itgenerates can be captured on the logicanalyzer, it’s good enough.

Setup the logic analyzer to trigger onBit 7 (of the 8-bit digital output port)transitioning from a 1 to a 0. Run thelogic analyzer and wait for trigger. Thenexecute the code. Consult your logic an-alyzer’s technical documentation for

help in setting up such a trigger.If all goes well, the logic analyzer will

get triggered when the code is executed,and the timing view on the logic analyzerscreen will be the step pattern as shownin Figure 4. If this doesn’t happen, re-view all steps above and troubleshoot themacros or hardware setup. If you don’tcapture the pattern at all, it could be an

Listing 5 Examples of defining macros for logic analyzer debugging.(a) LDEBUG macros for TI MSP430 Port 4

uint8_t *LDEBUG_dout8 = (uint8_t *) 0x001D; // reg addruint8_t *LDEBUG_dir8 = (uint8_t *) 0x001E; // reg addr#define LDEBUG_INIT() *LDEBUG_dir8 = 0xFF // output#define LDEBUG_HEX8(_h8) *LDEBUG_out = (_h8)

(b) LDEBUG macros for PXA270 using GPIOs 64 thru 71.uint32_t *LDEBUG_set8 = (uint32_t *) 0x40E00020; // GPIO2 setuint32_t *LDEBUG_clr8 = (uint32_t *) 0x40E0002C; // GPIO2 clruint32_t *LDEBUG_dir8 = (uint32_t *) 0x40E00014; // GPIO2 dir#define LDEBUG_INIT() *LDEBUG_dir8 |= 0x000000FF#define LDEBUG_HEX8(_h8) { \

uint32_t h32 = (uint32_t) hex8; \uint32_t sg2,cg2; \sg2= ( h32 & 0x000000FF); \cg2= ((~h32) & 0x000000FF); \*_gpsr2 = sg2; \*_gpcr2 = cg2; }

(c) LDEBUG macros for PXA270 using 84, 85, 93, 94, 116, 106, 107, and 108 as D0 thru D7 respectively.static volatile uint32_t *_gpdr2 = (uint32_t *)0x40E00014;// GPIO2 dirstatic volatile uint32_t *_gpsr2 = (uint32_t *)0x40E00020;// GPIO2 setstatic volatile uint32_t *_gpcr2 = (uint32_t *)0x40E0002C;// GPIO2 clrstatic volatile uint32_t *_gpdr3 = (uint32_t *)0x40E0010C;// GPIO3 dirstatic volatile uint32_t *_gpsr3 = (uint32_t *)0x40E00118;// GPIO3 setstatic volatile uint32_t *_gpcr3 = (uint32_t *)0x40E00124;// GPIO3 clr

// set directions#define _MASK2 0x60300000#define _MASK3 0x00101C00#define LDEBUG_INIT() {*_gpdr2 |= _MASK2; *_gpdr3 |= _MASK3;}#define LDEBUG_HEX8(_h8) LDEBUG_Hex8(_h8)

void LDEBUG_Hex8(uint8_t h8) {uint32_t h32 = (uint32_t) hex8;uint32_t gr2,gr3,ngr2,ngr3;

// don't write to registers yet; we want to write to registers// as quickly as possible, to minimize race conditions.

gr2=(((h32&0x0C)<<(29-2)|((h32&0x03)<<(20-0)))); // D3.D2|D1.D0gr3=(((h32&0xE0)<<(10-5)|((h32&0x10)<<(20-4)))); // D7.D6.D5|D4h32 = ~h32;ngr2=(((h32&0x0C)<<(29-2)|((h32&0x03)<<(20-0)))); // D3.D2|D1.D0ngr3=(((h32&0xE0)<<(10-5)|((h32&0x10)<<(20-4)))); // D7.D6.D5|D4*_gpcr2 = ngr2;*_gpsr2 = gr2;*_gpcr3 = ngr3;*_gpsr3 = gr3;

}

(d) LDEBUG macros on a system with an adequately-programmed FPGA connected to memory bus.#define LDEBUG_HEX8(_h8) fpga_Write(3,_h8)Listing 6 Function to test logic

analyzer debug macros#include “LDEBUG.h”void LDEBUG_TEST(void) { LDEBUG_HEX8(0xFF); LDEBUG_HEX8(0x01); LDEBUG_HEX8(0x02); LDEBUG_HEX8(0x04); LDEBUG_HEX8(0x08); LDEBUG_HEX8(0x10); LDEBUG_HEX8(0x20); LDEBUG_HEX8(0x40); LDEBUG_HEX8(0x80); LDEBUG_HEX8(0x55); LDEBUG_HEX8(0xAA);};

Logic analyzer output for test pattern.

Figure 4

issue with initialization of the debug I/Oports as output, a problem with writingto these I/O ports, or a channel or triggersetup issue on the logic analyzer. If youget some kind of pattern but it doesn’tmatch what is shown, review each indi-vidual signal connection.

Once you’re able to capture this testpattern, you’re ready to use this tech-

nique for debugging.

DEBUGGING BASICSWhat makes logic analyzer debuggingmore difficult than print statements orsymbolic debugging is that the data cap-tured is very cryptic.

Listing 7 shows a code segment us-ing the print DEBUG macros. Outputfrom executing this code would be:

[file.c/9] pos = 0x0063

[file.c/10] vel = 0x1D18

[file.c/11] acc = 0xFD4E

[file.c/12] tim = 0x0DE5

Listing 8 shows the same code seg-ment instead using the LDEBUG_HEX16()macro, which is built upon theLDEBUG_HEX8()macro created previously.

The logic analyzer capture for thefirst LDEBUG operation is shown in Fig-ure 5. The first code seen is the 0x40, fol-lowed by the 16-bit data, which was splitas two separate 8-bit values, first theMSB, then the LSB.

Zooming out on the logic analyzer,as shown in Figure 6, we see all four ofthe LDEBUG_HEX16 commands. Thecodes (i.e. 0x40 through 0x43) are usedto produce an easily recognizable patternon the logic analyzer. Once this patternis identified, the numbers in betweenrepresent the data variables. By correlat-ing the codes and data values back withthe source code, it’s possible to deter-mine the value of each variable.

Suppose this code was executing athousand times per second, and the logic

analyzer captured one second of data.This particular sequence would then berepeated a thousand times during theone capture. It’s obviously not possibleto look at all thousand items manually.

You can use two approaches to wadethrough thousands of data elements.One approach is to switch to state viewand save the data to a file. You can thenwrite a program or script that parsesthrough the file and looks for specific is-sues. This technique is more advanced; Idon’t describe it here. The other methodis much less formal and often just as ef-fective, and thus will likely be used muchmore often. I call this technique “visual-izing performance,” as described below.

TRACING EXECUTION You can use the logic analyzer to viewprecisely what portion of code was exe-cuted during any interaction of athread or driver or during any execu-tion of an interrupt handler. At thesame time, the logic analyzer capturestiming measurements of each code seg-ment within that function.

Consider the make-shift functionshown in Listing 9 and assume that it’spart of an interrupt handler in whichprint statements are not viable and in-serting breakpoints would break thefunctionality. How do you know whichbranches are taken when?

By instrumenting the code withLDEBUG_HEX8 statements scattered, it’spossible to determine both the preciseexecution trace and execution timemeasurement, as illustrated by the corre-

ESC classroom

Listing 7 Example of using printdebug macros

void get_vector() { pos = get_position(); vel = get_velocity(); acc = get_acceleration(); tim = get_timestamp();

DEBUG_HEX16(pos); DEBUG_HEX16(vel); DEBUG_HEX16(acc); DEBUG_HEX16(tim);};

Listing 8 Example of using logicanalyzer debug macros

#define LDEBUG_HEX16(code,value) {\ LDEBUG_HEX8(code); \ LDEBUG_HEX8(value >> 8); \ LDEBUG_HEX8(value & 0xFF); }

uint16_t pos, vel, acc, tim;

void get_vector(vector_t &data) { pos = get_position(); vel = get_velocity(); acc = get_acceleration(); tim = get_timestamp();

LDEBUG_HEX16(0x40,pos); LDEBUG_HEX16(0x41,vel); LDEBUG_HEX16(0x42,acc); LDEBUG_HEX16(0x43,tim);};

Logic analyzer capture of each item in function get_data() of Listing 8.

Figure 6

Scale: 1 microsecondper division

LDEBUG_HEX16(0x40,pos);LDEBUG_HEX16(0x41,vel);LDEBUG_HEX16(0x42,acc);LDEBUG_HEX16(0x43,tim);

pos vel timacc

A logic-analyzer capture of one data item. Lines in yellow are the per-bit timing diagrams. We’ll see later the value of these.

Figure 5

LDEBUG_HEX16(0x40,pos)


sponding logic analyzer capture shownin Figure 7. For example, during thisspecific iteration, we know that y wasnot less than zero, because the code 0xB3showed up, not the code 0xB2. The errorof z > 0 also did not occur, because thecode 0xBE was not in the capture.

From the log, we could also tell thatthe function do_calc(x,y,z) took ap-proximately 23 µs to execute, based onthe difference between 0xBB and 0xBF.A more precise measurement would bepossible by either zooming into thetime view, or switching to the state viewand looking at the relative time stamp.

As a personal convention, I find ithelpful to separate the 2-digit hex codesas “0xYZ”, where Y=unique numberfor the function, and Z is a locationwithin the function. In Figure 7, Y=B. Zis 1 for entry, F for a normal exit, E foran error exit, and 2 thru D for 12 othertrace points within the function. Thus0xB1 represents entry into the function,and 0xBF represents normal exit. Z=0 isreserved to use as a specific triggerpoint when necessary. As you gain ex-perience, however, you can choose anypersonal conventions that you findhelpful. Having conventions makes iteasier to parse the rather cryptic outputof the logic analyzer.

VISUALIZING PERFORMANCE A very hard—yet rather common—real-time issue that needs to be solved iswhen one or more threads aren’t meet-ing their timing requirements. Quiteoften, the thread that is failing is not theculprit. Rather, it may be some otherthread that has higher priority or usingresources needed by the failing threadthat is at the root of the problem, butknowing which thread to troubleshootcan be difficult to find.

The logic analyzer can be used tovisually spot performance anomalies.Very often, real-time issues show them-selves first at the very low level of exe-cution. By instrumenting code withLDEBUG tracing statements and simplymonitoring execution from a bird’s eyeview, you can acquire a trained eye to

zero in on parts of the code where thereis a potential anomaly. Then, look at thelogic analyzer captures more closely todetermine whether or not any of thoseanomalies are real problems.

An example is provided in Figure 8a.The top diagram is the bird’s eye view.This is a 2-second logic analyzer captureof a system that’s been instrumentedwith LDEBUG macros in various key partsof the code.

In this sequence of diagrams, thegreen is I2C communication, the whiteis the 8-bit debug codes, while the yel-low are the per-bit timing diagrams.While the per-bit diagrams are not nec-essary when looking closely at the 8-bitcodes, they’re helpful in these birds-eyeviews to show patterns. (Note that toconserve space in this article, only fourof the bits are shown; that is generallysufficient to see the patterns.)

Quite simply, most real-time errorsshow up as pattern anomalies whenlooking at a logic analyzer capture. Solet’s focus on some of the anomalies wecan see.

To the left of the top image, the redellipse is circled around an anomaly.Note how much more space there is be-tween adjacent pulses. There is a per-formance issue here that should be in-vestigated. By zooming in on this area,we can look at the individual hex codesto see what part of the code was execut-ing, and how long each code segmentwas taking. However, for this example,let’s instead focus on a different part.

The green represents I2C communi-cation, which seems to come in bursts offive messages at a time. However, nearthe middle, they stop. Is this normal? Oris this indicative of the problem?

To better understand this problem,we can zoom in to the rectangular por-tion, which is the last burst of communi-cation before they stop.

At the new zoom level (Figure 8b),we see more granularity, both with theI2C communication as well as with thepatterns generated by the LDEBUG state-ments. For the I2C, we could see nowthat in the burst of five I2C messages, thefifth one takes longer than the others.This is probably normal, if there aremore bytes transferred. Nevertheless,when trying to find a glitch or perform-ance anomaly, it’s always worthwhile todouble check that your observationsmatch expectations.

At this level, we begin to see detailsof a potential performance anomaly. Re-

Example of code tracing. This is a trace of the function shown in Listing 9. In addition to knowing which branches were taken, the time stamps and scale provide execution time measurements for every code segment, which is shown visually in the timing view.

Figure 7

Scale 10 µsec/div

Listing 9 Tracing code using logicanalyzer debug macrosvoid myfunc(int x) { int y,z; LDEBUG_HEX8(0xB1); y = f(x); if (y < 0) { LDEBUG_HEX8(0xB2); Do stuff; } else { LDEBUG_HEX8(0xB3); } z = g(y); if (z > y) { LDEBUG_HEX8(0xB4); Do stuff LDEBUG_HEX8(0xB5); Do more stuff LDEBUG_HEX8(0xB6); } else { LDEBUG_HEX8(0xB7); if (z > 0) { // ERROR! LDEBUG_HEX8(0xBE); } LDEBUG_HEX8(0xB9); do stuff LDEBUG_HEX8(0xBA); Do stuff LDEBUG_HEX8(0xBB); } do_calc(x,y,z); LDEBUG_HEX8(0xBF);}


ESC classroom

member it is always important to realizethat any anomaly may in fact be normal;calling it an anomaly simply means itmerits a closer look.

The specific anomaly to note is thatthe code 0x46 is always captured beforean extended break. Is this normal? If theconvention is followed that the seconddigit is an F at the end of any function,this means we have something blocking

or taking really long right after code0x46 is printed. This might not be aproblem; for example, it might be a callto a function in another module whichis not instrumented with LDEBUG state-ments. Or it might point to a real issue;perhaps the code is calling a functiontaking much longer than expected, orunexpectedly blocking when trying toacquire a resource.

Suppose the performance anomalywith the 0x46 is normal. We can zoom inagain. In this case, we select the part ofview in which the bit-patterns is differ-ent from everywhere else. It is surprisinghow often using that simple heuristic—adifference in the bit pattern—is enoughto zero in quickly on real problems.

In the zoom of that area (Figure8(c)) we see two more potential anom-alies, namely extended periods followingeach of codes 0xAF and 0x9F. The ques-tion then, is what happens immediatelyafter those codes are sent to the logic an-alyzer? Repeating this approach, takinga few minutes per anomaly, most per-formance issues in the system can beidentified within a few hours.

Figure 8(d) shows on additionalzoom, which is the I2C communication.Some logic analyzers will also show thehex values as well as protocol start andstop bits for serial communication, asshown in Figure 9.

SIMPLY LOGICALThe logic analyzer techniques describedin this article provide you one addition-al, powerful tool to use to debug real-time software issues.

Consider this metaphor: if you need-ed to cut a branch from a tree, you woulduse a branch cutter that takes a second topull out of the shed, “snip”, and it isdone. You would not go get a chainsawfor such a simple job. However, if youneeded to cut down a large tree, thebranch cutter is useless, and a chainsaw ismuch more powerful than an axe or saw.The logic analyzer is like that chain saw.It’s powerful, but more difficult to set upand more awkward to use. Use it whennecessary, but still hang on to all yourother tools and use those when they canget the job done more efficiently. ■

Dave Stewart is the director of Researchand Systems Integration at InHand Electron-ics, a developer of small, low-power elec-tronics solutions for handheld and wirelessdevice designers. He has significant experi-ence in numerous market segments, includ-ing mobile computing, wireless sensor net-works, aerospace, industrial control,consumer electronics, transportation, physi-ological monitoring, and environmentalmonitoring.

ESC classroomLogic analyzer capture of performance anomalies. (a) A high level “bird’s eye” view of performance. Green shows communication that stops for an extended period of time. (b) Zoom of (a), we could see the patterns resulting from the per-bit timings. (c) Zoom of (b). When we look closely we see several large gaps with the codes 0xAF, 0x9F, and 0x46. (d) Zoom of I2C communication.

Figure 8

A

B

C

D

Logic analyzer with built-in I2C protocol analysis.

Figure 9


How should processes running on different RTOSes communicate in systems with more than one operating

system? The author suggests you can manage inter-process communications with global-object networking.

Manage multiple processes & processors

in a deterministicmulticore design

BY CHRIS MAIN, TENASYS CORPORATION

been implemented as separate compute platforms interconnected via

a communication network.

As the computing capabilities of off-the-shelf PCs have multiplied,

original equipment manufacturers (OEMs) now can consolidate multiple

functions onto a single platform to improve the reliability and efficiency

of their systems, at the same time driving out dedicated hardware

in favor of software-based implementations.

However, there are potential problems with consolidating

the functions that were previously performed by separate

feature

Twww.embedded.com | embedded systems design | MARCH 2012 2

he typical embedded control system consists of a number of in-dependent dedicated functions that interact with each other toperform a task. Functions may include control elements, dataacquisition elements, and sometimes a human/machine inter-face, all working together. In many cases, these functions have

7

computing platforms, and these prob-lems can cause the system to fail if notdealt with successfully. Integrated sys-tems mzust not remove the advantagesthat multiplatform systems accom-plished by separating the functions.

One advantage of keeping systemelements separate is determinism. Em-bedded systems need to respond to ex-ternal stimulus in a reliable and pre-dictable manner. Therefore, the meansof communication between the appli-cation elements must be efficient, andthe receiving elements must be able to

rely on the data being timely and accu-rate.

Implementing each function as aprocess running under one real-timeoperating system (RTOS) might not bethe best method. Scheduler interferencebetween the functions results in lack ofindependence, which can causeprocesses that were previously executedwith highest priority in a separate com-puting platform system to fail in a con-solidated environment.

Virtualized operating systems havebeen used to achieve system consolida-tion, but when applied as an after-thought these methods have had limit-ed success in supporting real-timedeterminism and scalability of the

available processing power. Althoughsome solutions yield successful results,they generally require customers tomake compromises that limit the capa-bilities of their systems. Ultimately, de-terminism and scalability must be de-signed into the core of the softwarefrom its inception.

ESTABLISHING REQUIREMENTSAn idealized control system may be de-scribed by a series of independent func-tional blocks (FBs), each of which per-form a specific task at a specific regularinterval (Figure 1). FBs are made up ofapplication-specific hardware and soft-ware components and a processor orcontroller to implement the function.Each FB may have a dedicated connec-tion to external hardware such as mo-tors, cameras, or other I/O points. EachFB may also be connected to one or

more of the other FBs to implement awhole control system, but otherwise,each FB operates independently of oth-er FBs.

An example of such a system (seeFigure 1) could be where FB2 is a PLC(programmable logic controller) andFB3 an intelligent self-contained opticalanalyzer that captures the image of anobject it’s inspecting, analyzes it, andinforms the PLC of the result with aGO/NO-GO output signal. FB1 couldbe a motor drive that feeds the next ob-ject that needs to be inspected.

With the availability of increasedcomputing performance, it’s possible tomove some or all of the functions ontoa single hardware platform. The advan-tage of this is a more efficient intercon-nect between the functions, which leadsto better overall system performanceand reliability and possibly reducedcost. But consolidating the functionsmeans the development team needs toadd system management, because thesystem is now apportioning commonresources such as memory and access toI/O between the FBs.

SINGLE PROCESSOR SOLUTIONSA typical way to combine functions ona single platform such as a PC is to im-plement the function blocks as separateprocesses running under an RTOS.RTOSes have built-in functions—suchas preemptive task scheduling—to en-sure that each FB’s task is executed at aprescribed time and that a task fromone FB can preempt the processing ofanother FB’s task if it has a higher pri-ority. A drawback of this implementa-tion is that the platform resources arenot separated. Therefore, events such asunusual I/O activity needing to be han-dled by one of the FBs could affect theother FBs’ ability to perform their tasks.

The result is that setting up a sys-tem in this environment takes sometuning to ensure that tasks of the vari-ous FBs have the appropriate level ofpriority and resources. This typicallymakes the system somewhat custom-fitto the platform and thus making it dif-


feature

An idealized system as a series of independent functional blocks (FBs), each performing a specific task at a specific regular interval and each connected by a communication link.

Figure 1

Drive controller Video capture and analysisPLC

Inter-process communications (IPC) in single processor systems is straightforward.

Figure 2

The system manager determines how FBs are distributed among the CPUs in a multicore processor system.

Figure 3

It’s possible to movefunctions onto a singlehardware platform, butconsolidating means thedev team needs to addsystem management.

!!!

ficult to move from one platform to an-other. It’s also difficult to entertain anysubsequent changes to the system, suchas the addition of another FB or a per-formance improvement in one of theFBs, without extensive planning. Final-ly, the finite processing capabilities of asingle processor generally limit the in-crease in system performance that canbe obtained.

On the plus side, single processorimplementations are relatively simple.They’re typically run by one RTOS thathas built-in IPC (inter-process commu-nication) facilities, making it easy forthe FBs to communicate with each oth-er with minimal overhead (Figure 2).

MULTICORE SOLUTIONSSingle system manager: The introduc-tion of multicore processors made a bigimpact in the computing industry as awhole. However, the adoption of multi-cores in control-system applications hasbeen a little slow for two reasons. Thefirst and main reason is that OEMshave not been sure how to migrate ap-plications that were optimized to runon a single processor onto a multicoreprocessor platform without breakingthem. And second, the industry has notproduced a clear path to follow for mi-grating applications from single proces-sor systems to multicores that enablesOEMs to get the most out of multicoreprocessors without requiring significantchanges to the software.

In an effort to provide the easiestsolution, multicore system managerswere introduced (Figure 3). These areessentially operating systems with loadschedulers that make decisions on howto load the FBs (applications) on theprocessor cores. Generally, the methodused is SMP (symmetrical multipro-cessing). Although this processor load-ing method has proved acceptable fornon-real-time applications, it falls shortof the requirements for real-time con-trol systems. FBs scheduled under SMParen’t guaranteed to be totally unaffect-ed by the operation of other FBs, andthe system’s I/Os are not partitioned

and dedicated to the associated FBs.

Multi-system manager: A better ap-proach is to go back to the fundamen-tals and use embedded virtualizationtechniques to run each FB on a separate

core of a multicore platform, with allo-cated/ dedicated I/Os and memorymanagement (Figure 4). With each coreunder the control of an individual sys-tem manager (RTOS), the system is es-sentially an AMP (asynchronous multi-processing) system.

The missing bit in this implementa-tion is ability for the FBs to communi-cate with each other. This can be addedto the application layer or embeddedwithin each system manager (RTOS).The former is easier to implement butdifficult to make transparent to the de-veloper. It’s even harder to integratewith the application’s priority level sothat any inter-FB communication ismanaged as part of the prioritization ofother tasks that are running.

A better method is to integrate theIPC mechanism into the RTOS (Figure5) where it may be automatically han-

dled by the RTOS priority scheduler.This enables programmers to refer tostandard exchange objects—such asmailboxes, semaphores, and sharedmemory blocks—within their code toperform IPC between processor coresand across platforms. This embeddedIPC methodology is so transparent thatit allows FBs to be separated and dis-tributed among available processorswith minimal code changes.

For this system to work reliablythere must be a means for the FBs toknow the state of other FBs on whichthey depend. This is done by adding asystem-manager layer to the environ-ment. This layer provides a means ofsignaling system events, such as the cre-ation and deletion of interdependentprocesses. Both the IPC and the man-agement layers are designed to be ex-tended across multiple instances of theRTOS, thus making it easy to scale asystem from one to many processors, bethey on the same platform or even mul-tiple platforms.

IMPLEMENTING GLOBAL OBJECTS Global objects provide a means for im-plementing the above and have the fol-lowing characteristics:

• Each instance of the RTOS islaunched on each core of a multi-core processor. In this particularRTOS, system functions are imple-mented as objects, each referred toby a handle. Each instance of theRTOS operates independently ofthe others, with its own separatememory, scheduler, I/O devices,


Figure 4

A good way to ensure that real-time tasks don’t affect one another is to run them on separate cores, each controlled by its own RTOS.

Interprocess communications capability is integrated into each RTOS, allowing multi-FB applications to be hosted on multicore processors with minimal code changes.

Figure 5

feature

A better method is tointegrate the IPC mechanism into the RTOSwhere it may be automat-ically handled by theRTOS priority scheduler.

!!!

and associated interrupt management. Processesthat run on one instance of the RTOS have noimpact on processes that run on another instanceof the RTOS.

• An efficient, asynchronous message-passing sys-tem provides the common transport to supportthe IPC and management layers across multipleinstances of the RTOS.

• The system APIs are extended transparently toallow a process on one instance of the RTOS toaccess objects on another instance of the RTOS.This is the same method by which processes com-municate within a single instance of the RTOS,

but the scope of the object handles has beenextended by means of the IPC layer so that a han-dle may refer to an object elsewhere. Eachinstance of the RTOS has a global object managerprocess to manage this. This management processacts as an agent for remote processes interactingwith local objects.

• The system-management layer (called DSM, orDistributed System Manager, in the exampleshown in Figure 6) allows processes to registertheir dependence on other processes in order thatthey’re notified if another process in the relation-ship is terminated or compromised. The DSMlayer also monitors nodes (instances of theRTOS), so that if one is shutdown or fails,processes on other nodes are notified.

• In this example, which implements the examplesystem described earlier, an instance of theWindows OS has been added and the functionali-ty is also extended to that operating system.

Referring to the application in Figure 1, we cannow have the video camera application (which we’llcall FB3) run by Process C on Core 3 and the PLC(FB2) run by Process B on Core 2 of a multicore plat-form. As before, the video application needs to informthe PLC when it has completed its video capture and


The dual RTOS scenario.

Native Parallelism for High-Performance Reduced-Energy Computing

It all Rides On the Right Technology

TMGet Pattern Recognition Right with CogniMem!

$995

Technical Training

Sensor Input

Classification

Identification

Anomaly Detection

Nothing to Declare

TheCloud

!

Data Mining/Analytics

1,024 cognitive

memories CogniBloxTM 4 - CM1K chips/board

(4,096 cognitive memories)Stack for unlimited performance

ADCAct

The system-management layerallows processes to register theirdependence on other processes in order that they’re notified if another process in the relationshipis terminated or compromised.

!!!

analysis and tell the PLC whether theresults are successful or not. If both ap-plications were run on a single proces-sor, passing such information would beeasy to do using a mailbox transportobject.

With global objects and the associ-ated global object network, the samemechanism can be used with the addi-tion of three program steps. First, thePLC (Process B) needs to make theMailbox “global” so that processes run-ning on other nodes/processors can dis-cover it. Then two additional instruc-tions must be executed to allow aremote process, such as Process C, tofind a global object, in this case theMailbox, so that it can write to it.Table 1 shows a listing of the com-mands that would be executed by bothprocesses, with the additional instruc-tion shown in bold and underlined.

Note also that the DSM registersthe connection between the processeswhen the Mailbox is established. So ifthe Process B that is the PLC applica-tion faulted, the video app, Process C,would be notified and would take ap-propriate action.

The concept of passing informationfrom one independent functional blockto another is extendable with memoryobjects. The steps to setup a memoryobject are similar to those of setting upa Mailbox. Global memory objects arebetter suited for the transfer largeblocks of information. An examplewould be passing position-correctinginformation directly from the video ap-plication to the motor drive upon par-tial failure of results from the videocapture and analysis.

Because memory objects are oftensubscribed to by more than twoprocesses, it’s necessary to allow amemory object to remain present eventhough the process that created it mighthave terminated and has no longer anyneed for it to be present. Global objectsprovide such a mechanism with the in-clusion of an Index. For example, if amemory object is expected to be sub-scribed to by four processes, the process

that creates the memory object assignsit an initial Index =4. Every time an ob-ject unsubscribes from the memory ob-ject the index is decremented. Onlywhen the last process unsubscribes doesthe memory object get removed.

Global objects extend to a Windowsenvironment as well. They can be used

to allow applications that require an ad-vanced Windows-based human inter-face to communicate information withRTOS-based functional blocks. In ourexample system (Figure 1), a Windowsapplication could read video informa-tion directly from the memory objectthat is put there by the video applica-tion and display it for visual inspectionby an operator.

The above example shows howglobal objects can be used by independ-ent FBs to communicate informationand how the ability to pass large blocksof information can even extend the sys-tem capabilities. Moreover, becausethey provide a mechanism for enablingdistributing FBs to communicate, theuse of global-objects-based communi-


The sequence of operations that allows processes on different processors to communicate involves adding just two additional instructions to the flow that would have been used with a single-processor implementation.Process B Process CPLC Video app. CommentCreateRtMailbox Create Mailbox A

CatalogRTHandle Catalog Mailbox in root process

GetRtNodeLocationByName Get location of remote mailbox

GetRemoteRootRtProcess Get remote root process handle

ReceiveRtData Wait for message indefinitely

LookupRtHandle Get handle of Mailbox A

SendRtData Send data to Mailbox A

Wake up on receipt of data and read data

DeleteRtMailbox Delete mailbox

DeleteRtReferenceObject Delete reference object

Table 1

feature

The concept of passinginformation from oneindependent functionalblock to another isextendable with memoryobjects.

!!!

An example of a global object implementation, TenAsys GOBSnet facilitates communication between processes on different platforms.

Figure 6

cations can also be used to “break-up”an FB like a motor-drive across twoprocessors if needed.

IMPLEMENTATION ISSUESWhen a process on one RTOS is inter-acting with an object on another RTOS,the behavior of both instances of theRTOS must continue to be predictable,and the overhead of performing the op-eration must be negligible. More im-portantly, functional integrity of thesystem must be preserved. Task priori-ties must remain intact, and the imple-mentation must ensure that false prior-

ity and priority inversion problems,such as a low-priority thread on onenode blocking a higher-priority threadon another node, are avoided.

Overheads of the implementationare kept to a minimum and predictabil-ity is maximized with the implementa-tion of a light-weight message-passinglayer based on a simple shared memoryplus interprocessor interrupt for thetransport. Operations are handled by alocal agent on the receiving node on be-half of the sending thread, and are di-vided into two classes. The first class in-cludes creation, deletion, and signalingof objects, which are performed by theagent directly. The second class of oper-ations are those that wait on an object.The latter operations are handled byproxy threads that execute at the priori-ty of the original caller, so that the call-ing thread is effectively woken up ac-cording to the scheduling policy of theRTOS where the object is resident.

As an example, consider twoprocesses, each of which is implement-ing one of the FBs running on a singleinstance of the RTOS. Process A con-tains an object that’s being waited onby a thread in Process B (Figure 7).

When the object is signaled, the threadin Process B becomes ready to runwhen it’s the highest-priority thread onthe scheduler’s ready list.

In Figure 8, Process A and ProcessB are on separate instances of theRTOS. When the object is signaled, theproxy thread waiting on it is madeready. When it’s the highest-priorityready thread on the RTOS instance, itruns and completes the transaction onbehalf of the thread in Process B, whichis now made ready.

The effect of this is that the threadin Process B is subject to the schedulerstate of RTOS 1 when it’s waiting onobjects that reside on that instance ofthe RTOS. This ensures that there aren’tany arbitrary priority issues when theprocesses are interacting across separateinstances of the RTOS.

PERFORMANCE CONSIDERATIONSFor processes that have tight loops, it’snecessary for operations to carry negli-gible to no overhead. One way ofachieving that is to use a memory ob-ject. A memory object may be mappedinto many processes on different nodes,just as it may be mapped into manyprocesses on the same node. Access tothe memory carries no additional over-head once the mapping is complete,and controlling the access to the mem-ory can be managed, for example, bysome synchronization object such as asemaphore accessed by all processes, ifthe specific application requires it. Thisensures that processes across the nodesaccess data at a specific time to avoidany contention issues and provide datasynchronicity across the nodes/func-tional blocks. ■

Chris Main is CTO and cofounder ofTenAsys Corporation, where he has leddevelopment of the company's virtualiza-tion technologies. He has worked in real-time systems starting with minicomputersand was involved in the iRMS group atIntel. He was on the original develop-ment team for INtime. He earned a grad-uate degree in physics from York Univer-sity and postgraduate degree ineducation from Bath University.


feature

The dual RTOS scenario.

Figure 8

Two processes, each under the control of a separate instance of the RTOS. Thread in Process B waits on an object in Process A. In the dual RTOS scenario, Thread A runs as a Proxy in the local agent in RTOS 1. When the object is signaled, and the Proxy thread is the highest-priority thread on RTOS 1 then Thread A runs under Process B. In the TenAsys implementation, the Local Agent is known as the GOBS Manager and the IPC is called GOBSnet. Implementation and management of GOBSnet is embedded in the individual GOBS Managers.

Two processes under the control of a single instance of the RTOS. A thread in Process B waits on an object in Process A.

Figure 7

Stay up on new products. For thelatest new products anddatasheets, go to Embedded.com,

www.eetimes.com/electronics-products,and Datasheets.com.

Kontron announces deployable industrial temp-rated platform forsmart M2M applicationsToni McConnel, Embedded.com

Kontron’s newready-to-deploy Ma-chine-to-Machine(M2M) hardwareplatform, the Kon-tronKM2M806XT, istargeted for applica-tions that must op-erate in extended temperatures and in-dustrial environments.

The KM2M806XTcomes completewith application-ready middleware andsupports M2M digital device connec-tivity out of the box. An optimal plat-form specifically for factory and smartbuilding automation applications,Kontron’s new M2M system is indus-trial temp rated (-40 to +85 °C) and in-cludes a cast aluminium case with ther-mal management capabilities neededfor operation under extended tempera-ture conditions.

The KM2M806XT is based on theIntel Atom processor architecture. Thedevice is designed with a modular ap-proach, which includes a carrier board,audio/video board, a Kontron COMExpress mini Computer-on-Module(Kontron COMe-mTT10), an optionalexpansion board along with a M2Msoftware development kit from WindRiver. The modular approach enables asimple and cost-efficient tailoring tospecific application needs. Specific in-

terface demands or processor require-ments can be integrated. See full storyat www.eetimes.com/4236332.Kontronhttp://us.kontron.com/products/systems+and+platforms/m2m/km2m806xt.html

Elliptic unveils tVault security ar-chitecture for trusted executionenvironmentsToni McConnel, Embedded.com

tVault Security Architecture from Ellip-tic Technologies enables end-to-end se-curity in connected devices, helpingSoC vendors, OEMs, ODMs, and net-work operators create products withstandards-compliant security. tVaultprovides security inside Trusted Execu-tion Environments (TEEs) and enforcesthe protection of sensitive informationto ensure that it is stored, processed,and accessed only by authorized appli-cations.

tVault enables highly secure envi-ronments that allow for a wide range ofapplications including Digital RightsManagement (DRM) and content pro-tection, Over-The-Top (OTT) TV, mo-bile TV, and electronic financial trans-actions to be deployed in highperformance computing platforms.tVault consists of firmware, standardAPIs, hardware support, and accelera-tion that runs efficiently in limited re-source environments. tVault integrateswithin frameworks such as ARM Trust-Zone, where the critical security com-ponents are embedded in a trusted andsecure OS environment. The non-criti-cal components are executed by the richOS, such as Android.

See the full story online at www.eetimes.com/4236145.Elliptic Technologieswww.elliptictech.com

Telit’s M2M modules support AIAPioneer and GoFer remote monitoringToni McConnel, Embedded.com

Telit Wireless Solutions M2M moduleswill enable connectivity for remote mon-itoring applications from Alta IndustrialAutomation Ltd. (AIA). LeveragingTelit’s G24 and H24 modules for its Pio-neer and GoFer product lines, AIA en-ables real-time monitoring of a widerange of mobile and fixed industrial as-sets, such as gas lines, irrigation systems,chemical tanks and automotive fleets.

Equipped with Telit’s G24 and H24modules, variations of AIA’s Pioneer andGoFer devices feature low-power con-sumption to extend the life of remoteM2M deployments, maximizing cus-tomers’ return on investment. See fullstory at www.eetimes.com/4236134.Telit Wireless Solutionswww.telit.com

LED driver controller handles de-manding lighting applicationsIsmini Scouras, EE Times

By extending input voltage operationfrom 60 V down to as low as 5.0 V, theZXLD1371 LED driver controller fromDiodes Inc. has been designed to handleas broad a range of applications as pos-sible, including those subject to thegreatest variance in input voltage, suchas automotive lighting and 12-VACpowered systems.

Its ability to control an externaldriver FET in buck, boost and buck-boost driver topologies makes it an all-in-one solution, capable of driving asmany as 20 high current LEDs in boostmode with a high level of accuracy. Seefull story at www.eetimes.com/4236083.Diodes Inc.www.diodes.com

Smart M2M and secure architecture

new products



By Jack G. Gansslebreak points

Back when the Earth was startingto cool and the first creatureswere crawling from the slime, we

were building a very fast system usingplenty of discrete logic because theprocessors of those Paleozoic yearscouldn’t keep up with our data rates. Iwas probing a pair of signals on a fabu-lously-expensive oscilloscope, but theirrelationship made no sense. EventuallyI realized that one of the scope probeleads was a full meter longer than theother. This was the first, but certainlynot the last, time I cursed light’s snail-like pace. And it was the first time Iwas forced to think about scopeprobes, other than the routine of com-pensating them.

Eons later I still see engineers root-ing around in a bin and untangling arandom, often abused, probe, carelesslyconnecting it to the scope that theyspent a week evaluating and compar-ing to a host of other products. Butthat $10k or $20k instrument (ormore; I read a press release for a nifty$400k model this week) can be com-pletely hobbled by a lousy or poorly-selected probe.

A scope probe is not a wire. Sure,it’s an electrical connection between anode on a board and the oscilloscope.But its resistance, capacitance, and in-ductance have serious consequences asspeeds increase.

Plenty of white papers talk aboutproper probe selection, but to myknowledge none show how the wrongprobe can either cause your circuit tofail or even physically destroy compo-nents. Let’s look at some of the issues.

For readers without an EE, here’s abit of background. There will be a littlemath, but fear not! Mostly this is assimple as Ohm’s Law, which is:

E=IR

E is voltage, I current, and R resistance.But there’s really no such thing as aperfect resistor. Every component, be ita wire, resistor, or anything else hassome amount of capacitance and in-ductance associated with it. So, while

Ohm’s Law holds true, as frequenciesincrease we need to include these ef-fects in our thinking about resistance.

A capacitor passes AC but not DC,and the higher the frequency, or thelarger the capacitance, the more easilyit passes current. This is known as ca-pacitive reactance, which is a form ofresistance to alternating current. It’sexpressed as:

C is the capacitance in farads and fis the frequency in Hz. If f is zero—DC—the reactance is infinity and aperfect capacitor completely blockscurrent flow. As the signal gets fasterit’s reactance drops.

Inductors are similar:

In this case L is the inductance. Soan inductor, too, has a resistive-like ef-fect, although it is inverse of that of thecapacitor.

If a circuit has inductance and ca-pacitance, the total reactance is, well,it’s thankfully not important to thisdiscussion, but suffice to say that onecan compute a net reactance. Toss inpure resistance and the total imped-ance—which is merely resistance towaveforms—in a parallel circuit is:

1

2π=X

fCc

X fLl 2π=

Probing pointers

The wrong probe cancause your circuit to failor even physicallydestroy components.Here are some of theissues.

!!!

Jack G. Ganssle is a lecturer and consultant on embeddeddevelopment issues. He conducts seminars on embedded systemsand helps companies with their embedded challenges.Contact him at [email protected].


Ohm’s Law still holds, except wenow replace R with Z. So it’s possible tocompute current flow in a complicatedcircuit at any given frequency if oneknows the voltage and impedance.

I said there’s no such thing as aperfect resistor. A typical ¼ watt resis-tor has 0.5 pF of shunt capacitance. Awire-wound resistor is a coil and hasinductance (although some have re-verse windings to limit this). A normalfilm or carbon resistor has very littleinductance.

A scope probe has capacitance, re-sistance and some inductance (mostlyin that black ground wire), and lookssomething like Figure 1.

A scope probe, because of its im-pedance, is rather like a resistor thatgets placed in parallel with the nodeyou’re testing. The net impedance oftwo in parallel is:

Thus, placing a probe on a node re-duces the impedance of that point.Whatever drives the node must, in effect,push harder or the signal will degrade.

But is this really a big deal? Scopeprobes are typically specified at 10 mil-lion ohms, which is an enormous value.Put that in parallel with a digital device,whose resistance is orders of magni-tudes lower, and there will be no meas-urable effect.

But what’s important is impedance,not resistance, and as noted earlier, im-pedance depends on frequency. Ifyou’re probing a digital circuit odds arethat things are switching pretty quickly.The frequency may be quite high.

That’s where the second part of aprobe’s spec, the tip capacitance, comesin. This ranges from under a pF to100 pF or more. And that has a huge ef-fect on the probe’s impedance. Tektron-ix is one of the few vendors that givesgraphs of probe impedance. TheirTPP1000, a 1-GHz passive probe, hasthe impedance vs. frequency curve inFigure 2.

At 100 MHz the impedance appearsto be around 500 ohms. The probe’s ca-pacitance is 4 pF; running the reactancenumbers we get 400 ohms at that fre-quency, pretty darn close. At low fre-quencies the reactance gets very large sothe 10-MΩ resistor dominates.

This is a $900 probe. Cheap probesmay be considerably worse. The capaci-tance of Pomona’s $117 5812A, ratedfor 300 MHz, is listed as “<17 pF.” Let’sassume 17. That’s 93 ohms at 100 MHz.In a 5-volt circuit, the probe will add a54-ma load.

Surfing the net one finds lots ofcheapies at 100 pF or worse, which netsa terrifying 20 ohms at 100 MHz. To

put that into perspective, in a 2-volt cir-cuit it would take 100 ma to drive theprobe alone, and few gates can do that.Even at 10 MHz we’re talking 10 ma.74HCT gates, for instance, are generallyrated at 4 ma. Other parts, notablyprocessors, may be capable of drivingless current. TI’s OMAP35xx parts arespec’d at 2 ma on most pins.

In other words, connect one ofthese puppies to your board and thecircuit may very well stop working. Ithink some of those homeless people Isee in Baltimore are engineers whocracked when they found their trou-bleshooting tools made troubleshoot-ing impossible.

The important take-away is this:when buying a probe don’t be seducedby the 10-MΩ rating. Think imped-ance, which is related to tip capaci-tance.

(A side note for those working withanalog. Generally analog signals aren’tas fast as digital so the probe-loading is-sues may be less severe. On the otherhand, many analog nodes have a muchhigher impedance than digital. Even atDC putting a 10-MΩ probe across a 10-MΩ node will zap half the signal).

Better probes exist. Tek’s 4 GHzP7240 has a tip capacitance of only0.85 pF. Running the numbers we get2 KΩ at 100 MHz, which exactly match-es the chart in the device’s manual. Butthis is an active probe, one loaded withelectronics, and it’ll set you back $5,000.Which explains the sign held aloft by

1 2

1 2=

+Z

Z xZ

Z Znet

TPP1000 probe impedance.

Figure 2 Graphic courtesy of Tektronix, Inc. Reproduced with permission.

Equivalent circuit of a scope probe.

Figure 1

Probe tip

Ground

When buying a probedon’t be seduced by the10-MΩ rating. Thinkimpedance, which isrelated to tip capaci-tance.

!!!

break points


one gaunt street person last week: “Pleasehelp. Will work for a pair of P7240s.”

Logic analyzer probes also exhibit ca-pacitive characteristics. Agilent, for in-stance, sells sets that range from 0.7 pF to3 pF per tip. The probe sets for theirMSOs run 8 to 12 pF. Cheapies advertisedon the Internet have higher capacitances,and an astonishing number don’t have arating at all. Never connect a logic analyz-er to a circuit unless you’ve thoughtthrough the probe impedance issues.

FOURIER FITSBut it gets worse.

In 1822, Joseph Fourier released hisThéorie analytique de la chaleur (The Ana-lytic Theory of heat). That seminal workhas tormented generations of electricalengineering students (and no doubt oth-ers). Discontinuous functions—likesquare waves—are very resistant to mathematical analysiswith calculus unless one does horrible things like use unitstep functions. But Fourier showed that one can representmany of these periodic functions as the sum of sine waves ofdifferent amplitudes and frequencies. The Fourier Series for asquare wave is:

The series goes on forever, so all square waves have fre-quency components going to infinity. However, the ampli-tude of these decrease rapidly due to the division by an ever-larger odd number. The point, though, is that the“frequency” of a square wave is composed of many frequen-cies higher than that of the baseband.

Pulses, like the ones that race around every digital board,the ones we probe with our scopes and logic analyzers, aresquare-wave-ish. The good news is that they’re not perfectsquare waves: obviously, with the exception of clocks, theyrarely have a 50% duty cycle. Pulses are also, happily, imper-fect. Fourier’s analysis assumed that the signal transitions be-tween zero and one instantaneously. In the real world everypulse has a finite rise and fall time. If Tr is the rise time, thenthe frequency components above F in the following formulawill be so far down they’re not important:

This does mean that, assuming a 1-nsec rise time, even ifyour clock is ticking along at a leisurely rate about the same

as a Florida old-timer’s speedome-ter, the signals have significant fre-quencies up to 500 MHz.

Those unseen but very real fre-quency components will interactwith the scope probe.

AD HOC FORMULASLong troubleshooting sessions oftensee a board covered with connec-tions to test equipment, data loggers,etc. Long lengths of wire-wrap wireget soldered between a node and aninstrument. These connections allchange the AC properties of thenodes by adding inductance and ca-pacitance. Here are some useful for-mulas with which one can estimatethe effects. These are derived frombook High-Speed Digital Design(Howard Johnson and Martin Gra-

ham, 1993 PTR Prentice-Hall Inc, Englewood Cliffs, NJ), andthere’s much more useful data in that book.

Most of us use multilayer PCBs that have one or moreground and power planes. Solder a wire to a node and drapeit across the board as it runs to a scope or other instrument,and you’ll add capacitance. If d is the diameter of the wire ininches, h is the height above the PCB, and l is the length ofthe wire, then the capacitance in pF is:

(A better solution is to run the wire straight up from thenode, perpendicular to the PCB.)

AWG 30 wire-wrap wire is 0.0124 inches in diameter.Typical hook-up wires are AWG 20 (0.036 inches), AWG 22(0.028 inches), and AWG 24 (0.022 inches).

The inductance of the same wire in nanohenries is:

The inductance of a round loop of wire (for example ascope probe’s ground lead) in nH is, if d is the diameter ofthe wire and x is the diameter of the loop:

Next month we’ll look at some real-world data. ■

( )4

(sin(2 )1

3sin(6 )

1

5sin(10 ) ...

ππ π π= + + +f t ft ft ft

0.5=F

Tr

1.4

ln(4

)=C

lh

d

5.08 ln(4

)=Lh

d

15.6 (ln(8

) 2)= −H xx

d

Never connect a logicanalyzer to a circuitunless you’ve thoughtthrough the probe imped-ance issues.But it gets worse . . .

!!!

Get current, without the hassle.

Design News is a registered trademark of UBM plc © 2012

REGISTER AT

DesignNews.com/dkcec

With Digi-Key’s Continuing Education Center, catch up to 180 days of free,

interactive courses engineered by Design News, starting January 2012.

Build your engineering know how. Take one or all. It’s your choice.

Tracks include: Microcontrollers Basic, Microcontrollers Advanced, MEMS Sensor

Technology, Lighting, Embedded Internet and Cloud, Wirelesss, and Power.

@DigiKeyCEC

Digi-Key Continuing Education Center

FOLLOW US

A full featured development solution

for all ARM Powered® platforms.

1-800-348-8051www.arm.com/ds5

Leading EmbeddedDevelopment Tools

Cmp Esd0312

Documents

ee times group

safety integrity levels

green hills software

embedded systems design

object code verification

single function call

shape const largest

shapes kind member