Understanding IEC61508 Annex D Final - exida

DeterminingSoftwareSafety:UnderstandingthePossibilitiesandLimitationsof

InternationalSafetyStandardIEC61508-7AnnexD

JuliaV.Bukowski,PhDDepartmentofElectrical&ComputerEngineering

VillanovaUniversity

January2015

Copyright©exida.comLLC2000-2015

Copyright © exida.com LLC 2000-2015

TABLEOFCONTENTS pageExecutiveSummary 1 Introduction 3 Notation 3 Background 3 “WhiteBox”vs“BlackBox”SWTesting 3 BasicsofBernoulliSamplingandStatisticalAnalysisofResults 4 EstimatingpandtheConceptofConfidenceInterval 4RelatingBlackBoxTestingtotheBernoulliSamplingModel 6

RequiredSWCharacteristics 6SummaryofAssumptionstoThisPoint 8SWOperatingUnderLowDemandConditionswithAllRequiredAssumptionsSatisfied 8 EntriesofTableD.1forLowDemandModeofOperation 8

EffectsonTableD.1oftheSizeoftheInputSpaceAssociatedWithSafetyCriticalDemands 9

Caveat 10WhatAnnexDStatesaboutSWTestingforLowDemandModeofOperation 10 StatisticalTesting 10 UseofOperationalExperience 11 UsingOperationalExperiencetoSupplementStatisticalTesting13 UsingOperationalExperiencetoReplaceStatisticalTesting13WhatAnnexDStatesaboutSWTestingforHigh/ContinuousDemandModeofOperation 14 AssumptionsRequiredbyAnnexD 14ConceptofEquivalentPFDforHigh/ContinuousDemandModeofOperation 15References 20Appendix 21

Copyright © exida.com LLC 2000-2015 Page1

ExecutiveSummary

Thisdocumentis intendedforreaderswhoarefamiliarwiththeinternationalsafetystandardIEC61508[Ref.1]ingeneralandwiththatdocument’sPart7:AnnexD[Ref.2]inparticular.Ascurrentlywritten,AnnexDprovides“initialguidelinesontheuseofaprobabilisticapproachtodeterminingsafetyintegrityforpre-developedsoftware”(SW)includedinsafetyinstrumentedfunctions.Itfurtherstatesthat“theannexprovidesanindicationofwhatispossible,butthetechniques shouldbeusedonlyby thosewhoarecompetent in statisticalanalysis.” If theseguidelinesare tobeusedeffectively in the testingandcertificationof safety-relatedSW it isessential that individuals involved in testing and certifying such SW understand how tointerprettheseguidelinescorrectly. Tothisend,thisdocumentexplainsthepossibilitiesandlimitationsinherentintheinformationcontainedinIEC61508-7AnnexD.Thefollowingconclusionsarereached:• UsingAnnexD,itisnotpossibletoprovidesafetycertificationforSWthatcontainsahigh

probabilityofhavinginternalstates.Furthermore,forcertifiedSW,itistheresponsibilityofthe SW tester to document exactly how it was determined that the SW contained nointernal states. This is very difficult with “black box” testing only. If the SW tester isunaware of the presence of internal states, the consequence is that the SIL level, andassociatedconfidenceintervalandconfidencelevelwillbeoverestimated.

• Theentries inAnnexDTableD.1arebasedonaBernoullisamplingmodelandnumerousassumptionsmust bemet for the entries to be used in a validway. In particular, inputsamplestotheSWmustbeindependentanduniformlydistributed.Thisistruebothforoff-line testingwhere the SW inputs are randomly generated to be statistically uniform andindependentandfortestingclaimedastheresultofoperationalexperience.Ifthesampledinputs are not independent and uniformly distributed, the SIL level, and associatedconfidenceintervalandconfidencelevelwillbeoverestimated.

• It is importantfortheSWtestertoproperlyinterprettheinputspacefromwhichrandomsamplesaretobedrawnbothuniformlyandindependently.TherequiredinputspaceisasubsetoftheentireSWinputspace.Therequiredinputspaceconsistsofallcombinationsofexternalvalueswhichformthe inputsassociatedwithanysafetycriticaldemand. Alsonotethattheinputspacetobesampledmaybeapplicationspecific.(Pleaseseepage7foranexample.)

• The use of operational experience to certify SW for safety applications requires manysignificantassumptionsthataredifficulttosatisfy.Inparticular,theSWtestermusthaveacomplete record of all inputs associated with the safety critical demands (and theircorrespondingoutputs) thatoccurredduring theoperationalobservationalperiod. Thesemustthenbetestedfor independenceanduniformityofdistribution,a taskthatrequires


significant knowledge of statistical theory. Furthermore, such records are generally notavailable.

• Failure-free operating hours alone are not sufficient for SW certification under anycircumstances.Theremustalsobeaclearrecordofthenumberofsafetycriticaldemands(and consequently the number of inputs associated with safety critical demands) thatoccurredduring theoperationalobservationperiod. Theconsequenceof trying tocertifySWwithoutknowingthenumberofsafetycriticaldemandsintheobservationperiodisthattheSILlevelandassociatedconfidenceintervalandconfidencelevelwillbeoverestimated.Forlowdemandmodeoperation,statistical(off-line)testingisverylikelytobeeasierandmoreefficientthantheuseofoperationalexperiencedata.

• For high/continuous demand mode operation, statistical (off-line) testing where therequired number of tests is calculated based on an equivalent probability of failure ondemand (PFD) is, again, very likely to be easier and more efficient than the use ofoperationalexperiencedata.

• Insomecases,thenumberofinputs,n,requiredtoshowtheSWhasattainedaparticularSIL level exceeds thenumberof inputs associatedwith any safety critical demandand inthesecases,exhaustivetestingoftheappropriateinputspaceismoreefficientthanrandom“blackbox”testing.


INTRODUCTIONInternationalsafetystandardIEC61508-7AnnexDTableD.1summarizesthe“necessaryhistoryfor confidence to safety integrity levels” for pre-developed SW. Understanding how thecontentsofTableD.1arearrivedatisessentialtounderstandingwhenTableD.1canbeappliedandwhenitcannot.ThisdocumentprovidesdetailedanalysisofthecontentsofAnnexDwithemphasisontheassumptionsrequiredtoapplytheinformationinTableD.1inavalidmanner.NOTATIONhr(s) hour(s)HW hardwarej numberofmembersofthetotalpopulationhavingacertaincharacteristick numberofsampleshavingacertaincharacteristicn totalnumberofsamplesN totalpopulationfromwhichasampleistakenNe totalpopulationofSWinputsthatarecombinationsofexternalinformationvaluesonlyp j/N,i.e.,thetrueproportionofthetotalpopulationhavingacertaincharacteristicp* statisticalestimateofpbasedonnandkp*L lowerlimitoftheconfidenceintervalconstructedstatisticallyaroundpp*U upperlimitoftheconfidenceintervalconstructedstatisticallyaroundpPFD probabilityoffailureondemandPFD/hrprobabilityofdangerousfailureondemandperhour;afailureratesec secondSIL safetyintegritylevelSW software[x,y) anymemberoftheinterval,sayz,issuchthatx<z<y[x,y] anymemberoftheinterval,sayz,issuchthatx<z<yα parameter(usually0.05or0.01)specifyingthe(1-α)x100%confidencelevelofthe

statisticallyconstructedconfidenceintervalboundedby[p*L,p*U]BACKGROUND“WhiteBox”vs“BlackBox”SWTesting“White box” testing occurs when the SW source code and other SW design and testingdocumentation are available to the SW tester. Test inputs are chosen specifically to forcecertaincodetoexecutesothatitsoutputcanbeobservedandevaluatedagainstthefunctionaldesignspecifications. “Whitebox”testingisperformedinstages: initialtestingoccursattheunit and module levels; intermediate testing phases occur at higher levels as modules areintegrated into larger functional entities; final testing occurs with the entire SW packageintegrated in the target hardware (HW) environment andmay include purposefully selectedinputs,randomlyselectedinputs,unanticipatedinputsandinputsassociatedwithabusiveuse


oftheSW.“Whitebox”testingisthepreferredformoftestingandistheformusedbytheSWdeveloper.When“whitebox”testingisimpractical–oftenbecausetheSWtesterisnottheSWdeveloperand does not have access to the SW source code and other relevant SW design and testdocumentation – “black box” testing is performed on the entire SW package. “Black box”testing consists of selecting and presenting input test cases to the SW, evaluating thecorrectnessofthecorrespondingoutputs,andusingtheresultstoinferthecorrectnessoftheSWasawhole.AnnexDaddressestheissueof“blackboxtesting.”InorderfortheSWtesterto use Table D.1 in Annex D correctly, the SW tester must understand the required SWcharacteristicsandthemathematicalassumptionsusedintheproductionofTableD.1.BasicsofBernoulliSamplingandStatisticalAnalysisofResultsAssumethatavery largebox is filledwithNpingpongballs. Furtherassumethat jballsarecoloredredandtheremainingN-jballsarecoloredgreen.Onemightliketoknowthevalueofp=j/N.Forexample,onemightask“Isp=0.001?;”inotherwords,dotheredballsaccountfor1per 1,000of theballs in theurn so that if a ballwere selectedat random from thebox itwould have a probability of 0.001 of being red? The straightforward way to answer thisquestionwouldbetoexamineallNballs,countthenumber, j,ofredballsandthenformthefractionj/N=p.Withthisapproachthequestioncouldbeansweredwithcompletecertainty.However,theremaybegoodreasonswhythisapproachisimpractical.Forexample,ifapricemustbepaidforeachballthatisexaminedandNislarge,thecostofexaminingeveryballmaybeprohibitive.Soonemayask,canthisquestionbeanswered,notwithcompletecertaintybutwithahighdegreeofcertainty,bysamplingonlynballswherenislessthanN?Thisisaclassicprobleminstatisticsandlikeallmathematicalproblemsneedstobespelledoutclearly.Firstassumethatwhenonesampleistaken,eachballhasthesameprobability,1/N,ofbeingselected; in probability terminology this means that the sample is drawn with a uniformdistribution.Nextassumethatthesamplesareindependent,whichisthesameassayingthatafteraballissampleditisreturnedtotheboxanditmaybesampledagain.Thismayseemlikean unusual assumption since in a real physical situation it may appear that there is anadvantagetoputtingthesampledballaside,butthisassumptionmakesthemathematicsmucheasier and does not make a significant difference in the final answer provided N is large.Supposethisprocess iscontinueduntilnsampleshavebeentakenandthenumber,k,ofredballssamplediscounted.Underthegivenassumptions,nBernoullitrialshavebeenperformedandananswer to theoriginalquestion canbegin tobe formedbasedon thevalueofk, thenumberofredballsobservedinnsamples.


EstimatingpandtheConceptofConfidenceIntervalSupposethat,infact,kisgreaterthan0indicatingthatkredballswereselectedamongthensamples.Thenitiseasytoestimatethevalueofpasp*where p*=k/n. (1)Further,aconfidenceintervalcanbeconstructedaroundp*withlowerandupperconfidencelimitsp*Landp*U,respectively. (Note: Theusualformulasforcomputingp*Landp*U(seeforexample [Ref.3]p.327)cannotbeused forverysmallp*andaWilsonscore interval [Ref.4]shouldbecomputedinstead).Iftheconfidencelevelis95%thismeansthatiftheexperimentof selecting uniformly and independently n balls, counting the number of red ones, andestimatinganewp*,p*Landp*Ueachtime,wererepeatedmanytimesthenthetruevalueofpwould be captured by 95% of the statistically constructed intervals and the true value of pwould lieoutside theboundsof 5%of the statistically constructed intervals. Sincegenerallyonlyoneestimateandone interval iscomputed it isnotpossibletoknowforsure if thetruevaluep lieswithintheboundsofthesinglestatisticallyconstructedintervalornot,butoneis95%confidentthatitdoes.Thesituationisdifferentifkequals0meaningthatnoredballsappearedinthensamples.Inthiscase,p*cannotbeobtainedfromEq.(1).Butaconfidenceintervalcanbeconstructedthatlikelyincludesthetruevalueofp. Duetothepreviousassumptions,theprobabilitythatallnsamplesaregreenballs,i.e.,theprobabilitythatk=0,isgivenbythebinomialdistributionfortheparametersnandpandevaluatedfork=0.Specifically,theprobabilitythatk=0is P(k=0|n,p)=(1-p)n. (2)Itispossiblesincek=0that,infact,p=0,i.e.,thattherearenoredballsinthebox,sologically,the lower bound of the confidence interval, p*L, is 0. The upper bound of the confidenceintervalisfoundbysolving P(k=0|n,p*U)=(1-p*U)n=α (3)for p*U where p*U is the upper bound of the (1 - α)x100% confidence interval and n is thenumberofballssampled.Generally,αissetequalto0.05givinga95%confidenceintervalorαis set equal to 0.01 giving a 99% confidence interval. However, α can be set to any valuebetween0and1.NotethatEq.(3)containsthreeparametersandanyoneofthemcanbefoundintermsoftheother two. Table 1 summarizes the results of holding twoparameters in Eq. (3) fixed andsolvingforthethird.


Table1.VariationsonEquation(3)FixedParameters SolutionforRemainingParameter Equation

p*U&n

α=(1-p*U)n

α≈exp(-np*U)

(3)

(3a)1

n&α p*U=1-α(1/n) (3b)

p*U&αn=(lnα)/ln(1–p*U)

n≈-(lnα)/p*U

(3c)

(3d)21FromtheapproximationforninEq.(3d),multiplyingeachsidebyp*UandtakingtheexponentialofeachsidegivesEq.(3a).2Sinceln(1–p*U)≈-p*Uforp*U<0.1theapproximationforngiveninEq.(3d)followsdirectlyfromEq.(3c).RELATINGBLACKBOXTESTINGTOTHEBERNOULLISAMPLINGMODELIn “black box” testing the SW tester presents a variety of inputs to the SW to producecorresponding outputs. These outputs are then evaluated for correctness and the overallcorrectnessoftheSWisinferred.Thequestionofinterestforsafetypurposesis“whatistheprobability thatwhena safety critical demandoccurs the input (to the SW) associatedwiththatsafetycriticaldemandwillresultinanincorrectoutput,i.e.,thatthesafetycriticaldemandisnotproperlyhandledduetoaSWfailure?”AllpossibleSWinputsassociatedwithanysafetycriticaldemandconditionsareanalogoustotheNpingpongballsinthebox.TheinputswhicharecorrectlyprocessedbytheSWareanalogousto thegreenballsandthe inputswhichareincorrectlyprocessedbytheSWareanalogoustotheredballs.ButthissimpleanalogybetweenacorrectSWinputandasinglegreenpingpongballmayleavetheimpressionthataSWinputconsistsofjustonepieceofinformation.Infact,thesituationismore complicated. Thus, some further requirementsmust be placed on the SW before theBernoullisamplingmodel(andhencetheinformationinAnnexD)canbeapplied.RequiredSWCharacteristicsSafetySWnormallytransformsavarietyofinformationfromdifferentsources,bothexternaltothe SWandpotentially internal to the SW, into anoutput response. External information isgeneratedoutsidethe“blackbox”andisthereforegenerallyobservable,andlikelycontrollable,by the SW tester in simulated conditions. External information consists of such items asinstrument set points, temperature or pressure measurements, etc. Internal information iscontainedinsidethe“blackbox”andisthereforeneitherobservablenorcontrollablebytheSWtester. Internal informationconsistsofsuchitemsasthecurrentvalueofaninternalcounterwhichmaytakeitsvaluenotjustfromthecurrentexternalinformationbutfromasequenceofprior external information or other internal information. Another source of internalinformationmaybeparameterswithintheSWsetting itsmodeofoperation,e.g.,debugandmaintenancemodeofoperation.WhileitisunlikelythatanexternalSWinputwouldaccidentlyactivate the SW’s debug and maintenance mode, it is entirely possible for a human to


appropriately activate the SW’s debug and maintenance mode for a specific task and thenforgettocorrectlyexitthatmodeandreturntheSWtoitsproperoperationalmode.One SW input in the above analogy corresponds to a particular combination of all possibleinformationvalues(bothexternalandinternal)thatmayexistbecauseasafetycriticaldemandoccurs. ConsiderthefollowingsimpleexampleinvolvingasingleinternalSWstate. Suppose,allthepossiblecombinationsofexternalinformationvaluesassociatedwiththeexistenceofasafetycriticaldemandtotaltheintegervalueNe.AssumethattheSWcontainsaninternalstatethatmayassumeoneofthreevalues.Thenthetotalnumber,N,ofSWinputstatesassociatedwithasafetycriticaldemandisN=3xNe.NowiftheinternalstateremainsatitsfirstpossiblevalueduringalloftheSWtesting,theninputsamplesweredrawnfromonlyone-thirdofthetotal input spaceof sizeN associatedwith a safety critical demandand the assumption thatsampling occurred uniformly from the entire input space associated with safety criticaldemandsisnotmet.Ifaninternalstateispresent,sinceaninternalstateisunobservable,onemust suppose that the assumption of uniform, independent sampling over the entire inputspace associated with the presence of a safety critical demand will not be met in general.Consequently,under currently developed theory it is not possible to certify SW containinginternalstatesbasedonTableD.1;thisappliesbothtolowdemandmodeofoperationandtohighdemandorcontinuousmodeofoperation.Toprovethata“blackbox”containsinternalstates,itissufficienttofindoneinputconsistingofaparticularcombinationofexternalvalueswhichwhenprocessedrepeatedlygivesdifferentoutputsatleastsomeofthetime.Theexistenceofdifferentoutputsforthesamecombinationof external information values points to the existence of an internal SW state. However, itshould be noted that determining that a “black box” contains no internal states is a verydifficultproblemifbasedon“blackbox”testingalone.RecalltherequirementthatthenSWinputsamplesbeselecteduniformlyandindependentlyfromthetotalpopulationofNinputsassociatedwiththeexistenceofasafetycriticaldemand.ThisassumesthattheSWtestercancorrectlyidentifyallinputsassociatedwiththeexistenceofa safetycriticaldemand. In somecases, this identificationmaybe relatively straightforward.However,considerthecaseofasafetycriticaldemandduetoa“pressureevent”.Dependingon the systemconfiguration, thedemandmaybeapressurewhich is toohighorapressurethat istoolow. Usually,theSWwillhaveaconfigurationparameterthatrecognizeswhetherthesafetycriticaldemandisconsideredhighpressureor lowpressure. FromtheSWtesters’viewpoint,theinputspaceassociatedwithasafetycriticaldemandwillchangedependingonwhether the system configuration presents demands which are high pressure or demandswhicharelowpressure.Sincetheinputspaceassociatedwiththeexistenceofasafetycriticaldemandcanchange,SWtestingandcertificationneedstobeapplicationspecific!Thefocusofthesamplingisonthecorrectnessor incorrectnessoftheinputsassociatedwithsafetycriticaldemands.ItmayseemtothereaderthatthefocusshouldbeonthecorrectnessorincorrectnessoftheSWitself.However,in“blackbox”testing,thereisnowaytoguaranteethattheSWisactuallycorrectlyprocessingtheinputs. It ispossible(thoughnot likelyovera


large input space) that the internal algorithms are incorrect but by chance produce theexpectedoutput foreach input. Therefore, it isonlypossible in“blackbox”testingtoassertstatisticallythatthereisahighprobabilitythatinputswillcausethecorrectoutputstoappear.Whilethefocusofthesamplingisonthecorrectnessorincorrectnessoftheinputsassociatedwith safety critical demands, it is really the outputs corresponding to those inputs thatdeterminethecorrectnessoftheinputs.Therefore,theremustbeanappropriatemechanismfor recognizing all correct outputs as “correct” and all incorrect outputs as “incorrect.”Labellingacorrectoutputas“incorrect”willcausethecount,k,ofincorrectinputstobegreaterthan its true valuewhichwould be a conservative error from a safety viewpoint. However,labellinganincorrectoutputas“correct”meansthatthecount,k,of incorrectinputsinthensamples is artificially low and, perhaps, k will equal 0 when it should not. This gives falseconfidenceinthesafetyoftheSW.Furthermore,thefailuretorecognizeanincorrectoutputas“incorrect”meansthatthereexistsadangerousundetectedfailureintheSW.SUMMARYOFASSUMPTIONSTOTHISPOINTInorderfor“blackbox”testingperAnnexDtobevalidthefollowingassumptionsmustbemet:

• TheSWbeingtestedcannotcontainanyinternalstates,i.e.,outputsmustdependsolelyonexternalinformationwhichiscompletelyobservableandcontrollable.

• TheSWtestermustbeable todocument themethodsused to conclude that theSWcontainsnointernalstates.

• The SW tester can correctly identify all combinationsof all external values associatedwiththepresenceofasafetycriticaldemandanddistinguishdifferentinputspacesfordifferentapplicationsofthesameSW.

• Theremustbeanadequatemechanismtocorrectlyidentifytheoutputaseithercorrectorincorrect.

• ThenSWinputsmustbesampleduniformlyand independently fromtheentirespaceassociatedwithallexternalvaluesthatmayexist in thepresenceofanysafetycriticaldemand.

• Amongthensampleinputs,noneareidentifiedasbeingincorrect,i.e.,k=0.


The remainder of this document addresses SW which produces outputs based solely onexternalinformation,i.e.,addressesSWthatcontainsnointernalstates.SWOPERATINGUNDERLOWDEMANDCONDITIONSWITHALLREQUIREDASSUMPTIONSSATISFIEDEntriesofTableD.1forLowDemandModeofOperationForSWmeetingalloftheaboveassumptions,theentriesinAnnexDTableD.1forLowDemandOperationarearrivedatasfollows.RecallEq.(3d)fromTable1abovewhichisrepeatedhereforconvenience:

n≈-(lnα)/p*U. (3d)

Forα=0.05,lnα=-2.9958≈-3andforα=0.01,lnα=-4.6052≈-4.6.So,forexample,for(1-α)x100%=95%andp*U=10-4whichisthelowerlimitforSIL3underconditionsoflowdemandoperation,n=3x104.Similarlyfor(1–α)x100%=99%andp*U=10-4,n=4.6x104.OtherentriesforthelowdemandmodeforotherSILlevelsaresimilarlycomputedandarepresentedinTable2below.

TABLE2.NumberofSWtestinputsassociatedwiththeexistenceofasafetycriticaldemandrequiredtoachieve95%or99%confidenceforvariousSILlevels

SIL

PFD

n,numberofuniformlydistributedinputsassociatedwiththeexistenceofanysafetycriticaldemand

α=0.05;ConfidenceLevel=95% α=0.01;ConfidenceLevel=99%1 [10-2,10-1) 300 4602 [10-3,10-2) 3,000 4,6003 [10-4,10-3) 30,000 46,0004 [10-5,10-4) 300,000 460,000What thisexamplesmeans is, ifall30,000 inputs selected independentlyanduniformly fromtheentireSWinputspaceassociatedwiththeexistenceofanysafetycriticaldemandproducecorrectoutputswhenprocessed,thenitcanbestatedwith95%confidencethatthetruevalueofp,thePFD,iscontainedintheinterval[0,10-4].AnnexDclassifiesthisasproofofattainingSIL3.Notethatevenwith46,000testsresultinga99%levelofconfidencethatpiscontainedintheinterval[0,10-4],itisstillpossiblethatpisactuallylargerthan10-4;hence,itisSIL3whichisattained, not SIL 4. The reader who wishes to explore further the concept of confidenceintervalas itpertains toSIL levels is referredtoAppendixAof thisdocumentwhichcontainsmoredetails.It is important for the reader to realize that while it is easy to simulate inputs which areindependent and uniformly distributed it is much more difficult to insure that inputs fromoperatingexperienceareindependentanduniformlydistributed.Thistopicisaddressedlaterinthisdocumentbothforlowdemandandhigh/continuousdemandmodesofoperation.


EffectsonTableD.1oftheSizeoftheInputSpaceAssociatedWithSafetyCriticalDemandsA few final observations are important. The readermay notice that the number, n, of testinputs requiredbyTableD.1 is independentofN, thesizeof the inputspaceassociatedwithsafetycriticaldemandconditions.Thismayseemsintuitivelyincorrectsinceafixednumber,n,of inputs will cover a smaller and smaller portion of the entire input space associated withsafetycriticaldemandconditionsasNincreases.However,recalltheassumptionthatp=j/N,wherejisthenumberincorrectinputs.Undertheassumptionsofthemathematicaltheory,ifpremains thesamethenasN increases, jalso increasesproportionately. Therefore,while thenumberoftests,n,remainsfixed,increasingthenumberofincorrectinputs,j,proportionaltothe increase in thenumber,N,of totalpossible inputsassociatedwithsafetycriticaldemandconditions means that the fixed number of tests, n, has the same probability of finding anincorrectinput.ThisisanappealingfeatureofthestatisticsinTableD.1asforsomeSILlevelsitplacesrelativelysmalllimitsonthenumberofinputswhichmustbetested.Whymight thenumberof inputs associatedwith safety criticaldemandconditions increase?ManyoftheexternalvaluesthatserveasinputstotheSWareactuallyanalogsignalswhicharediscretized.Ifaparticularrangeofvaluesassociatedwithasafetycriticaldemandisdiscretizedtofivelevelsandlater,discretizedmorefinelyto10levels,thenthesizeNoftheinputspaceassociatedwith safety critical demandswill double. If every external value in that particularrange was associated with a safety critical demand, and every external value in two of theoriginalrangeswasassociatedwithanincorrect input,thenwhenNdoubled, sodid j.Undertheseconditionsthemathematicalassumptionwouldbesatisfied.However,itisalsopossiblethat it isnotonlyarangeofvalueswhichcreates incorrect inputs,butalsoaboundaryvaluebetweentworanges.Inthiscase,doublingthenumberofdiscretizedlevelswillnotdoublethenumber of incorrect inputs. However, j will be slightly smaller than expected, so p will beslightly smaller than before the finer discretizing and, n, the number of sample inputsassociatedwithsafetycriticaldemandconditionswillbeslightlylargerthanisrequiredbyTableD.1 for a given levelof confidence. Thus, theassumption that thenumberof correct inputsincreasesproportionallywith increases in the size,N, of inputsassociatedwith safety criticaldemandconditionsisnotacriticalone.Itshouldalsobenotedherethattheremaybecaseswherethesizeoftheentireinputspaceassociatedwith safety critical demands is smaller than the required number of test cases inTableD.1 for a given SIL level. In such cases, it is clearlymore efficient to exhaustively testevery possible input associated with the existence of a safety critical demand and havecompleteconfidenceintheresult.


CaveatAllof theabove is true for lowdemandconditionsprovidedall theunderlyingassumptionsaremet.WHATANNEXDSTATESABOUTSWTESTINGFORLOWDEMANDMODEOFOPERATIONForthelowdemandmodeofoperation,AnnexDdiscussesbothstatisticaltestingasdescribedabove and the use of operational experience to determine which SIL levels the SW meets.Thesearereviewedbelow.StatisticalTestingAnnexDsectionD.2.2allowsforthetypeofstatistical(off-line)testingdescribedaboveforlowdemandmodeofoperation. SectionD.2.2placesfewerspecificprerequisitesonsuchtestingthan have been described previously. Specifically, Annex D Section D.2.2 states “The onlyprerequisite is that the test data is selected to give a random uniform distribution over theinputspace(domain).”Unfortunately,thissingleprerequisiteisinsufficienttoguaranteethattheBernoullimodelunderlyingTableD.1 is infactsatisfied. Ofspecialnote istheuseoftheterm“overtheinputspace(domain)”whichcouldeasilybemisinterpretedtomeantheentireinputspaceoftheSWasopposetothecorrectrequirementthatsamplingbefromtheentireSWinputspaceassociatedwiththeexistenceofanysafetycriticaldemand.ThislatterinputspaceisasubsetoftheentireSWinputspaceandasubsetthatmaychangewiththespecificapplicationoftheSW.This isnot to imply that theauthorsofAnnexDareunawareof thecorrect requirements; itmerely suggests that the readernotwell versed in statisticalanalysismayeasilymisinterpretwhatiswritten.ItissuggestedthatthisstatementinAnnexDberevisedtobemorespecificsoastominimizethepossibilitiesofmisinterpretation. TheBernoullisamplingdescribedcanbeimplemented if the SW tester has ameans of randomly generating off-line combinations ofexternalvaluesassociatedwithsafetycriticaldemands,recordingthesevalues,havingtheSWprocess the inputs, recording the corresponding outputs, determining the correctness of theoutputs,andcountingkthenumberofcorrectinputsoutofninputs.


UseofOperationalExperienceAnnexDalsostatesthat:“Operating experience canbe treatedmathematically… to supplement or replace statisticaltesting, and operating experience from several sites may be combined (i.e., by adding thenumberoftreateddemands…),butonlyif

- [1] the software version to be used in the [Electrical/Electronic/ProgrammableElectronic]E/E/PEsafety-relatedsystemisidenticaltotheversionforwhichoperatingexperienceisbeingclaimed;

- [2]theoperationprofileoftheinputspaceissimilar;- [3]thereisaneffectivesystemforreportinganddocumentingfailures;and- [4]therelevantprerequisites[foroperationalexperience]aresatisfied.”

Theseassumptionsrequiresomeclarificationsandadditionalcomments.Regarding Assumption [1]: Clearly this assumption needs to be met. However, it is alsoimportanttonotethatnotonlymusttheSWversionbethesameatmultiplesitesbutalsothat:

• anysubsequentSWupdatesandpatchesinstalledtotheSWatonesitemustalsohavebeeninstalledtotheSWatallsitesatapproximatelythesametime,and

• any SW customization parameters be identically set at all sites throughout thetimeframe fromwhich operational experience is collected so as to guarantee that aninputassociatedwithsafetycriticaldemandconditionsinoneoperationalsettingisalsoan input associated with safety critical demand conditions in another operationalsetting.

Verifyingthesefactsoverseveralsitesislikelytobeaformidabletask.Regarding Assumption [2]: If the operational profile in any way limits the subset of inputsassociatedwith safety critical demands thatwill be observed, i.e., if a particular operationalprofileismorelikelytoexperiencecertainsafetycriticaldemandconditionsandnotexperienceothersafetycriticaldemandconditions,thanthesampleofinputsassociatedwithsafetycriticaldemandswill not beuniformover that entire subset of inputs associatedwith safety criticaldemands. To require that the operating profiles be similar between sites means that thecumulative samples will have similar non-uniform distributions and will not fulfill theassumptionofuniformsamplingasrequired.Generally, if an operating profile encompasses small to medium size deviations around anoperatingpoint,itislikelythatthedistributionofinputsassociatedwithsafetycriticaldemandswhich are sampled will not come from all possible inputs associated with safety criticaldemands. Furthermore, ifanoperatingprofilecreatesclustersof repeated inputsassociatedwithsafetycriticaldemands inthesample,then ifk=0,theconfidence levelassociatedwiththen tests isartificiallyhighbecauseit isnotbasedonnuniformlydistributesamples. Thus,whenoperationalexperienceistobecollected,itisimportantthattherebeacompleterecordof every combination of external values that, in fact, created a SW input associated with a


safetycriticaldemand,alongwiththecorrespondingoutput.Intheauthor's’experience,suchdetailed input/outputdata isoftennotavailable. However, if such information is availablethen this total collection of actual inputs must be analyzed to see if they do form anindependent, uniform sample of size n. An example of an approach to testing for uniformdistributionsof inputs is given in [Ref.3pp.397-400]under the topicGeneralizedLikelihoodRatio. The reader is cautioned that the proper use of such techniques requires a significantunderstandingofstatisticaltheory. AnotherreasonnottolimittheoperationalprofileisthatassessmentofSWsafetyachievedwouldonlybeassignabletothatprofile.OperatingprofilesmaychangeovertimeandtheSWsafetyassessmentwouldhavetobeperformedagain.RegardingAssumption[3]:ClearlythisassumptionneedstobemetandtheSWtestershouldclearlydocumentevidenceusedtoshowthattheassumptionwasindeedmet.RegardingAssumption[4]:WithrespecttooperationalexperiencethefollowingprerequisitesarelistedinAnnexD:“a)Testdatadistributionequaltodistributionfordemandsduringon-lineoperation.b)Testrunsarestatisticallyindependentfromeachother,withrespecttothecauseofafailure.c)Anadequatemechanismexiststodetectanyfailureswhichmayoccur.d)Numberoftestcasesn>100.e)Nofailureoccursduringthentestcases.”RegardingAssumptiona): Requiringthatthetestdatadistributionequalthedistributionfordemandsduringon-lineoperationlimitsthesubspaceofinputsassociatedwithdemandsthatisbeing sampled and likely violates the uniform distribution assumption required to use theentries in Table D.1. A consequence of failing to meet the uniform distribution samplingrequirementisthat,ifoperationalexperienceisusedwithanon-uniformdistributionofinputsassociatedwithsafetycriticaldemands,thentheSILlevel,itscorrespondingconfidenceintervalandconfidencelevelareoverestimated.ThisassumptionrelatestoAssumption[2]aboveandthosecommentsaboveapplyhereequally.Thereaderisencouragedtoreviewtheparagraphs“RegardingAssumption[2]”onpage12.Regarding Assumptions b), c) and e): These assumptions are already included in theassumptionsinitiallystatedinthisdocumentandrequirenofurthercomment.RegardingAssumptiond):Thisassumption,whichstatesthatnneedonlybegreaterthan100(implying that101 samplesof inputsassociated with safety criticaldemands is sufficient), isincorrectbasedonthestatisticaltheoryofsamplingwhichformsthefoundationfortheentriesin Annex D. The number, n, of independent uniformly distributed inputs associated withsafetycriticaldemandsmustbe thesameasgiven inAnnexDbySIL level for lowdemandoperation.


UsingOperationalExperiencetoSupplementStatisticalTestingIt is not clear to the author why a SW tester would want to use operating experience tosupplement statistical testing in the lowdemandmodeofoperation. Oncea random inputgenerator is constructed to drive off-line SW testing, it is easier and quicker to generatesufficient random tests tomeet the requirementsof statistical analysis than it is toestablishthatoperatingexperiencefulfillssignificantassumptions(includingtheuniformdistributionofinputs) and then gather the combinations of external values that formed inputs that wereactuallygeneratedbysafetycriticaldemandsaswellasthecorrespondingoutputsandanalyzethem.Further,doingsoassumestheoperatingcompanyhascompleterecordsoftheseeventswhich,asmentionedabove,intheauthor’sexperiences,isoftennotthecase.UsingOperationExperiencetoReplaceStatisticalTestingLow demand operation is defined to be notmore than one safety critical demand per year.Some industrial processes have safety critical demandson theorder of once in five years orevenoncein10years.Tocollectdataoversufficientoperationalexperiencewhichmeetstheassumptionsrequiredseemsamuchmoredifficulttaskthantakingthestatisticalapproach.Itistherecommendationoftheauthorthat,if“blackbox”testingistheONLYoption,statisticaltesting (rather thanoperational experience)beused to certify SW for safety applications forlowdemandoperationalmode. However, the reader shouldnote that, ingeneral,otherSWtesting,.e.g.,“whitebox”testing,alongwithotherSWdevelopmentapproachesarepreferredover“blackbox”testing.WHAT ANNEXD STATES ABOUT SW TESTING FORHIGH/CONTINUOUSDEMANDMODEOFOPERATION AssumptionsRequiredbyAnnexDThefourassumptionspreviouslydiscussedasAssumptions[1]–[4]aboveapplyequallytoSWoperationunderhigh/continuousdemandconditionsand the remarksabove regarding thoseassumptionsapplyhereaswell.UndertheheadingofSectionD.2.3SimplestatisticaltestforhighdemandorcontinuousmodeofOperation,AnnexDlistsfiveprerequisitesforstatisticaltestingintheseoperationalmodes;viz.,“a)Testdatadistributionequaltodistributionduringon-lineoperation.b) Therelativereductionfortheprobabilityofnofailure isproportional tothe lengthof theconsideredtimeintervalandconstantotherwise.c)Anadequatemechanismexiststodetectanyfailureswhichmayoccur.d)Thetestextendsoveratesttimet.e)Nofailureoccursduringt.”RegardingAssumptiona): Requiringthatthetestdatadistributionequalthedistributionfordemandsduringon-lineoperationlimitsthesubspaceofinputsassociatedwithdemandsthatis


being sampled and likely violates the uniform distribution assumption required to use theentries in Table D.1. A consequence of failing to meet the uniform distribution samplingrequirementisthat,ifoperationalexperienceisusedwithanon-uniformdistributionofinputsassociatedwithsafetycriticaldemands,thentheSILlevel,itscorrespondingconfidenceintervalandconfidencelevelareoverestimated.ThisassumptionrelatestoAssumption[2]aboveandthosecommentsaboveapplyhereequally.Thereaderisencouragedtoreviewtheparagraphs“RegardingAssumption[2]”onpage12.RegardingAssumptionb):Thisassumptioncaneasilybemisinterpreted.Forexample,ifaSWtesterobserves3x107operatinghourswithnocriticalfailuresthisdoesnotmeanthattheSWtestercancertifytheSWatalevelofSIL2with95%confidence.Failure-freeoperatinghoursalonearenotsufficientforSWcertificationunderanycircumstances. Theremustalsobeaclearrecordofthenumberofsafetycriticaldemands(andconsequentlythenumberofinputsassociated with safety critical demands) that occurred during the operational observationperiod.Furthermore,thenumberofsafetycriticaldemandsrequiredisquitehigh,farinexcessof the n > 100 incorrectly assumed in the case for low demand operation. The number ofrequireddemandsrelatestotheconceptofequivalentPFDforhigh/continuousdemandmodesofoperationwhichisexplainedbelow.Itistheexperienceoftheauthorthattherequiredrecordsofsafetycriticaldemandsareoftennotavailable.TheconsequenceoftryingtocertifySWwithoutknowingthenumberofsafetycritical demands (and consequently the number of inputs associated with safety criticaldemands)intheobservationperiodisthattheSILlevelalongwithitsassociatedconfidenceintervalandconfidence levelwillbeoverestimated. Furthermore,even if the safetycriticaldemandrecordsareavailableitisstillincumbentupontheSWtestertoanalyzedtherecordstosee if thesafetycritical inputsdo forman independent,uniformsampleof sizen. Again,anexampleofanapproachtotestingforuniformdistributionsofinputsisgivenin[Ref.3pp.397-400]underthetopicGeneralizedLikelihoodRatio.Thereaderiscautionedthattheproperuseofsuchtechniquesrequiresasignificantunderstandingofstatisticaltheory.Theneed toknowhowmany inputsassociatedwith safety criticaldemandsoccurredcanbecomparedtotheoriginalsamplingsfromaboxofgreenandredpingpongballsnowextendedasdescribedinthefollowing.Withsamplingtakingplaceaccordingtothedistributionsoftheoperating profile, the inputs potentially come not only come from the space of inputsassociatedwithsafetycriticaldemands(greenandredballs)butalsofromthespaceofinputsnotassociatedwiththesafetycriticaldemandsandthesecanbethoughtofasbluepingpongballs. Nowthesampling is fromaboxwith threecolors. But theunderlyingmodel requiresexactlytwocolors(redandgreen),and,byassumptione),thatoneofthosecolors(red)notbesampled.HencewhensamplingoccursperoperationalprofileitisnecessarythattheSWtesterbeabletoseparatethegreenballsfromtheblueballswhich,ofcourse,requiresthatthetesterknow the number of green balls sampled. Only in this way does the sampling match theunderlyingmathematicalmodelonwhichtheentriesinAnnexDTableD.1arebased.Clearly,ifonlyblueballsweresampled,theSWtestercansaynothingaboutthecorrectnessoftheSWrelativetocriticaldemands.


Theaboveconceptsaresoimportanttheyareworthrepeating: Failure-freeoperatinghoursalonearenotsufficientforSWcertificationunderanycircumstances. Theremustalsobeaclearrecordofthenumberofsafetycriticaldemands(andconsequentlythenumberofinputsassociated with safety critical demands) that occurred during the operational observationperiod. The consequence of trying to certify SW without knowing the number of safetycritical demands in the observation period is that the SIL level along with its associatedconfidenceintervalandconfidencelevelwillbeoverestimated.RegardingAssumptionsc)ande):TheseassumptionsareidenticaltoassumptionsrequiredbytheBernoullisamplingmodel.RegardingAssumptiond):Thisassumptionisobvious.CONCEPTOFEQUIVALENTPFDFORHIGH/CONTINUOUSDEMANDMODEOFOPERATIONWhile it may appear necessary to invoke a continuous time parameter for high/continuousmodesofoperation,infact,allinputsarediscreteintimeandthereneedbenorealdifferenceinthemathematicalmodellingforSWtestingbetweenhigh/continuousdemandmodeandlowdemandmode,otherthantherateatwhichsafetycriticaldemandsoccur.The reader should also be aware that there is a fundamental difference in theway that SILlevelsaredefinedbetweenlowdemandmodeandhigh/continuousdemandmodewhichgivesrisetothedifferencesinthewaySWtestingistreatedforeachmodeinAnnexD.SILlevelsforlowdemandmodearebasedonthePFD, i.e., theprobabilitythattheSWwill fail torespondcorrectlytoanyonesafetycriticaldemand. SIL levelsforhigh/continuousdemandarebasedon the probability of a dangerous failure/hour (PFD/hr). The SIL requirements forhigh/continuousdemandperAnnexDalongwiththerequiredhoursofoperationtoclaimSILlevelattainmentaredisplayedinTable3.


TABLE3.NumberofTotalHoursofOperationrequiredtoachieve95%or99%confidenceforvariousSIL

SIL

PFD/hr

TotalHoursofOperationα=0.05;ConfidenceLevel=95% α=0.01;ConfidenceLevel=99%

1 [10-6,10-5) 3x106=300*104 4.6x106=460*1042 [10-7,10-6) 3x107=3,000*104 4.6x107=4,600*1043 [10-8,10-7) 3x108=30,000*104 4.6x108=46,000*1044 [10-9,10-8) 3x109=300,000*104 4.6x109=460,000*104ComparingTables2and3,notethattheentriesinthevariouscolumnsforSILlevelsand95%and99%confidencelevelsareidenticalexcept:

• for SIL intervals, the ranges in Table 3 are a factor of 10-4 smaller than the ranges inTable2,

• for the amount of required failure-free testing, the number of hours of operation inTable3is104largerthantherequirednumberoftestsinTable2.

AnnexD does not explain the source of this difference. However, for equal SIL levels thereshouldbesomeparitybetweenlowdemandandhigh/continuousdemand.Theauthorofthisdocumenthasthefollowingexplanation.To convert the PFD to PFD/hr requiresmultiplying the PFD by the number of safety criticaldemandsperhour,i.e., PFD/hr=PFDxnumberofdemands/hour. (4)Lowdemandisdefinedasasafetycriticaldemandoccurringnotmorethanonceperyear.Thiswouldbeamaximumdemandrateof1demand/8,760hrs. Ifonegrantedasmall leewaytothis number and accepted themaximumdemand rate to be 1 demand/10,000 hrs then theequivalent PFD/hr for low demandwould be as shown in Table 4 for the various SIL levels.Thus,PFD/hrwouldbethesameforlowdemandandhigh/continuousdemandforeachoftheSILlevels.Foroperationswithdemandsratesmuchlessthan1/10,000hrs,say1/100,000hrs,meeting the SIL requirements for low demand mode in Table D.1 set with a safety criticaldemandrateof1/10,000hoursmeansthatthesesmallerdemandrateoperationsexceedtheSIL level byoneorderofmagnitude, i.e., they are actually anorderofmagnitude safer thantheirSILratingaccordingtoAnnexDTableD.1wouldsuggest.

Table 4. Equivalence of SIL Levels in Terms of Probability of Dangerous Failure/Hr

SILPFD

(LowDemand)FromAnnexD

EquivalentPFD/hr(assumingmaximumdemandrateof1

demand/10,000hours)(LowDemand)

PFD/hr(High/ContinuousDemand)

FromAnnexD

1 [10-2,10-1) [10-6,10-5) [10-6,10-5)2 [10-3,10-2) [10-7,10-6) [10-7,10-6)3 [10-4,10-3) [10-8,10-7) [10-8,10-7)4 [10-5,10-4) [10-9,10-8) [10-9,10-8)


If PFD can be converted to PFD/hr for low demandmode of operation bymultiplying by anassumeddemandrate,thenforhigh/continuousdemandmodesofoperation,PFD/hrasgiveninAnnexDTableD.1canbeconverted toacorrespondingequivalentPFDbasedondemandrateusingEq.(4).ForagivenSILlevel,ifthedemandratewere1demand/1,000hrsthentheequivalentPFDwouldbesuchthatPFDxdemandrate=PFD/hr.Forexample,ifthePFD/hrisin the range [10-7, 10-6), i.e., level SIL 2, and the demand rate is 1/1,000 hrs, i.e., 10-3demands/hr,thentheequivalentPFDwouldbeintherange[10-4,10-3)becausetherange[10-4,10-3)x10-3equalstherange[10-7,10-6).Thecategoryofhigh/continuousdemandcoversawiderangeofdemandrates.Forexample,aplantwhichexperiencesademandapproximatelyevery4dayswouldhaveademandrateofabout1/100hrsandwouldbecharacterizedashavingahighdemand. Otherhighdemandsmaybe1demand/10hrsoreven1demand/hr. Whenthedemandrate is1demand/secorgreater, operation is considered continuous. Consider Table 5 which explores the PFDequivalenttoPFD/hrforvariousdemandratesinhighdemandmodeoverthevariousSILlevels.

Table5.PFDEquivalenttoPFD/hrinHigh/ContinuousDemandModeBasedonDemandRateSIL PFD/hr

(high/con’tdemandmode)=PFDxdemandrate

PFDrange(low

demandmode)

EquivalentPFDrangeforonedemand(high/continuousdemandmode)rangebasedondemandrates

demandrates

1/10,000hrs 1/1,000hrs 1/100hrs 1/10hrs 1/1hr

1 [10-6,10-5) [10-2,10-1) [10-3,10-2) [10-4,10-3) [10-5,10-4) [10-6,10-5)2 [10-7,10-6) [10-3,10-2) [10-4,10-3) [10-5,10-4) [10-6,10-5) [10-7,10-6)3 [10-8,10-7) [10-4,10-3) [10-5,10-4) [10-6,10-5) [10-7,10-6) [10-8,10-7)4 [10-9,10-8) [10-5,10-4) [10-6,10-5) [10-7,10-6) [10-8,10-7) [10-9,10-8)

Eachmagnitude increase indemand ratecorresponds toamagnitudedecrease inequivalentPFD,i.e.,PFDbecomessmallermeaningasmallerproportionofincorrectinputsassociatedwithdemandsarepermittedforagivenSILlevel.Thisequivalenceservesaveryusefulpurpose.There isnoneedtocollect largenumbersofhoursofoperatingexperience. Usingthe lowerlimit of the PFD equivalent range for a given demand rate and SIL level, the number, n, ofuniform,independentsamplesfromthesubsetofinputsassociatedwithdemandsthatmustbegeneratedandtestedoff-linecanbecalculatedfromEq.(3d).AllofthetheorydevelopedforevaluatingSWsafetythroughoff-linetesting for the lowdemandmodenowapplies tothesehigh/continuousdemandcases.Forexample,considerademandrateof1/1,000hrsatalevelofSIL2.Accordingtotheentriesin Table 3, using PFD/hr for SIL 2 in high/continuous demand mode a total of 3 x 107 hrs


(30,000,000hrs)ofoperationalexperiencearerequiredwithoutfailuretoattainaratingofSIL2 with confidence level 95%, and 4.6 X 107 hrs (46,000,000 hrs) of operational experiencewithout failurearerequiredforaconfidence levelof99%. Notethat,withademandrateof1/1,000hrs,in30,000,000(46,000,000)operationalhoursonewouldexpecttocollectdataonatotal30,000(46,000)demandsbutthesewouldnotnecessarilybedistributeduniformlyoverthespaceofallinputsassociatedwithdemands.UsingTable5withSIL2,thePFDequivalentforthedemandrateof1/1,000hrsisintherange[10-4,10-3);usingp*U=10-4(thelowerlimitoftheequivalentPFDrange)andEq.(3d)givesn=30,000testsfora95%confidencelevelandn=46,000testsfora99%confidenceinterval.Thusthetwodifferentwaysofviewingtheproblemlead to the say testing requirements, i.e., the inputs associatedwitheither30,000or46,000safety critical demandsmust be processed failure-free. But randomly generating 30,000 or46,000inputsassociatedwithsafetycriticaldemandsforoff-linetestingallowstheSWtestertoguarantee independent uniformly distributed samples over the entire input space associatedwithsafetycriticaldemandsleadingtocertificationthatisnotlimitedtoaparticularoperatingprofile.SomehavecriticizedAnnexDforrequiringthesamenumberofoperatinghoursforthesafetyassessmentof SW functioningatoperating siteswithverydifferenthigh/continuousdemandrates. Some reason that this tends topenalizeSWused inanoperatingenvironmentwheresafetycriticaldemandratesareontheorderofmonthsorweeksbyrequiringexcessivetestingintheseenvironmentswhilelikelyoverestimatingthesafetyofthesameSW(byundertestingit)inanenvironmentwherethesafetycriticaldemandratesareontheorderofdaysorhours.Fromamathematicalmodelingperspective,thesecriticismsareunwarranted.Consideranotherexamplewithahighersafetycriticaldemandratethanintheexampleabove.Suppose thedemand rate is equal to1demand/10hrs. UsingTable3withPFD/hr for SIL2againgivesatotalof3x107hrs(30,000,000hrs)ofoperationalexperiencewithoutfailuretoattain a rating of SIL 2 with confidence level 95%, and 4.6 X 107 hrs (46,000,000 hrs) ofoperationalexperiencewithoutfailureforaconfidencelevelof99%.Eventhoughthedemandrateissignificantlyhigherinthiscase,thesameamountofoperationalexperienceisrequiredforaSIL2ratingaswasrequiredinthefirstexample.Nowconsiderthat,withasafetycriticaldemand rateof 1/10hrs, in 30,000,000 (46,000,000) operational hours onewould expect tocollect data on a total 3,000,000 (4,600,000) demands. Using Table 5 with level SIL 2, theequivalentPFDfordemandof1/10hrsisintherange[10-6,10-5);usingp*U=10-6(thelowerlimitoftheequivalentPFDrange)andEq.(3d)givesn=3,000,000testsfora95%confidencelevelandn=4,600,000testsfora99%confidenceinterval.OverthesamenumberofoperatinghoursasthefirstexamplemanymoredemandsaresampledbecausethedemandrateishigherbutmanymoredemandsneedtobesampledbecausetheequivalentPFDneedstobemuchsmallertocompensateforthehigherdemandrateandstillhavePDF/hrbe inthesameSIL2levelrange.Finally,itmustbenotedthatwhenSWistestedoff-lineusingsimulatedinputs,thenumberofrequiredtests,n,mayinsomecasesexceedthesizeoftheentirespaceofallinputsassociatedwithsafetycriticaldemands.Inthesecases,itismoreefficienttoexhaustivelytestoff-lineall


inputsassociatedwithsafetycriticaldemands.Thisstatementistruebothforthecaseoflowdemandmodeofoperationandforthecaseofhigh/continuousdemandoperationwherenhasbeencomputedbasedonequivalentPFD.


REFERENCES1. IEC 61508 ed2.0 (2010-04), Functional safety of electrical/electronic/programmableelectronicsafety-relatedsystems,Geneva:InternationalElectrotechnicalCommission.2. IEC 61508-7 ed2.0 (2010-04), Functional safety of electrical/electronic/programmableelectronic safety-related systems - Part 7: Overview of techniques and measures, Geneva:InternationalElectrotechnicalCommission.3.Larsen,R.J.,Marx,M.L.,AnIntroductiontoMathematicalStatisticsandItsApplications,3rdEd.,PrenticeHall,UpperSaddleRiver,NJ,2001.4."BinomialProportionConfidenceInterval."Wikipedia,theFreeEncyclopedia.Web.17Jan.2015.<http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval>.


APPENDIXBasedonEq.(3), it ispossibletofixnandsolveforαasafunctionofp*Utherebyfindingthelevelofconfidenceonecanhaveinpbeingintheinterval[0,p*U]forafixednumberoftests,n.ReturningtoEq.(3d)andtakingtheexponentialofeachsidegivesEq.(3a)fromTable1,whichisrepeatedherefortheconvenienceofthereader,viz., exp(-np*U)≈α. (3a)Fig.A.1showsaplotof(3a)forn=30,000andn=46,000andp*Uintherange[0.5x10-4,0.5x10-3],i.e.,therangewhichincludestheupperhalfofSIL4throughthelowerhalfofSIL3.Alogscaleisusedforthehorizontalaxistoshowtheordersofmagnitudemoreclearly.Asaresult,theplotappearsnotquiteexponential. Note that the figureconfirmswhat is intuitively felt,that for a fixed value ofp*U increasing the number of tests deceasesα and correspondinglyincreasestheconfidencelevel.

Figure A.1 Plot of α vs p*U for n = 30,000 and 46,000 over a range of p*U

consistent with the upper half of SIL 4 and the lower half of SIL 3. ConsiderthesetofpointslistedinTableA.1andshownonFig.A.1.(NotethatinFig.A.1thereareactuallysixblackmarkersplottedconsistentwiththesixrowsofTableA.1. Twomarkersoverlapatp*U=5.0x10-4whichexplainswhyonlyfivemarkersarevisible.)ThevalueofαforeachpointwasobtainedfromEq.(3a)basedonthecorrespondingvaluesfornandp*U.Notehowasp*Udecreases,theconfidenceinterval[0,p*U]decreasesinlength,αincreasesandthecorresponding confidence level decreases. Conversely, as p*U increases, the confidenceinterval [0, p*U] increases in length, α decreases and the corresponding confidence levelincreases.Thisisalsointuitivebecause,forafixednumberoftests,onewouldexpecttohavelessconfidenceinasmallerintervalthaninalargerone.Alsonotethatusing30,000tests,ifno


incorrectinputsarediscovered,onecanbemorethan99.99%confidentthatpliesinthelowerhalfofSIL3orbelowandwith46,000testswithoutdiscoveryofanincorrectinput,onecanbeessentially100%confidentthatpliesinthelowerhalfofSIL3orbelow.Thisfurtherexplainswhyp*UischosenasthelowerlimitofSIL3whenEq.(3d)issolvedfornwithp*Uandαfixed.

Table A.1 Table entries indicate how confidence levels change as p*U changes for two fixed values of n.

n p*U α Confidence(1–α)x100% Truevalueofpliesintheinterval46,000 0.5x10-4 0.100 90% [0,0.5x10-4]30,000 0.5x10-4 0.223 77.7% [0,0.5x10-4]46,000 1.0x10-4 0.010 99% [0,1.0x10-4]30,000 1.0x10-4 0.050 95% [0,1.0x10-4]46,000 5.0x10-4 1.026x10-7 ~100% [0,0.5x10-3]30,000 5.0x10-4 3.059x10-7 99.99997% [0,0.5x10-3]

Understanding IEC61508 Annex D Final - exida

Documents