Toward a Quality Certification Model for Grid Research Projects: an ETICS feasibility Study

www.eu-etics.org

INFSOM-RI-026753

Toward a Quality Certification Toward a Quality Certification Model for Grid Research Model for Grid Research Projects: an ETICS feasibility Projects: an ETICS feasibility StudyStudyAndrea ManieriAndrea ManieriAdriano RippaAdriano Rippa - Engineering Ingegneria Informatica s.p.a.- Engineering Ingegneria Informatica s.p.a.On behalf of ETICS ProjectOn behalf of ETICS Project

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 2INFSOM-RI-026753

The starting point

• Current quality assurance standards are useful Current quality assurance standards are useful but:but:– are process oriented are process oriented – provide theoretical guidelines on the organizationprovide theoretical guidelines on the organization

• TheThe problemproblem::– Guidelines can be applied but it’s hard to Guidelines can be applied but it’s hard to

systematically verify goodness of resultssystematically verify goodness of results

• OurOur solutionsolution::– A quality assurance model A quality assurance model

– fully automatable in measuring and verifying activitiesfully automatable in measuring and verifying activities– not subjectivenot subjective

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 3INFSOM-RI-026753

The timeline

FIRST BETA FIRST BETA

VERSIONVERSION

FINAL BETA FINAL BETA

VERSIONVERSION

OCTOBEROCTOBER

FIRST OFFICIALFIRST OFFICIAL

VERSIONVERSION

DECEMBERDECEMBER

ECHOGRID/EUChinagridECHOGRID/EUChinagrid

Conf. Beijing 24-25 AprilConf. Beijing 24-25 April

MAYMAYFEBRUARYFEBRUARY

OGF 20/EGEE UFOGF 20/EGEE UF

7-11 May7-11 May

Belief ConferenceBelief Conference

RIO – BRASIL 25-28 JuneRIO – BRASIL 25-28 June

OGF 21OGF 21

15-19 Oct15-19 Oct

QUALIPSO ConferenceQUALIPSO Conference

Dec 2007Dec 2007

NowNowALL HANDSALL HANDS

MEETINGMEETING

23-25 May23-25 May

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 4INFSOM-RI-026753

Terminology

• Measure: Measure: a value (number or category) assigned to an a value (number or category) assigned to an attribute of an entity (e.g. software component)attribute of an entity (e.g. software component)

• MeasurementMeasurement: : The act or process of assigning a The act or process of assigning a number or category to an entity to describe an attribute number or category to an entity to describe an attribute of that entity. of that entity.

• MetricMetric: : A function whose inputs are software measures A function whose inputs are software measures and whose output is a single numerical value that can and whose output is a single numerical value that can be interpreted as the degree to which software be interpreted as the degree to which software possesses a given attribute that affects its quality.possesses a given attribute that affects its quality.

• PerspectivePerspective: A set of homogenous software attributes : A set of homogenous software attributes that lead to some quality assumptionsthat lead to some quality assumptions

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 5INFSOM-RI-026753

Gird Quality Assurance Certification Model (GQACM ): a 3D QA Model

• Let’s suppose A, B, C to be Let’s suppose A, B, C to be different perspectivesdifferent perspectives

• Most of Common QA Most of Common QA Standards (e.g. CMMI) are Standards (e.g. CMMI) are Waterfall-likeWaterfall-like

• GQACM uses 3 independent GQACM uses 3 independent perspectives to provide perspectives to provide client-friendly resultsclient-friendly results– Users can be interested in Users can be interested in

some attributes and less in some attributes and less in others..others..

– So-made perspectives can So-made perspectives can have sublevels…have sublevels…

AA

BB

CC

AA

BB

CC

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 6INFSOM-RI-026753

GQACM: Preliminary remarks

• GQACM is GQACM is independentindependent from the ETICS tools from the ETICS tools

• However, any GQACM implementation However, any GQACM implementation must use must use automationautomation

– – the most useful characteristic of the model is that it the most useful characteristic of the model is that it can be automatically applied!can be automatically applied!

• GQACM has three perspectives (I.e. independent points GQACM has three perspectives (I.e. independent points of view) to derive the quality degree of a software :of view) to derive the quality degree of a software :– Code and testCode and test– PlatformsPlatforms– StandardsStandards

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 7INFSOM-RI-026753

Perspectives composition

• Code Test Perspective: to qualify the goodness Code Test Perspective: to qualify the goodness of the code and of the testof the code and of the test– Coding ConventionsCoding Conventions– Code CommentingCode Commenting– Other QA metrics related to the corresponding Other QA metrics related to the corresponding

programming languageprogramming language– Unit and functional testsUnit and functional tests

• Platform perspective: to qualify B&T Platform perspective: to qualify B&T requirements on different platformsrequirements on different platforms– The set of GQACM available platformsThe set of GQACM available platforms

• Standard perspective: to qualify the compliance Standard perspective: to qualify the compliance with standards (e.g. IPV6, BES, JSDL, SRM, with standards (e.g. IPV6, BES, JSDL, SRM, WS-I)WS-I)– The set of GQACM available standardsThe set of GQACM available standards

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 8INFSOM-RI-026753

Test execution metrics

• Cyclomatic complexityCyclomatic complexity• CCT (method level): CCT (method level):

– if C is the cyclomatic complexity (of the method) and T is the if C is the cyclomatic complexity (of the method) and T is the number of executed tests then CCT = C/Tnumber of executed tests then CCT = C/T

• RFCT (class level): RFCT (class level): – if CK RFC is the the number of methods which can be invoked if CK RFC is the the number of methods which can be invoked

from an instance of the class target to answer a messagefrom an instance of the class target to answer a message ) ) and and T is the number of executed tests on it then RFCT = RFC/TT is the number of executed tests on it then RFCT = RFC/T

• ET (class level): ET (class level): – if E is the number of events which interact with the class and T is if E is the number of events which interact with the class and T is

the number of executed tests then ET = E/Tthe number of executed tests then ET = E/T

• NOCT (class level): NOCT (class level): – if CK NOC is the Number Of Children of a class and T is the if CK NOC is the Number Of Children of a class and T is the

number of executed tests on the class then NOCT = NOC/Tnumber of executed tests on the class then NOCT = NOC/T

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 9INFSOM-RI-026753

Code comment metrics

• FC: FC: – if F is the number of functions and C is the number of if F is the number of functions and C is the number of

comment paragraphs then FC = F/Ccomment paragraphs then FC = F/C

• AD:AD:– Automatical documentation programs use: e.g. Automatical documentation programs use: e.g.

doxygendoxygen

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 10INFSOM-RI-026753

Coding convention metrics 1/2

• Only one class definition per header fileOnly one class definition per header file• Filename = ClassnameFilename = Classname• Filename in lowercaseFilename in lowercase• No cycles in “include” operationsNo cycles in “include” operations• No C instruction (in C++ programming)No C instruction (in C++ programming)• Coherent Programming ConventionCoherent Programming Convention

– Curly Brackets positioningCurly Brackets positioning– First character in variables namesFirst character in variables names– Lower case in macros namesLower case in macros names– Coherent use of new linesCoherent use of new lines– Semicolons only at the end of a lineSemicolons only at the end of a line

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 11INFSOM-RI-026753

Coding convention metrics 2/2

– No Never used functionsNo Never used functions– No Not reachable code pathsNo Not reachable code paths– No Not defined prototypesNo Not defined prototypes– No Never used constantsNo Never used constants– No Never used macrosNo Never used macros– No Never used types definitionsNo Never used types definitions– No Never used variablesNo Never used variables– Too frequent “breaks” useToo frequent “breaks” use– No absence of variables de-alloactionNo absence of variables de-alloaction– No Null pointersNo Null pointers– Management of one instruction long “if” or one instruction long Management of one instruction long “if” or one instruction long

“cycle”“cycle”– Shared resources managementShared resources management

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 12INFSOM-RI-026753

other metrics

• Other software related metricsOther software related metrics– (e.g. Chidamber and Kemerer (CK) Metrics for OO (e.g. Chidamber and Kemerer (CK) Metrics for OO

software)software)

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 13INFSOM-RI-026753

Final Score1/2

• For every perspective must be genrated:For every perspective must be genrated:– a score between 0 and 100a score between 0 and 100– a list of passed and not passed testsa list of passed and not passed tests

• The way to calculate the score is different for every The way to calculate the score is different for every perspective.perspective.– For code perspectiveFor code perspective

– The score is simply calculated using this mathematical formula: The score is simply calculated using this mathematical formula:

(X/N)*100(X/N)*100

where X is the number of successes, N the number of used metricswhere X is the number of successes, N the number of used metrics

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 14INFSOM-RI-026753

Final Score 2/2

– For platform perspectiveFor platform perspective– If If

– DP is the number of declared compliant platforms,DP is the number of declared compliant platforms,– CP the number of really compliant platforms,CP the number of really compliant platforms,– TP the number of GQACM available platforms ,TP the number of GQACM available platforms ,– CTP the total number of compliant platforms ,CTP the total number of compliant platforms ,

– then the score can be calculated with this mathematical formula:then the score can be calculated with this mathematical formula:

(CP/DP)*50+(CTP/TP)*50(CP/DP)*50+(CTP/TP)*50

– For standard perspectiveFor standard perspective– IfIf

– DS is the number of declared compliant standards,DS is the number of declared compliant standards,– CS is the number of really compliant standards,CS is the number of really compliant standards,– TS is the number of available standards,TS is the number of available standards,– CTS is the total number of compliant standards (declared + not declared),CTS is the total number of compliant standards (declared + not declared),

– then score can be calculated with this mathematical formula:then score can be calculated with this mathematical formula:

(CS/DS)*50+(CTS/TS)*50(CS/DS)*50+(CTS/TS)*50

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 15INFSOM-RI-026753

Score visualization

• The graphical view of the score is related to the used The graphical view of the score is related to the used test tooltest tool

• In general, all tools have to use at least the “classical In general, all tools have to use at least the “classical version”:version”:– Final score: <total>/<max possible score> Final score: <total>/<max possible score>

([<perspective><score>])([<perspective><score>])– WhereWhere

– Perspective can be: Code (C), Platforms (P), Standards (S)Perspective can be: Code (C), Platforms (P), Standards (S)– Score: 0-100 (where 0 is 0% compliant, 100 is 100% compliant)Score: 0-100 (where 0 is 0% compliant, 100 is 100% compliant)– Total score: sum of the scoresTotal score: sum of the scores

– E.g. component X: GQACM score = 151/300 (C90-P51-S10)E.g. component X: GQACM score = 151/300 (C90-P51-S10)

• ETICS could complement the standard basic ETICS could complement the standard basic visualization using a 3d graph to give the user a better visualization using a 3d graph to give the user a better feeling interfacefeeling interface

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 16INFSOM-RI-026753

GQACM integration in an ISO 9126 certified organization

• Facts:Facts:– ISO 9126 define quality attributes .ISO 9126 define quality attributes .– GQACM support the measure and evaluation of quality metrics.GQACM support the measure and evaluation of quality metrics.

• Hypothesis:Hypothesis:– ISO9126 Portability: adaptability, installabilityISO9126 Portability: adaptability, installability

– ISO9126 asks OS adaptability and installability for software of ISO9126 asks OS adaptability and installability for software of certified organizations.certified organizations.

– GQACM may test deployment of SW on different platforms GQACM may test deployment of SW on different platforms (HW+OS)(HW+OS)

– ISO9126 Maintainability: all.ISO9126 Maintainability: all.– ISO9126 asks to develop a software that is easy to analyse, easy to ISO9126 asks to develop a software that is easy to analyse, easy to

modify, Stablemodify, Stable– GQACM metrics (such as the CK for OO or coding convetions GQACM metrics (such as the CK for OO or coding convetions

metrics) helps to verify in concrete these requirementsmetrics) helps to verify in concrete these requirements

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 17INFSOM-RI-026753

GQACM integration in an ISO 9126 certified organization

• ISO9126 Functionality: accuracy.ISO9126 Functionality: accuracy.– ISO9126 asks that a software have to give results according with ISO9126 asks that a software have to give results according with

requirements.requirements.– GQACM certification can run functional test (or plug-in external GQACM certification can run functional test (or plug-in external

test suits) of declared compliances.test suits) of declared compliances.

• ISO9126 Functionality: adequacy.ISO9126 Functionality: adequacy.– ISO9126 asks for correct functions with specific goals.ISO9126 asks for correct functions with specific goals.– GQACM may check not dead functions.GQACM may check not dead functions.

• Our thesis:Our thesis:– Many ISO9126 quality metrics can be measured and Many ISO9126 quality metrics can be measured and

evaluated based on GQACMevaluated based on GQACM– Any tool implementing it and can be integrated as Any tool implementing it and can be integrated as

supporting ISO9126 adoptionsupporting ISO9126 adoption

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 18INFSOM-RI-026753

GQACM integration in a CMMi certified organization

• FactsFacts– CMM is process oriented (it gives guidelines).CMM is process oriented (it gives guidelines).– GQACM is product oriented.GQACM is product oriented.

• Hypotesis:Hypotesis:– CMM2 CMM2

– asks to “control and value”asks to “control and value”– GQACM propose the measures and metrics to be collected, hence GQACM propose the measures and metrics to be collected, hence

controlling and evaluating specific sw related itemscontrolling and evaluating specific sw related items

– CMM3 CMM3 – asks to “use internal standards and have a reference model”asks to “use internal standards and have a reference model”– GQACM proposes code and implementation conventions (how to GQACM proposes code and implementation conventions (how to

organise and structure code, how to define tests, etc.)organise and structure code, how to define tests, etc.)

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 19INFSOM-RI-026753

GQACM integration in a CMM certified organization

– CMM4 CMM4 – asks to “have measured and controlled processes using quantitative asks to “have measured and controlled processes using quantitative

and statistical techniques”and statistical techniques”– GQACM propose the continuous collection of quality data to GQACM propose the continuous collection of quality data to

perform trends analysis.perform trends analysis.

– CMM5 CMM5 – asks to have an “improvement process and quality based on asks to have an “improvement process and quality based on

measurement”measurement”– GQACM provide numerical reference to set the improvement GQACM provide numerical reference to set the improvement

processes!!processes!!

– Our ThesisOur Thesis– GQACM is not in contrast with CMMI and can be integrated in GQACM is not in contrast with CMMI and can be integrated in

organization as a tool to support CMMI adoptionorganization as a tool to support CMMI adoption

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 20INFSOM-RI-026753

ETICS B&T tool and GQACM

• ETICS:ETICS:

– Could perform many measurements and calculate Could perform many measurements and calculate metrics (e.g. JPmdPlugin and JFindbugsPlugin) metrics (e.g. JPmdPlugin and JFindbugsPlugin) Code and test perspectiveCode and test perspective

– Can build and test software on several distinct Can build and test software on several distinct platforms platforms Platform perspective Platform perspective

– Could test the software compliance with standards Could test the software compliance with standards Standard perspectiveStandard perspective

– Can automate all the processes (build, deployment, Can automate all the processes (build, deployment, test)test)

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 21INFSOM-RI-026753

Conclusions

• GQACM is a certification modelGQACM is a certification model– AutomatableAutomatable– 3D3D– Not in contrast with classical standardsNot in contrast with classical standards– FreeFree

• GQACM doesn’t require big efforts to be usedGQACM doesn’t require big efforts to be used

ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007 22INFSOM-RI-026753

Thanks

http://www.eu-etics.org

Questions?Questions?

Feedbacks?Feedbacks?