presentation - Henry Muccini

d

USC, October 2003 - Henry MucciniSoftware

Engineering and Architecture Group

i

1/41

SA-based Code SA-based Code TestingTesting

Henry MucciniComputer Science Department

Universita’ dell’Aquila – Italy

([email protected])[http://www.HenryMuccini.com]

d



i

2/41

My ResearchMy Research

● Ph.D. Thesis on “Software Architecture for Testing, Coordination Models and Views Model Checking”, year 2002.

● Software Architecture and Coordination● Software Architecture and Model Checking

● Software Architecture and Testing

d



i

3/41

● Brief introduction on Software Architecture and Testing

● SA-based Conformance Code TestingSA-based Conformance Code Testing ● SA-based Regression TestingSA-based Regression Testing● PLA-based TestingPLA-based Testing

● Ongoing and future work● Some references

Talk OverviewTalk Overview

d



i

4/41

Software Architecture (SA)Software Architecture (SA)

● 1992 and 1993: – SA is recognized as an independent area of research;

● 1993-1995: – SA described through box and line, informal, diagrams;

● 1995-2002: – Architectural Description Languages (ADLs) are introduced to

formally describe SA;● 1997-2003:

– more interest on dynamic aspects of SA;– SA used in academia and industry;– SA and analysis– SA recognized as a valid tool to describe complex, concurrent, real

systems;

The History: [PhDThesis, Ch2] [PhDThesis, Ch2]

d



i

5/41

In general terms…In general terms…● Describes (in a formal language) how a system is

structured into components and connectors…

– Components

● computation

– Connectors● interaction

– Ports, Interfaces, …

… how these

components interact

SA Structure (topology)

SA Dynamics (behavior)

d



i

6/41

SA dynamicsSA dynamics

● A model of Software Architecture behavior:– Labeled Transition System (LTS)– Finite State Machine – Petri Nets– …

● Given the SA formal description, the model can be automatically generated; the dynamics is expressed in terms of components interaction via connectors

d



i

7/41

Motivations on SA-based Analysis Motivations on SA-based Analysis ● For Analysis: [PhDThesis] [PhDThesis]

– SA management is expensive and time consuming ● maximize the benefits

– Analyze SA as much as possible

– SA and Testing – SA and Coordination models– Model checking SA

– SA and Performance Analysis– Deadlock detection on SA-based specification– SA and Security– Refining SA – SA-based development processes– ADL- based analysis of SA

d



i

8/41

Module Unit Integration SystemModule Module

M

M

M

M

Structural FunctionalCode LevelOriented to SyntaxUnit Tests

Functional propertiesFormal specifications

TestingTesting

● Testing is the process of executing a program with the purpose of – finding errors or defects– increasing the confidence in the proper functioning of the

software (dependability).● Not exaustive approach…

… based on the selection of an adeguate set of tests

d



i

9/41

Architecture-based Integration Architecture-based Integration and Conformance Testingand Conformance Testing

● Integration Testing: unit tested components are integrated to build complex systems, and – Integration Testing is aimed at exposing problems in the

interfaces and interactions between the system components

● Functional: focus on the functional requirements● Architectural: information for testing is extracted from

the Architectural specification... ● Down to the Code: … and propagated down to the

Code... Conformance Testing

d



i

10/41

The main ideaThe main idea

ClientA

ClientB

ServerMaster

ServerSlave

Recovery

ClientC

SA CodeXClient

ClientA ClientB

ClientC

Class Client--------------------------

● Goal:– Derive test cases– using software architectural description– run test cases on the Code

Identify SA-Tests Apply the Tests

Transform SA-Tests

Into Code-level

tests

d



i

11/41

Working DirectionsWorking Directions

1 - SA-based Conformance Code Testing(Joined work with Antonia Bertolino and Paola

Inverardi)

2 - SA-based Regression Testing– Testing C2-style architectures

(Joined work with Marcio Dias and Debra Richardson)

3 - PLA-based Testing (Joined work with Andre’ van der Hoek)

d



i

12/41

A

B

A

B

D

A

B C

D

Test ClassesTest Execution

Obj1

m1

Obj5Obj3Obj2

m6

m3

ObjN

Objk

Obj8

ObjL

m1m6

m3

m1

m2

m2

îe¤íiÛìoö

1act

ïî

ïíS

ì

3act

{2act{2act

step4

[ICSE01][ICSE01]

LTS Model

201964

1

144

1

12

9

1 1332

2019532

1172

4

12

1

96 1

113

2

75

2

6

4

1

5

32 1

0

8

7

6

5

321

0

4

2926

2

2822

20

12

10

13

11

3

9

36

35

33

32

34

15

18

17

16

3114

13

16

37

292830

24

34

27

33

22

Step2

Abstraction

step3

ALTS Model

A

B C

D

Components:User, Router, Server, TimerUser, Router, Server, Timer

Connectors:AlarmUR, AlarmRS, AckSR,AlarmUR, AlarmRS, AckSR,

AckRU, Check, Nofunc, Clock AckRU, Check, Nofunc, ClockComponents Behavior:

…, …, ……, …, …Connectors Behavior:

…, …, ...…, …, ...

SA Specification

Step1

modeling

[ICECCS’97][ICECCS’97]

[ICSE00][ICSE00]

1. SA-based Conformance Code Testing1. SA-based Conformance Code Testing [PhDThesis, ch. 4][PhDThesis, ch. 4]

[BookChapter03][BookChapter03]

d



i

13/41

Step1: Formal SA specificationStep1: Formal SA specification

Molecule syntax:Molecule: Process| Operation| M.MProcess: User, Router, Server, TimerChannel: check, alarmUR, alarmRS,ackRU, ackSR, nofunc, clockOperation: i(Channel), o(Channel), ...

InitialState (S0):Multiset of Molecules: m1, m2, ..., mn

Rules:m1, m2, ..., mk m1', m2', ..., ml'

User1

Router Server

Timer

AlarmUR AlarmRS (c)Check1

Nofunc

Clock

AckSR (c)

AckRU1

User2

AlarmUR1

AlarmUR2

Check2

Check

AckRU2

● Architectural Description Language (ADL) that produces a Labeled Transition System (LTS)

● Chemical Abstract Machine (CHAM) [ICSE00][ICSE00] ● Finite State Process (FSP) [ICSE01][ICSE01]

● Dynamics in terms of Component interactions

d



i

14/41

20

1911

10

6

3

6

1614126

3

63

4

5

3

97

5

11

7

6

11

96

5

4

3127

5

4

117

6

4

97

65

20

196

4

1

14

4

1

12

9

1 13

32

20

19

532

117

2

125

4

3

11

6

43

96

53

6

5

4

7

4

12

1

9

61

11

3

2

7

5

2

65

4

3

6

4

1

5

3

2

2 1

0

10

11

8

7

6

5

4

321

0

9

4

2926

2

28

2220

12

10

13

11

3

9

36

35

33

32

34

15

18

17

16

31

14

13

12

16

2051439051

37

22

21

20

2928

27

26

25

24

23

30

19

25

24

21

23

20

22

19

24

34

27

33

22

44

43

42

48

46

4140

39

38

43

40

41

42

39

38

45

43

38 49

72

71

159 107

67

107

117

77

69

68111

113

107

112

109

70

43

23

The The AArchitecture-level rchitecture-level BBehaviorehavior

d



i

15/41

● Architectural Views [Kruchten]

– Flow view– Component Based view– Concurrency view– ...

● Obs_Functions to view the system only from a – perspective that is relevant for testing– Obs : L D {}

● Abstract LTS (ALTS)● Minimization and Reduction (trace-equivalence)

Step2 : The Step2 : The AbstractionAbstraction

d



i

16/41

F R n o

F R n o F R n o

T R a ck 1T R a ck 2

F R n o

T R a ck 2T R a ck 1

F R a 1F R a 2

F R a 2F R a 1

D

A

BC

Flow view:

ComponentBased View

Alarm Flow

Check Flow

Clock Flow

User Component

Router Comp.

Server Comp.

R e c e iv e A c k 1

R e c e iv e A c k 2R e c e iv e A c k 1

S e n d A la r m 1S e n d A la r m 2


A

B C

D


d



i

17/41

Step3 - Architectural Test ClassStep3 - Architectural Test Class

● ALTS Path Coverage● Each complete path corresponds to an Architectural Test Class

– McCabe path coverage criteria

– FSP hiding operators

...





A

B C

D






A

B C

D






A

B C

D


d



i

18/41

Step4 - Testing the SoftwareStep4 - Testing the Software ImplementatioImplementationn– How are the LTS paths implemented by the Code?

● We can test whether the system as-built implements this architectural behavior

Comp1 Comp4Comp3Comp2

act1

act2

act3

act1

Obj1

m1

Obj5Obj3Obj2

m6

m3

ObjN

Objk

Obj8

ObjL

m1

m6m3

SA test Code test

d



i

19/41

ApplicationsApplications

● Tele Remote Medical Care System (TRMCS) [ICSE2000,ICSE2001]

● The Whiteboard case study [BookChapter]

● The Elevator case study [Submitted]

d



i

20/41

● LTS can be automatically derived from an Architectural specification (LTS generator Tool, LTSA Tool);

● ALTS can be semi-automatically generated (FC2Tools + The Abstractor Tool)

ToolTool SupportSupport

LTS generatorLTSA tool

LTSAbstractor

SA formal specification LTS ALTS

d



i

21/41

● Step1:– SA specification and modelization– State explosion problem… completeness – Considerations:

● we used Cham and FSP

● Step2:– Observation and Abstraction– Considerations:

● It is an empirical task, based on the software architect experience● Classification of observations could help● Methods similar to SAAM or SCENT, in which architectural information

is empirically captured, could help in this task● The ALTS construction has been automated using the

CAESAR/ALDEBARAN tool

Topics of interest and Topics of interest and ConsiderationsConsiderations

d



i

22/41

● Step3:– Path coverage– Considerations:

● McCabe's test technique seems a reasonable coverage criterion ● it may be automated

● Step4:– Traceability and development process– Deterministic and non deterministic Testing– Considerations:

● it is the most difficult part● relating SA specification to code● key concepts: traceability, development process

TopiTopics of interestcs of interest and and ConsiderationsConsiderations

d



i

23/41

Step4: Step4: Some ConsiderationsSome ConsiderationsMapping problems:

● It is not so obvious which classes and methods implement an architectural functionality

● More sequences of method calls can implement the desired architectural behavior

● SA description is abstract and hides functionalities and objects defined at the code level

● The SA model usually describes only the expected behaviors, while the code also catches exceptional ones

● Test execution: nondeterministic or deterministic [CarverTai_TSE98] [CarverTai_TSE98] approach

d



i

24/41

User2User2User1User1 RouterRouterAlarm1Alarm1

Alarm2Alarm2

Ack2Ack2

Ack1Ack1

SA behavior Source CodeSource Code

class MasterRouter{ ServerConnection allarm; ServerConnection okFunction; // the services ReceiveUserAllarm serviceReceiveUserAllarm; ReceiveUserOkFunz serviceReceiveUserOkFunz; String name,serverName; static public PrintWriter user_router_ok_funz; static public PrintWriter user_router_alarm; ...

??????

drivesdrives

drivesdrives

SA (topology and model)

Design and Source Code Def.Design and Source Code Def.

Source Code Source Code (abstractions)(abstractions)

SA (model)

SA (model)

Source CodeSource Code

d



i

25/41

● Motivations and Goals:

– To reduce the testing effort during architecture or code maintenance

– To handle the “architectural drift”– To maximize benefits in using Software

Architecture

2. SA-based 2. SA-based RegressionRegression Testing Testing [Submitted][Submitted]

d



i

26/41

● Test modified software and provide a certain confidence that no new errors are introduced into previously tested code.

● Given program P and P’, T is a test suite for P, regression testing techniques:– provide a certain confidence that P’ is still correct

with respect to a set of test T’, which is a subset of T;

– Helps to identify new test cases for T’.

Regression TestingRegression Testing

d



i

27/41

● Select T’, subset of T and relevant for P’;● Test P’ with respect to T’;● If necessary, create T’’, to test new

functionality/structure in P’;● Test P’ with respect to T’’;● Create T’’’, a new test suite and test history;

Regression Test Selection Regression Test Selection techniquetechnique

d



i

28/41

Project GoalProject GoalSoftw are Architecture SA1

(version 1)

Code1(version 1)

Code2(version 2)

Thecode

evolves

Softw are Architecture SA2(version 2)

The SA evolves

Goal1: Test thenew code

conformance tothe initial SA

Architectural Test Cases(ATCs)

are extracted to test thesource code

Component AComponent C

Component B

Component AComponent C'

Component B

Component Y

Goal2: Test thenew SA and

code, reusingprevious results

Code

d



i

29/41

From Traditional to SA-based Regression From Traditional to SA-based Regression TestingTesting

Build GP',the graph

for P'

Run T over Pand Evaluate Results

Create aTest History

Select T' for P'

Compare GPw ith GP'

SA-based Testing

SA-basedRegression Testing

ExtractingSA-level Test Cases

Mapping SA-level testsinto Code-level tests asa w ay to select T for P

SA-spec for S

a: S tep 0

b: S tep 1

c: S tep 2

e:S tep 4

2) S tep B

3) S tep C

4) S tep D

TestingCriterion

d: S tep 3Build GP,the graph

for P

1) S tep A

All such stepsallow the

selection of atest suite T for P,driven by the SA

Build GP,the graph

for P

Build GP',the graph

for P'

Select T for P

Create aTest History

Select T' for P'

Compare GPw ith GP'

Testing

Regression Testing

Run T over Pand Evaluate

Results

d



i

30/41

Initial experimentInitial experiment● The approach has been applied to the Elevator case study

– Architecture in the C2 style– Implemented using the C2 framework

● Tool support:– The SA is formally specified using the C2 specification language

and FSP. – A behavioral model of the system is automatically produced from

the FSP specification, using the LTSA tool.– The abstraction over the LTS behavioral model is realized again in

FSP. – The mapping between architectural tests and code-level tests is

provided for by the C2 Framework. – Test cases are run over the code using the Argus-I environment.– The selective regression testing step is supported by the DejaVOO

tool.

d



i

31/41

3. PLA-based Testing3. PLA-based Testing [Tacos 2003][Tacos 2003]

● A product line architecture precisely captures, in a single specification, the overall architectureoverall architecture of a suite of closely-related productssuite of closely-related products [Bosch2000]

● A PLA explicitly specifies: – i) elements that are present in all productspresent in all products, – ii) elements that are optionaloptional, and – iii) elements which may be incorporated in one of

many forms (variantsvariants)

● Whereas a “regular” architecture defines the structure of a single product, a product line architecture (PLA) defines the common architecture for a set of common architecture for a set of related productsrelated products [Bosch2000]

d



i

32/41

Testing PLATesting PLA

● A new challenge: – how to deal with optional elements or with the

magnitude of productsmagnitude of products that may be present?

● The goal of this paper is to highlight the challenges and opportunities for software challenges and opportunities for software testing of PLAstesting of PLAs

● We believe that existing mechanismsexisting mechanisms with which SAsSAs are tested can be adaptedcan be adapted to PLAs

d



i

33/41

PLA examplePLA example

foo

barbar

goop

bar foobar

mandatory

optional

variant

variant

In total, twenty-four different product architectures can be formed.

d



i

34/41

Testing PLA- Unit TestingTesting PLA- Unit Testing

● SA:– Each SA component need to be unit tested

● PLA:– allall components should be unit tested as well

● Including each optionaleach optional component and each varianteach variant of a variant component

– However,the order in which they have to be tested can be adjusted based on “prioritypriority”.

d



i

35/41

Testing PLA – Integration Testing PLA – Integration TestingTesting

● SA:– components and connectors are combined together

according to the architecture configurationconfiguration

● PLA:– no single architectural configuration exists

– Possible solutions:● Iterative build-upIterative build-up integration approach: powerful but

expensive● big bangbig bang: limited but “effortless”

– Core-first + big bang approachCore-first + big bang approach● Integrated with “heuristics” to test only particular combinations

d



i

36/41

Testing PLA – Conformance Testing PLA – Conformance TestingTesting

● SA:– conformance testing has been used to detect detect

conformance errors between the SA and its conformance errors between the SA and its implementationimplementation.

● PLA:– an implementation an implementation II conforms to its PLA when conforms to its PLA when:

● I conforms to a single PA● I conforms to all the possible PAs out of the PLA● I conforms, at least, to all the constraintsconstraints and functionalitiesfunctionalities

associated to the mandatory, core elementscore elements of the PLA

– a product architecture PA conforms to its PLAa product architecture PA conforms to its PLA

d



i

37/41

Testing PLA – Regression Testing PLA – Regression TestingTesting

● SA:– If a new version P1v2 of an implementationimplementation P1v1 is

produced, regression testing techniques can be used to test the conformance of P1’ to the initial SAto the initial SA

– If a new version (SAv2) of an architecturearchitecture SAv1 is produced, SAv2 test cases may be selected reusing SAv1’s test cases.

● PLA:– PLA evolutionPLA evolution: PLA that becomes PLA’

– PA becomes PA’PA becomes PA’

– Code becomes Code’Code becomes Code’

d



i

38/41

Future WorkFuture Work

SA-based Code SA-based Code TestingTesting

d



i

39/41

● SA-based Conformance Code TestingSA-based Conformance Code Testing – Testing and Formal Methods

Integration with the TGV Integration with the TGV [TGV] [TGV] (Test Generation (Test Generation and and Validation) environment used to test Validation) environment used to test formal formal specificationsspecifications

– Testing C2 style architectures

C2 framework and Dradel C2 framework and Dradel [ongoing work]

– Integration with XADL, Menage, Behavioral modelIntegration with XADL, Menage, Behavioral model

● SA-based Regression TestingSA-based Regression Testing– Implement the second goalImplement the second goal

● PLA-based TestingPLA-based Testing– Introduce an approach for PLA-based testingIntroduce an approach for PLA-based testing– Integration with XADL, Menage, Behavioral modelIntegration with XADL, Menage, Behavioral model

d



i

40/41

Integration with XADL, Integration with XADL, Menage, Behavioral modelMenage, Behavioral model

● SA specified in XADL – with an additional behavioral specification

● The XADL architecture is implemented by using the new C2 Framework – May be something different

● Menage features are used to help the regression testing steps

● The testing capability is embedded into Menage

d



i

41/41

Some ReferencesSome References● [ICSE2000][ICSE2000] A. Bertolino, F. Corradini, P. Inverardi and H. Muccini. "Deriving Test Plans from

Architectural Descriptions". In ACM Int. Conf. on Software Engineering (ICSE2000), pp. 220-229, Limerick (Ireland), June 2000.

● [ICSE2001][ICSE2001] A. Bertolino, P. Inverardi and H. Muccini. "An Explorative Journey from Architectural Tests Definition downto Code Tests Execution". In IEEE Int. Conf. on Software Engineering (ICSE2001), Toronto, May 2001.

● [PhDThesis][PhDThesis] H. Muccini. Software Architecture for Testing, Coordination Models and Views Model Checking, PhD thesis, year 2002.

● [Tacos 2003] [Tacos 2003] H. Muccini, A. van der Hoek. "Towards Testing Product Line Architectures“In Proc. of the ETAPS2003 workshop on "Test and Analysis of Component Based Systems" (Tacos), Warsaw, Poland, April 2003. In Electronic Notes of Theoretical Computer Science, Vol. 82, N. 6 (2003).

● [BookChapter][BookChapter] A. Bertolino, P. Inverardi and H. Muccini. “Formal Methods in Testing Software Architectures”. Chapter in Formal Methods for Software Architectures, SFM-03:SA Lectures, Eds. M. Bernardo, P. Inverardi, LNCS 2804, 2003, p. 124-149.

● [Submitted] [Submitted] H. Muccini, M. Dias and D. Richardson. Software Architecture-based Conformance and Regression Testing. Submitted for publication.

● [ongoing work] [ongoing work] H. Muccini, M. Dias. Systematic Testing of C2 style Software Architecture. Under Submission for publication.

● [CarverTai_TSE98][CarverTai_TSE98] Carver, R.H., Tai, K.-C.Use of Sequencing Constraints for Specification-Based Testing of Concurrent Programs. IEEE Trans. on Software Engineering 24, 6 (June 1998).

● [Kruchten][Kruchten] P. Kruchten. Architectural Blueprints - The “4+1” View Model of Software Architecture. IEEE Software 12 (6), November 1995, pp. 42-50.

● [TGV] [TGV] Fernandez, J.-C., Jard, C., Jeron, T., Nedelka, L., Viho, C.: An Experiment in Automatic Generation of Test Suites for Protocols with Verification Technology. Special Issue of Science of Computer Programming, Vol. 29, pp. 123-146, 1997.

presentation - Henry Muccini

Documents