D2.3 FITMAN Verification & Validation Generic Assessment Package · 2017-04-24 · FITMAN Verification & Validation Generic Assessment Package 20/09/13 – V1.0 Document Owner: Giacomo

Project ID 604674 FITMAN – Future Internet Technologies for MANufacturing

30/09/2013 Deliverable D2.3 – V1.0 - Final

1 DISSEMINATION: PUBLIC FITMAN CONSORTIUM

D2.3

FITMAN Verification & Validation Generic Assessment Package

20/09/13 – V1.0

Document Owner: Giacomo Tavola – POLIMI – [email protected]

Contributors: Fenareti Lampathaki NTUA, Guy Doumeingts IVLAB, Kim Jansson VTT, Iris Karvonen

VTT, Outi Kettunen VTT, Stefano Perini – POLIMI, Chiara Galbusera - POLIMI

Dissemination: Public

Contributing to: T 2.3 Verification & Validation generic Assessment Package

Date: 30/09/2013

Revision: V1.0 - Final



2

DISSEMINATION: PUBLIC FITMAN CONSORTIUM

VERSION HISTORY

VERSION DATE NOTES AND COMMENTS

0.1 19/05/2013 INITIAL TOC – G.TAVOLA - POLIMI

0.2 25/05/2013 AFTER HELSINKI MEETING AT VTT 21,22 MAY WITH PROPOSED

OWNERSHIP

0.5 07/08/2013 FINAL TOC + REQUESTS OF CONTRIBUTIONS FROM THE

PARTNERS

0.6 14/08/2013 VERSION FOR INITIAL REVIEW AND REQUEST OF

CONTRIBUTION

0.7 23/08/2013 SECOND VERSION FOR INITIAL REVIEW AND REQUEST OF

CONTRIBUTION

0.9 26/08/2013 VERSION FOR INTERNAL REVIEW

1.0 26/09/2013 FINAL VERSION FOR INTERNAL PEER REVIEW

FINAL 30/09/2013 FINAL VERSION RELEASED

DELIVERABLE PEER REVIEW SUMMARY

ID Comments Addressed ()

Answered (A)

1

Very nice document and comprehensive.

Highlight two-three major take-away

messages.

(Chap.6)

2

Provide clear indication/estimation of

ICT SMEs effort to implement/deploy

FITMAN V&V.

Note: I am not arguing that V&V should

be skipped. On the contrary, the proper

resourcing should be put in place.

(Chap.4.2.4)

3

It would be nice if V&V method can be

put in context of Software Capability

Maturity Models (CMM) or other

frameworks. A connection to

development methodologies would be

nice.

The issue has not be taken into account in the

previous D2.1. As a consequence, it cannot be

considered as one of our specific operating

objectives, even if the investigation of this

aspect could surely be interesting. (A)

4

In the introduction indicate more clearly

when and where (in the development

cycle) end-user groups should be

involved and the different forms in which

they will participate. The importance of

end-user (manufacturing) involvement

should be stressed further.

Also try to distinguish between barriers,

methods to operate with SME, Large

A more specific point has been added in the

proper section, i.e. Paragraph 3.3.1.1 (A)



3


Industry (at least the differences observed

in FITMAN trials).

5

OK, methods and criteria clear also for

industry. Implementation method is

simple and the detail per GEs, SEs and

TSCs is useful to organize data collection

6

It would be advisable to indicate how to

measure the results: scale, absolute

number, % of improvement (this last one

very useful for “as is” vs “to be”

comparison)

In some cases the single performance

cannot be enucleated and overall result is

greater than the sum of the parts,

measured per GEs, SEs and TSCs. It

could be useful an aggregated measure at

business scenario level*

The indication of ROI coming from the

application of the scenario could be

useful to give a single standard and

common economic indication of the

expected benefit from each scenario

The issue will be deeply addressed in the

following phase , i.e. the Instantiation phase. For

every Trial and Trial’s Business Scenario will be

defined the more suitable Business Performance

Indicators. (A)

7

Update forms in relation with point 6

comments, % of benefit from “as is” vs

“to be” could be advisable

See Comment 6. (A)



4


Table of Contents

EXECUTIVE SUMMARY ................................................................................................................................... 6

1. INTRODUCTION ....................................................................................................................................... 8

1.1 FITMAN Generic V&V Package - Objectives & Scope ....................................................................... 8 1.2 Structure of the document ..................................................................................................................... 9 1.3. Contribution to other WPs and Deliverables................................................................................... 9 1.4 FITMAN V&V Generic Method and Criteria Implementation ........................................................... 10 1.5 FITMAN V&V Business and Technical Indicators Implementation ................................................... 14 1.6 FITMAN Generic V&V Package Reference Model ............................................................................ 15

2. V&V PACKAGES STATE OF THE ART .............................................................................................. 21

2.1 V&V methodological solutions ........................................................................................................... 21 2.2 V&V in other FI-PPP projects ........................................................................................................... 22 2.3 V&V technological solutions .............................................................................................................. 23 2.4 V&V SotA findings.............................................................................................................................. 27

3. FITMAN V&V PACKAGE ...................................................................................................................... 28

3.1 V&V Package functional and non-functional requirements ............................................................... 28 3.2 V&V Package methodological foundations ........................................................................................ 29 3.3 V&V Package framework definition and description ......................................................................... 38

4. V&V PACKAGE TOOLS AND TECHNOLOGIES .............................................................................. 66

4.1 V&V Communication Package ........................................................................................................... 66 4.2 Data Collection .................................................................................................................................. 66 4.3 Data Elaboration, Presentation and Distribution .............................................................................. 71 4.4 V&V Package instantiation approach ................................................................................................ 72

5. CONFIDENTIALITY ASPECTS ............................................................................................................ 74

6. CONCLUSIONS ........................................................................................................................................ 75

7. APPENDIXES ............................................................................................................................................ 76

7.1 Appendix 1 - Glossary ........................................................................................................................ 76 7.2 Appendix 2 – A practical example ...................................................................................................... 76 7.3 Appendix 3 – Acronyms and abbreviations ........................................................................................ 78 7.4 Appendix 4 – SurveyMonkey tool examples........................................................................................ 78

8. REFERENCES ........................................................................................................................................... 87



5


FIGURES

Figure 1 FITMAN V&V Package Reference Model ............................................................... 18

Figure 2 V&V Package match with the Reference Architecture.............................................. 19 Figure 3 Example of Software Components for a Trial’s Business Scenario .......................... 39

Figure 4 Data Collection’s Solutions and related Steps ........................................................... 40 Figure 5 Data Collection Setup Process ................................................................................... 41 Figure 6 Example of the Self-certification Form’s structure for a Trial’s Business Scenario . 47 Figure 7 Example of the PIs Measurement Form’s structure for a Trial’s Business Scenario. 49 Figure 8 Example of the Community-based Form structure for a Trial’s Business Scenario .. 50

Figure 9 Example of triplets Objectives, DV/AV, PI for TRW ............................................... 54 Figure 10 Example of triplets Objectives, DV/AV, PI for Agusta Westland ........................... 56 Figure 11 Example of triplets Objectives, DV/AV, PI for Geoloc System .............................. 58 Figure 12 FITMAN V&V Package support page ..................................................................... 66 Figure 13 Data Collection Forms conceptual schema for a Trial’s Business Scenario ............ 67

Figure 14 Example of a SE’s Self-certification Sub-form ....................................................... 68 Figure 15 Example of a PIs Sub-form for a SE ........................................................................ 69 Figure 16 Example of a Business Sub-form for PIs Measurement .......................................... 69

Figure 17 Example of an IT Sub-form for PIs Measurement ................................................... 70 Figure 18 Example of a Community-based Sub-form.............................................................. 70 Figure 19 MonkeySurvey Export function ............................................................................... 71

Figure 20 SurveyMonkey web storage function ...................................................................... 71

TABLES

Table 1 FITMAN V&V Method: Product Specific Perspective .............................................. 12

Table 2 FITMAN V&V Method: Trial’s Business Scenario Specific Perspective .................. 13

Table 3 Methodological solution for Data Collection, Elaboration and Presentation .............. 23

Table 4 SurveyMonkey vs Google Form ................................................................................. 25 Table 5 Comparison between MS Access, My SQL and MS Excel ........................................ 27 Table 6 Links among Trials’ Business Scenarios, Steps, Criteria Categories, Criteria and their

nature ........................................................................................................................................ 32 Table 7 PIs associated to each Criteria and classified by their own nature .............................. 35

Table 8 Kinds of Forms linked to the 7 Steps of the V&V Methodology ................................ 37 Table 9 Data Collection’s Solutions ......................................................................................... 39 Table 10 Data Collection Form structure ................................................................................. 44 Table 11 Example of Self-certification Form for GEs ............................................................. 46

Table 12 Example of Self-certification Form for SEs, TSCs and TICs ................................... 46 Table 13 Self-certification Form’s FITMAN Stakeholders ..................................................... 48

Table 14 Logical structure of Data Collection for a generic Quantitative IT or Business

Performance Indicator .............................................................................................................. 50 Table 15 PIs Measurement Form’s FITMAN Stakeholders ..................................................... 50 Table 16 Community-based Form’s FITMAN Stakeholders ................................................... 51 Table 17 Supplementary Community-based Techniques ......................................................... 52

Table 18 Example of Views of Self-certification results ......................................................... 59 Table 19 Example of Views of Self-certification results for each Trial’s Business Scenario.. 61 Table 20 View of PIs Measurement Form results .................................................................... 61 Table 21 Example of view of Community-based results ......................................................... 62 Table 22 Example of view of supplementary Community-based results ................................. 63

Table 23 Support partners for the Trials ................................................................................... 73

Table 24 Confidentiality level according to the kind of data ................................................... 74



6


Executive Summary

FITMAN is an FI-PPP phase II project aiming at providing the FI-PPP with a set of industry-

led Use Case Trials in the Smart, Digital and Virtual Factories of the Future domains, in order

to test and assess the suitability, openness and flexibility of FI-WARE Generic Enablers

(GEs). In this context, the overall objective of WP2 is to develop a method for the evaluation

and assessment of the aforementioned FITMAN Use Case Trials.

In the light of this, the main objective of the Deliverable D2.3 is the consolidation of the

developed V&V Generic Method, Criteria, Technical and Business Performance Indicators

into a generic package, the generic FITMAN Verification and Validation Assessment

Package. Taking as inputs the results of the previous Deliverable D2.1 [1] and Deliverable

D2.2 [2], the final goal of the present document is to integrate all the necessary Indicators and

their metadata, to support the application of these Indicators and the documentation and

visualization of the assessment results and to allow the context-based comparison with other

Use Case Trials. The Deliverable D2.3 also constitutes the input for the subsequent

Deliverable D2.4 (FITMAN V&V Assessment Package Instantiations per Trial), where the

generic FITMAN Verification and Validation Assessment Package will effectively be

instantiated in each of the eleven FITMAN Use Case Trials.

In essence, the present Deliverable D2.3 develops, describes and discusses in detail both from

a methodological and technological perspective the generic FITMAN Verification and

Validation Assessment Package that will be subsequently instantiated in the eleven FITMAN

Use Case Trials in the Deliverable D2.4. It constitutes the generic integrated reference model

for the practical achievement of the three following objectives (according to the Deliverable

D2.1 [1]):

the Verification and Validation of the Generic Enablers (GEs);

the Verification and Validation of the Specific Enablers (SEs), Trial Specific

Components (TSCs) and Trial Interface Components (TICs) under development

during the FITMAN project;

the Validation of the complete solution which will be developed for each Use Case

Trial’s Business Scenario in the framework of the FITMAN project.

In order to reach this goals, the following steps have been followed:

1. the 7-Steps FITMAN V&V Methodology developed in Deliverable D2.1 [1] has been

integrated with the IT and Business Performance Indicators established in Deliverable

D2.2 [2] in a common logical framework;

2. the three different kinds of Forms to adopt in order to collect the necessary data for the

FITMAN V&V Process, i.e. Self-certification Form, PIs Measurement Form and

Community-based Form have been identified and defined from a methodological point

of view;

3. the aforementioned three kinds of Forms have been integrated in the conceptual

framework of the generic FITMAN V&V Assessment Package, which constitutes of

three main Sections, i.e. the Technology Section, the General Information Section and

the Instructions and Support Section. The last two Sections form in particular the

FITMAN V&V Communication Package, i.e. all the information and instructions

needed to manage and support a correct and satisfying use of the Forms themselves;

4. the whole conceptual FITMAN V&V Assessment Package has been practically

implemented on (or linked to) the FITMAN Website. The Technology Section has

been in particular managed by means of the online survey tool SurveyMonkey and

will be available to the Trial Owners through differentiated log-in systems.



7


Furthermore, the FITMAN V&V Assessment Package’s Technology Section’s “lifecycle” has

been properly taken into account. In fact, three different chronological phases have been

individuated and defined in the exploitation of the FITMAN V&V Assessment Package’s

Technology Section, always both from a methodological and technological point of view:

a. Data Collection, regarding the correct and efficient gathering of all the kinds of data

by means of different kinds of Forms, implemented thanks to the online survey tool

SurveyMonkey directly on the web. In this phase, also the ECOGRAI Simplified

Methodology is integrated and deployed for Business Performance Indicators (see

Deliverable D2.2 [2]);

b. Data Elaboration, regarding all the possible guidelines for the subsequent data analysis

and aggregation, such as the definition of the different dimensions of analysis and of

the ways to compare the data, implementable thanks to MS Excel and completely

integrated with the online survey tool SurveyMonkey;

c. Data Presentation, regarding all the possible guidelines for the report and discussion of

the final results of the FITMAN V&V Process, such as the different ways to present

the final data and the selection of the more representative findings, implementable

thanks to PDF files and completely integrated with MS Excel. Raw data will in fact

also be available in MS Excel and MySQL format for free customized elaborations

managed by each Use Case Trial.

The generic FITMAN V&V Assessment Package developed is consistent with the FITMAN

Reference Architecture of the FITMAN project. It is hence able to highlight and keep

granularity on:

Reference Architecture levels (i.e. Future Internet Cloud, Business Ecosystem,

Individual Factory/Enterprise)

GEs, SEs, TSCs, TICs

Smart/Digital/Virtual domains

Small Medium Enterprise/Large Enterprise

Trials, Trials’ Business Scenarios, V&V Methodology Steps, Verification and

Validation Techniques, Business and Technical Indicators

Customers Queries can be defined by users thanks to the open data format provided

The components of V&V Generic Package are:

V&V Methodology for instantiation of Assessment procedures (this document)

A technology environment (the tool) for carrying out data collection surveys1

A documentation set for informational, training and support purposes, physically

available on the FITMAN portal2

1 www.surveymonkey.com user: FITMAN_V&V password: fitman1

2 http://www.fitman-fi.eu/intranet/wp-folders/wp2-fitman-verification-validation/test-environment/fitman-

support-page for user enabler to access FITMAN portal

http://www.surveymonkey.com/

http://www.fitman-fi.eu/intranet/wp-folders/wp2-fitman-verification-validation/test-environment/fitman-support-page




8


1. Introduction

1.1 FITMAN Generic V&V Package - Objectives & Scope The generic FITMAN Verification & Validation Assessment Package (D2.3) addresses the

definition and implementation of an effective tool for Trials’ Business Scenarios assessment.

The generic FITMAN Verification & Validation Assessment Package actually consolidates

Criteria, Tools, Methods and Performance Indicators defined in previous tasks:

T2.1 V&V Generic Method and Criteria Identification (NTUA)

T2.2 V&V Technical and Business Performance Indicators Definition (IVLAB)

The generic package defined in D2.3, will be then instantiated to each Trial’s Business

Scenario by T2.4 (Instantiation of V&V Assessment Package per Trial’s Business Scenario)

according to the characteristics of each of them.

The generic FITMAN Verification & Validation Assessment Package (D2.3) has the

objective to be a lean and agile tool for assessment of the Trials’ Business Scenarios results.

With this in mind these are the key characteristics implemented:

1. Easy to understand and to implement

2. No additional effort required by the development team without a clear return

3. Able to gather significant data for an assessment of the Trials’ Business Scenarios

results

4. Possible utilization for other Phase II and Phase III projects

Key components of the generic Verification & Validation Assessment Package are:

I. A clear and exhaustive reference framework which put together in a well-defined and

systematic way the D2.1 [1] 7-Step Methodology, the D2.2 [2] IT and Business

Performance Indicators and necessary integrative systems to complete and make

effective the FITMAN V&V Process.

II. Based on the Methodology defined in D2.1 [1] and the Indicators identified in D2.2

[2] a set of data collection forms and tools are made available. These tools are mainly

implemented via an online based tool (i.e. SurveyMonkey:

https://it.surveymonkey.com/). A set of customized (per each Trial’s Business

Scenario) Forms are made available to collect consistently with the Generic

Methodology different classes of information. Key advantages of such a web based

tool are:

Easier data collection - no excel forms going back and forward

Survey process monitoring – data inserted are immediately available for checking

and support

Data Availability – Data are automatically stored on a web based repository for

consolidation and presentation

Low costs

Possibility to implement different survey approach as repetitive data collection,

crowd survey, time-boxed survey, etc.

III. A set of functions that (based on inputs from D2.2 [2]) will made data available for

their visualization and consequent elaboration. A set of graphical representation of the



9


data will also be made available. Finally an open format (i.e. MS Excel, MySQL)

repository is made available for further analysis.

IV. A V&V Communication Package, for a clear presentation of the tool to stakeholders.

This component is very important to gain the acceptance and support of people

involved (both technical and business owners). In the V&V Package are presented the

methodological approach, the tools and techniques utilized, the expectations and

results envisaged and what the data generated for eventual Trial’s benefits are.

Data Collection is carried out consistently with the FITMAN Reference Architecture (see

Figure 2 V&V Package match with the Reference Architecture) and with a granularity able to

enable aggregation for significant classes (Smart/Virtual/Digital, Large/Small, etc.).

ECOGRAI Simplified Methodology for Business Performance Indicators is also fully

supported by the generic V&V Package.

Specific attention will be paid to avoid confidentiality issues for the collected data for their

sensitiveness (see Chapter 5).

D2.3 is not aiming to consolidate experience of Trials (T7.1) or to do their comparative

evaluation (T8.1), even if provides tool and part of necessary knowledge to contribute on that.

1.2 Structure of the document

In Chapter 1, FITMAN V&V Package objectives and scope in agreement with the

DOW and the contribution to other WPs and to other Deliverables are presented, and

subsequently summaries of FITMAN V&V generic method and criteria identification

and of FITMAN V&V Technical and Business Performance Indicators definition are

provided. Finally, the FITMAN V&V Package Reference Model is outlined.

In Chapter 2, the V&V Packages SotA is addressed, i.e. V&V methodological

solutions, V&V in other FI-PPP projects, V&V technological solutions and V&V

SotA findings.

In Chapter 3, the generic FITMAN V&V Package is presented in detail. In particular,

starting from V&V Package functional and non-functional requirements (from

Deliverable D2.1 [1]), the V&V Package methodological foundations and the V&V

Package framework definition and description are explained.

In Chapter 4, the V&V Package tools and technologies and the V&V Package

instantiation approach for T2.4 are addressed.

In Chapter 5 the Confidentiality Aspects for the Trials are analyzed.

In Chapter 6 are reported the conclusions and the possible evolutions of the Generic

V&V Package are presented.

1.3. Contribution to other WPs and Deliverables The work we have developed within T2.3 summarized in D2.3 is an important research

activity (together with D2.1 [1] and D2.2 [2]) to establish a sound foundation to Trials’

objective assessment. The outcomes of this task can also be considered a contribution for

other Trials (in Phases II and Phase III). The results obtained in this document and related

deliverables (survey tool implementation and communication/training material) will be

connected and will contribute directly to other work packages and to specific deliverables,

apart from WP2 FITMAN Verification & Validation.

Taking it into account, these results will be connected with WP7, defining the foundation for

gathering of outcomes from different Trials in order to elicit and consolidate Lesson Learned,

Recommendations and Best Practices (T7.1). According to this, the Use Case Trials’



10


experiences will cover a broad range of contents in terms of industrial domains, business

models, operational approaches and management attitudes that will ensure a remarkable data

base of information. The deliverable will contribute to get a set of goals in T7.1, setting up an

operational environment for information collection and storing, according to defined methods

and methodology, data gathering form Use Case Trials and organization, consolidation and

presentation of performance indicators and other relevant data.

T2.3 and D2.3 will also contribute to WP8, more specifically to T8.1 FITMAN Use Case

Trials comparative evaluation, providing appropriate information and significant hints from

heterogeneous industrial/business environments, as well to T8.2 FITMAN Expanded Trials

Proposition, in charge of evaluating the feasibility for expanded Trials.

Concerning the contribution to other deliverables, D2.3 is mainly linked to D2.4 which is the

deliverable in charge to instantiate the Generic V&V Package for each single Trial. Generic

tool and communication materials defined as a frame in D2.3 will be customized for each

Trial, based on specific aspects.

1.4 FITMAN V&V Generic Method and Criteria Implementation To ensure high quality outcomes, an intense FI-PPP phase II project like FITMAN has

defined an all-inclusive framework for verifying, validating and evaluating a software product

from its conception to final release and implementation in real-life, trial settings. As D2.1 [1]

states, the goal is to provide FITMAN with the appropriate methodology in order to verify

that the FI-WARE generic and FITMAN specific enablers (as well as Trial Specific

Components) satisfy the technological, platform and architectural integration requirements

imposed; validate that the FI-WARE generic and FITMAN specific enablers (as well as Trial

Specific Components) satisfy the requirements of Smart-Digital-Virtual use case trials; and

identify the evaluation and assessment criteria to be used in all Use Case Trials, taking into

account sustainability (STEEP) and future business benefits.

The FITMAN V&V methodology introduces a new and innovative way of performing V&V

activities in various ways by:

Bringing together and getting the best of breed of the agile and waterfall software

engineering philosophies which have been essentially dealt as contradictory trends and

schools of thought until now.

Infusing a crowd assessment mentality within the V&V activities, under each

methodological step from code verification to business validation.

Balancing and bridging the business and technical perspectives in an effort to assess

the software and “fit for purpose” requirements of trials and to evaluate the overall

software’s added value.

Attempting a categorization and classification of a number of tools and techniques

which in most cases had no clear distinction between V&V and Evaluation aspects.

Providing proper sets of criteria covering all the verification and validation steps

described in the developed methodology. The business validation criteria are

categorized into two sets: the generic criteria based on SCOR and the specific criteria

based on the trials business requirements. Indicative categories of business generic

criteria concern customer reliability, agility, responsiveness, cost and assets, while

indicative categories of business specific criteria involve cost, sustainability,

innovation, efficiency, flexibility and quality. IT V&V criteria involve FITMAN-

relevant categories (including openness and versatility) and more generic categories

like functionality, maintainability, usability, reliability, efficiency, and portability

based on the ISO 9126 standard.



11


Elaborating on a set of guidelines to assist all actors involved in implementing the

FITMAN V&V methodology in practice. An initial set of guidelines for the

application of the V&V method and the creation of the V&V package has been created

while with regard to the trials, a “V&V decalogue” has been constructed in order to

infuse a very practical and applicable philosophy to the V&V steps.

The developed V&V method is essentially divided into two perspectives:

The trial specific perspective (T) which assesses whether the IT and business

requirements and domain’s needs are met, and

The product-specific perspective (P) which describes how to verify and validate the

product (i.e. the Generic Enabler (GE), the Specific Enabler (SE), the Trial Solution

Component (TSC) or the Tral Interface Component (TIC)) during its development.

In particular, the FITMAN V&V method is elaborated step-by-step, providing, apart from the

general description of the procedure, the potential techniques to be employed, the

stakeholders to be engaged, and the potential crowd engagement methods to be applied, and

featuring:

I. Code Verification (P-1) to ensure functionality, correctness, reliability, and

robustness of code.

II. Model Verification (P-2) to coordinate the alignment between design and

requirements, as well as between design and code.

III. Backlog Verification (P-3) to determine whether the requirements of the product

after each sprint are met.

IV. Release Verification (P-4) to determine whether the requirements of the final

product release are met.

V. Product Validation (P-5) to examine whether the product satisfies intended use

and user needs.

VI. Trial Solution Validation (T-1) to guarantee that the overall Trial’s Business

Scenario solution satisfies intended use and user needs.

VII. Business Validation (T-2) to assess whether the overall Trial’s Business Scenario

solution eventually offers sufficient added value to the Trial.

The method is summarized in the following Table 1 and Table 2 (the first one concerns the

Product-specific perspective Steps and the second one the Trial’s Business Scenario-specific

perspective Steps). It needs to be clarified that all V&V Steps have aspects and perspectives

that may require a “Self-certification” approach depending on the needs of each Trial’s

Business Scenario.




Activity “Self-certification” approach Stakeholders Crowd Engagement

Method

Mandatory Step for

Recommended

Technique

Alternative

Techniques

FITMAN

Stakeholders

External

Stakeholders

Code

Verification

White Box

Testing

Unit Testing Development

Team

- - SEs, TSCs, TICs

Model

Verification

Traceability

Analysis

Walkthrough Development

Team, Sprint

Master

Phase I and

Phase II

Projects and

Public

Physical or online

workshops, Readily-

available prototyping

applications, Social

deliberation platforms

SEs, TSCs, TICs

Backlog

Verification

Regression

Testing

Alpha

Testing,

White-box

Testing

Development

Team, Sprint

Master,

Product

Owner

Phase I and

Phase II and

Public

Deliberation and feedback

tools for ex-ante

crowdsourcing, Dedicated

IT tools for ex-post

crowdsourcing

SEs, TSCs, TICs

Release

Verification

Regression

Testing

Alpha

Testing,

White-box

Testing

Development

Team, Sprint

Master,

Product

Owner

Phase I and

Phase II and

Public


tools for ex-ante



crowdsourcing


Product

Validation

Black Box

Testing for

Validation

System

Testing, User

Acceptance

Testing,

Inspection

Product

Owner (and

Trial

Supporting

Team)

Phase I and

Phase II and

Public

Online deliberation and

feedback tools, Open calls

for testing scenarios/ trials,

Workshops and specific

working groups,

Traditional online survey

tools, Social networks


Table 1 FITMAN V&V Method: Product Specific Perspective



13


Activity Recommended

Technique

Alternative

Techniques

Stakeholders Crowd Engagement

Method Mandatory Step for FITMAN

Stakeholders

External

Stakeholders

Trial

Solution

Validation

User

Acceptance or

Beta Testing

Black Box

Testing,

Inspection,

Traceability

Assessment.

Trial Solution

Owner, Trial

Team

Public

Enterprise collaboration

environments, Traditional

methods (like

questionnaires)

Integrated Trial’s Business

Scenario Solution

Business

Validation

Simplified

ECOGRAI

Methodology

for decisional

models

Balance

Scorecard,

Manufacturin

g and Value

Chains’

SCOR-

VCOR

methods

Trial Team Public

Open collaboration and

innovation platforms,


feedback tools, Physical

and online workshops,


tools, Game-like


networks

Integrated Trial’s Business

Scenario Solution

Table 2 FITMAN V&V Method: Trial’s Business Scenario Specific Perspective


30/09/2013 Deliverable D2.3 – V1.0


1.5 FITMAN V&V Business and Technical Indicators Implementation 1.5.1 Scope and results of T2.2

The goal of the WP 2 FITMAN Verification & Validation (V&V) Method is to develop a

method for the evaluation and assessment of the FITMAN Trials, regarding various aspects.

The goal of the Deliverable D2.2 [2] was to identify and define a selection of Business

Performance Indicators and Technical Indicators for the FITMAN Verification & Validation

Methodology.

The main results of the Deliverable D2.2 [2] are the following:

Definition of concepts connected with Business Performance Indicators and Technical

Indicators.

o For Performance Indicators (PIs): the method to define PIs is the simplify

method ECOGRAI

The simplified version of ECOGRAI has only three phases instead of 6 phases

(see D2.2 [2]):

First Phase: Description of the system in which the performance indicators

will be defined.

For that we use System Modeling .

To describe a system using System Modeling we need to determine:

The elements which compose the system and the relations

between these elements

The objectives assigned to the system,

The functions which allow to reach the objectives

The processes which support the dynamic transformations

The boundary which delimits the elements which don’t belong

to the system.

The dynamic of evolution of the system.

Second Phase: According to the objective of the system the owner of the

system determines the potential actions to reach these objectives (called

Decision Variables (DV) or Action Variables (AV)).

In this phase we define also:

the constraints, which represent the limits of the DV/AV.

the criteria, in order to choose the DV/AV.

Third Phase: the performance indicators indicate or characterize the reaching

of the objectives by using the DV/AV.

o For Technical indicators: Validation, Verification,

In addition D2.2 [2] supplies a list of potential PIs, this list has been determined starting from

the ENAPS list of 117 generic Indicators which were modified in order to fit with the domain

of FITMAN.



15


In the last part of the deliverable, we present examples to determine Technical and Business

Performance Indicators. We have kept the initial proposition to consider Technical Indicators

more oriented to evaluate GEs, SEs, TSCs, TICs and IT platforms defined by the V&V

method and the Business Performance Indicators to evaluate Trials’ Business Scenarios

performance by a simplified version of ECOGRAI.

This initial work, after three months, demonstrates the suitability of the proposed methods.

The work must be pursued in order to generalize the implementation of ECOGRAI in the

Trials’ Business Scenarios. Particularly we prepare a list of Indicators derived from ENAPS

project (see Appendix 2 – A practical example) which will be adapted to FITMAN Trials.

1.5.2 Implementation method

This paragraph describes the implementation of ECOGRAI in the Trial’s Business Scenario.

The ECOGRAI method is a participative method, and the results are due to constructive

discussions between the actors of the system.

The first phase is to collect the information on the system (the elements are given in

ECOGRAI method described above).

Already the main elements have been described in the Thematic Handbook: it is necessary to

complete this description. The Trial Owner must be involved. The ECOGRAI specialist will

prepare some questions in order to orient the complementary collect.

The second phase is to determine the Action Variables (AV) or Decision Variable (DV) if the

action is the result of a decision. It will be necessary also to evaluate the PIs for such AV/DV

because we must be sure that the implementation phase is performed in the right way based

on the “algorithm” already describe: “How to evaluate the result of the AV/DV in the

reaching of the objective. This interaction could be organized by face to face meeting or by

TEL CONF. It is important that the human interaction will be maintained.

The third phase will require an interaction between the Trial Owner and the ECOGRAI

specialist to determine exactly the PIs.

It is necessary to establish a planning in order to be sure that the elements will be collected

and that the interaction will be organized in the best way. A document will be produced in

advance.

The originality of the ECOGRAI method is the interaction between the Owner of the Trial

and the specialist. The result is a clear understanding by the Owner of the Trial on the

meaning of the PIs.

A user document will be produced to describe the phases and the calculation of the PIs.

We underline that the PIs determination is different from Benchmarking. The PIs can be used

in Benchmarking if the basic conditions to benchmark are satisfied. It is another subject.

Concerning the frequency of the evaluation of the PIs it depends on the dynamic of evolution

of the system. GRAI model is a good tool to determine the periodicity: it is necessary to

determine the level of decision; short term (Operational Level), middle term (Tactical Level)

and long term (Strategic Level).

1.6 FITMAN Generic V&V Package Reference Model As already highlighted, the aim of the generic FITMAN V&V Package is the integration and

consolidation of the Evaluation and Assessment Criteria developed in D2.1 [1] and of

Technical and Business Performance Indicators established in D2.2 [2] in a generic unique

framework. In doing this, the main inputs which have been taken into account in order to

build the FITMAN V&V Package are:

the FITMAN 7-Steps V&V Methodology (D2.1)

the FITMAN Business Validation Criteria and the IT Verification Criteria (D2.1)



16


the Recommendations for the FITMAN V&V Package functions (i.e. functional and

non-functional requirements) (D2.1)

the FITMAN Technical (IT) Indicators (D2.2)

the FITMAN Business Performance Indicators (D2.2)

the methodologies and the modalities to analyze and elaborate the abovementioned

Indicators (e.g. ECOGRAI Simplified Methodology) (D2.2)

These elements have been properly combined and re-elaborated in order to be put together in

an original and effective structure. In this respect, both methodological and technological

issues related to the generic FITMAN V&V Package have been addressed in order to provide

to the following T2.4 all the necessary strategies and tools necessary for its correct

instantiation in the FITMAN eleven Trials.

As it is possible to note in Figure 1, the generic FITMAN V&V Package has been obtained

through and articulated process composed by three main passages:

A) Creation of a FITMAN V&V Common Logical Framework, composed by a Product-

specific and a Trial’s Business Scenario-specific part, based on the combination of the

results of D2.1 [1] and D2.2 [2]. The former has been individuated according to the

first five Product-specific Steps established in the FITMAN V&V Methodology (D2.1

[1]), allowing the separate Verification and Validation of the already developed FI-

WARE GEs and of the rising FITMAN SEs, TSCs and TICs. The IT Criteria

necessary for this first part have been provided by D2.1 [1]. The latter has been

identified according to the last two Trial’s Business Scenario-specific Steps

individuated in the second part of the FITMAN V&V Methodology (D2.1 [1]),

allowing the specific Validation of the overall Trial’s Business Scenario software

solutions, considered as original compositions of different GEs, SEs, TSCs and TICs.

The IT and Business Performance Indicators necessary for the two parts are the results

of the efforts of D2.2 [2], which also linked them to the proper underlying IT and

Business Criteria defined in D2.1 [1];

B) Identification of three different kinds of Data Collection modalities, i.e. Self-

certification, PIs Measurement and Community-based Forms, for the necessary data

for the FITMAN V&V Process. In particular, the Self-certification Form has been

established in order to cover the IT Criteria identified for the five Product-specific

Steps, while the PIs Measurement Form and the Community-based Form in order to

cover the IT and Business Performance Indicators identified for the fifth Product-

specific Step and the Trial’s Business Scenario-specific Steps;

C) Creation and implementation of the generic FITMAN V&V Package, i.e. the

practical implementation and orchestration of all the methodological and technological

elements in a unique framework. The generic FITMAN V&V Package has been hence

organized in three different sections:

1. the Technology Section, including all the above-mentioned kinds of Forms and

published and managed by means of the specific data collection software

SurveyMonkey;

2. the General Information Section, published directly on the FITMAN Website,

explaining to all the possible beneficiaries the main points related to all the

different aspects of the FITMAN V&V Package;

3. the Instructions and Support Section, published directly on the FITMAN

Website, explaining in detail to all the possible beneficiaries the modalities for

the correct implementation and use of the FITMAN V&V Package in the

eleven Trials.



17


The last two Sections, put together, contribute to form the so-called “FITMAN V&V

Communication Package”, in order to distinguish the pure “Forms-based” Section (i.e. the

Technology one) from its consequent “meta-structure”, i.e. all the information and

instructions needed to manage and support a correct and satisfying use of the Forms

themselves.

Then, a further important dimension regarding the Technology Section “lifecycle” has been

considered. In fact, three different chronological phases in the exploitation of the V&V

Package Technology Section have been individuated and properly defined:

i. Data Collection, i.e. the correct and efficient gathering of all the kinds of data by

means of different kinds of Forms, implemented thanks to SurveyMonkey directly

on the web. For Business Performance Indicators, the ECOGRAI Simplified

Methodology has been adopted and integrated (see Deliverable D2.2 [2]);

ii. Data Elaboration, i.e. all the methodological and technological aspects of data

analysis and aggregation (e.g. definition of the different dimensions of analysis,

ways to aggregate and compare data, practical implementation in MS Excel);

iii. Data Presentation, i.e. the different modalities to report and discuss the final

results of the FITMAN V&V Process, both from a methodological and a

technological point of view (e.g. different ways to present the final data, selection

of the more representative findings, practical implementation in PDF files).

All these considerations are clearly resumed and depicted in the following Figure 1. Further

explanations of all the issues presented will be addressed in the proper sections.



18


Figure 1 FITMAN V&V Package Reference Model

It is evident from Figure 1 that the reference element for all the necessary V&V operations is

the Trial’s Business Scenario, defined as “each of the specific selected cases within the

TRIAL, in which we can generically improve the current outcome by applying the Future

Internet Core Platform”. In general, each of the eleven Trials can in fact involves one or more

different Business Scenarios, which have to be verified and validated independently.



19


We know that all the V&V activities must be consistent with the FITMAN Reference

Architecture and the FITMAN Project, providing inputs to Phase III. As a consequence, the

generic FITMAN V&V Package is able to highlight and keep granularity on:

Reference Architecture levels (i.e. Future Internet Cloud, Business Ecosystem,

Individual Factory/Enterprise)


Smart/Digital/Virtual domains


Trials

Trials’ Business Scenarios

In particular, the three different levels of the Reference Architecture have been properly taken

into account. It is in fact possible to associate to each level of the Architecture a specific

category of Software Component (i.e. GEs, SEs, TSCs and TICs) or transversely for all the

three levels the overall Trials’ Business Scenarios software solutions that will be tested (we

have to remember that each Trial’s Business Scenario’s package implies a different

combinations of GEs, SEs, TSCs and TICs) (see Deliverable D2.1 [1]). The following Figure

2 depicts this concept, with also the correct reference to the specific 7 Steps of the FITMAN

V&V Methodology:

Figure 2 V&V Package match with the Reference Architecture

At the same time the V&V Package process needs to provide to WP7 and WP8 feedback

information to Trials at different levels of granularity. For this reason, other dimensions of



20


aggregation of the data have been taken into account in the generic FITMAN V&V Package,

i.e.:

Trials

Trials’ Business Scenarios

Steps

Verification and Validation Techniques (eventually)

Indicators (eventually)

All the issues related to the granularity and the aggregation of the data will be properly

deepened in the following of the present document, in particular when talking about Data

Elaboration and Data Presentation.



21


2. V&V Packages State of the Art

As already highlighted in the Deliverable D.2.1 [1], there is no common agreement in

literature on the various definitions among the terms verification, validation, assessment and

evaluation. Furthermore, no “holistic” V&V Approach has been developed yet. For these

reasons, for what concerns the general software V&V packages, solutions or systems, only

partial considerations or blurry indications have been found. In particular, the pure

technological aspects are rarely considered, and seem to change extensively according to the

single situations and needs of the testers. In the light of all these elements, a brief but

representative SotA research about software V&V Packages is addressed, considering

respectively the methodological and the technological aspects of the issue. Moreover, the

software V&V solutions already adopted in other FI-PPP projects are briefly investigated.

2.1 V&V methodological solutions From the methodological point of view, two general integrated evaluation methods are the

most widespread both in literature and in practice, i.e. the analytic hierarchy process (AHP)

and the weighted scoring method (WSM) (Jadhav and Sonar [3]). The former is a decision

support system based on the comparison of different alternatives thanks to the assignment of

specific criteria and marks to each of them, while the second constitutes a simplified version

of points system that can be used and elaborated according to the specific situation. Both the

methodologies are focused on the assignment, even if in lightly different ways, of system of

marks able to produce synthetic results of the single alternative or of a part of it, allowing the

rapid comparison of different elements and sub-elements of the particular software

applications considered.

Some scientific examples of the first approach can be found in particular in Samoladas et al.

[4], who developed the SQO-OSS Quality Model, which evaluates source code and

community processes, and in Saaty [5], who established on the other hand the theoretical

foundations of the method. The second approach is practically addressed for example in

Harrison et al. [6] , who developed MOOD, a system of metrics for the specific evaluation of

object-oriented software, and in Bandor [7], who proposed a very operating decision analysis

spreadsheet based on different weights for the various decision criteria previously established.

Beyond the two aforementioned approaches, many other types of solutions, modalities,

techniques or guidelines for integrated V&V systems have been proposed. Pure user-based

software evaluation methods, i.e. logged data, questionnaires, interviews and verbal protocol

analysis, are studied by Henderson et al. Lin et al. further investigated the theme of usability

specifically by means of questionnaires, developing the Purdue Usability Testing

Questionnaire (PUTQ). Chin et al. [8] explored another time the measuring of user

satisfaction, focusing in particular on the human-computer interface. Also in this case the

final goal has remained the building of a questionnaire to submit to users, i.e. the

Questionnaire for User Interface Satisfaction (QUIS).

A more theoretical approach is adopted by Vieira and Goncalves [9] , whose VALTE

Methodology for validation and testing of supply chain software components analyzes the

modalities to generate the evaluation criteria for the functional and non-functional features,

and by Boloix and Robillard [10], who focus on a software system evaluation framework

with the aim to provide a set of attributes for the characterization of the main points of

software systems.

Finally, some other operating tools for integrated V&V solutions that has been possible to

find in literature concerns Balanced Scorecards (Rosemann and Wiese) [11], where the

performance measurement system of an organization is customized for ERP software

solutions; expert systems (Vlahavas et al.) [12]; the formal application of ISO standard-based

methodologies (Oppermann and Reiterer) [13] (see Deliverable D2.1 [1] for an extensive

analysis of the ISO standards for software Verification and Validation).



22


From the just analyzed methodological point of view, it is clear how difficult it is to find

common and well-established solutions reusable for the FITMAN project’s V&V objectives.

The methodological aspects are in fact always neatly detached from the technological ones,

and even by considering only the first ones, the different integrated solutions systems are

strongly influenced by the specific problem and by the particular operating situation in which

the testers are developing the tools. For the generic FITMAN V&V Package, as a

consequence, the specific inputs and needs of the project will be deeply taken into account in

the development of the practical solution.

2.2 V&V in other FI-PPP projects Now, on the base of the results already obtained in the Deliverable D.2.1 [1], it is useful to

briefly summarize the software V&V solutions adopted in other FI-PPP projects, taking

obviously into account for each of them the different requirements and objectives compared

to the FITMAN project:

FINEST: (High Relevance)

- The Validation of the requirements of the platform is managed by means of a

questions and answers (Q&A) process, on the base of which is then elaborated a

specific assessment table. Thanks to the assessment table, a qualitative mark can

be assigned in order to evaluate the level of support provided by the technical

solution.

- Development of an evaluation framework by means of the measurement of

performance improvement in five different use cases. The issue is addressed

thanks to the use of specific KPIs for each of the five scenarios.

(Questionnaire + KPIs)

FINSENSY: (Low Relevance)

Evaluation of the different software components by means of a two-level system based

first on questionnaires and then on TELOS (Technical, Economical, Legal,

Operational and Scheduling) methodology.

(Questionnaire + TELOS methodology)

FI-CONTENT: (Medium Relevance)

- The different SEs are evaluated by means of the assignment of a set of

prioritization criteria in order to obtain a rank for each of them.

- Subsequently, the critical SEs have been validated in a workshop by collecting the

feedbacks of about one hundred people.

(Ranks of SEs + Feedbacks in Workshop)

INSTANT MOBILITY: (Low Relevance)

Use of electronic questionnaires in order to assess the social acceptability of

innovative transportation services.

(Questionnaires)

SMART AGRIFOOD: (High Relevance)

- Use of recurrent design workshops, national panels, on spot meeting or phone

calls, interviews and focus groups in order to collect useful feedbacks.



23


- Use of requirement tables: validation by means of the control whether the different

requirements have been inserted or not in the pilots.

- Validation of the requirements by means of workshops and open discussions with

15-20 consumers.

- During all the development phase, different feedbacks from the community.

(Requirement Tables + Feedbacks in Workshop + Feedbacks from the

community during the development)

Analyzing the solutions adopted by the five just mentioned FI-PPP projects, it is possible to

note the lack of common and repeatable V&V solutions. The principal motivation is in the

fact the every project has very different objectives, restrictions and practical conditions to

implement the operating solution.

However, in fully agreement with the Deliverable D.2.1 [1], we have to highlight that the two

projects more relevant to the concepts of FITMAN Methodology are the FINEST and the

Smart Agrifood. For this reason, the different solutions adopted by these two projects will be

taken properly into account in the process of defining of the methodologies and tools of the

generic FITMAN V&V Package.

In the light of both last paragraphs about respectively V&V Methodological solutions and

V&V in other FI-PPP projects, the real clear conclusion that is possible to do is that in the

practical implementation of V&V Packages a plethora of different methods, techniques,

solutions and tools can be used, according to the particular needs and requirements that have

to be addressed in the particular situation. Anyway, a thing seems to be very clear after the

just concluded analysis, i.e. that it is necessary in the development of a generic V&V Package

to logically separate in two different parts the methodologies used, on one hand for Data

Collection (i.e. logged data, questionnaires, interviews, verbal protocol analysis, workshops,

phone calls) and on the other hand for Data Elaboration and Presentation (i.e. AHP, WSM,

Balanced Scorecard (i.e. KPIs), expert systems, TELOS, requirement tables), as depicted in

Table 3:

Data Collection Data Elaboration and

Presentation

Methodologies - Logged data

- Questionnaires

- Interviews

- Verbal protocol

analysis

- Workshops

- Phone calls

- AHP

- WSM

- Balanced Scorecard

(KPIs)

- Expert systems

- TELOS

- Requirement tables

Table 3 Methodological solution for Data Collection, Elaboration and Presentation

On the base of these considerations is now possible to conclude the present SotA research by

means of the analysis of the technological tools actually available for the practical

implementation of the two aforementioned logical phases, i.e. Data Collection and Data

Elaboration and Presentation.

2.3 V&V technological solutions



24


2.3.1 Data Collection

As emerged from the previous paragraphs, several methodologies for Data Collection exist.

That area aims at focusing and better describing the questionnaires technique that could be

used in the V&V package to collect both qualitative and quantitative data. In order to provide

a smart and easy-to-use tool, it seems to be reasonable to focus on the Web-based forms used

to submit questionnaires to users in an easier and faster way. In particular, the focus of that

area will be on the description and the comparison of the most common web-form generators:

Google Form and SurveyMonkey. These two tools basically provide the same results, but

they have some differences following highlighted (please note that the underline part in the

SurveyMonkey description represents the differences compared with the Google Form one).

Google Form [14]: It is a web-based tool to collect and elaborate data and general surveys.

Summarizing its main characteristics:

1. It’s possible to implement complex “jump logics”; indeed, it’s possible to skip

directly to a dedicated page according to the answer in the previous one and

consequently to customize the type of questions .

2. The survey can be shared to users through different means: including HTML code

within a web-site, sending the link by e-mail, sharing the link through the web site

itself or by including it in a Facebook page. All these options could be used in the

same time.

3. Data are stored directly within the account web page to which the survey is linked.

4. The overall answers are summarized within a final report.

5. It’s possible to insert within the survey several kind of questions, but also pictures,

graphs, scheme and text, in order to completely customize the contents. It’s also

possible to copy and paste text, with unlimited number of letters.

6. Filters, comparison, graphs and simple tables can be manually created, after the

exportation on an Excel file.

7. Unlimited questions and answers could be included.

8. Collected data can be exported directly in .PDF, .XLS, .CSV format, for

subsequent elaborations.

9. Users can continuously update the answers (until the pre-defined deadline).

10. It’s possible to control and limit the access to the survey thanks to the definition of

passwords and IP address management.

11. It’s possible to organize data collection in personalized folders.

12. The full version of Google Form is completely available for free (even if a Google

account is required).

SurveyMonkey [15]: It is also a web-based tool to collect and elaborate data and general

surveys [15]. Summarizing its main characteristics:

1. It’s possible to implement complex “jump logics”; indeed, it’s possible to skip directly

to a dedicated page according to the answer in the previous one and consequently to

customize the type of questions .

2. The survey can be shared to users through different means: including HTML code

within a web-site, sending the link by e-mail, sharing the link through the web site

itself or by including it in a Facebook page. All these options could be used in the

same time.

3. Data are stored directly within the account web page to which the survey is linked.

Furthermore, is possible to create different “gatherers” of answers related to different



25


groups of users considered (i.e. the ones involved through “web link”, “email”, “web

site”, “Facebook”). Each “gatherer can have its own settings and specific limitations,

and could be independently open or closed to users.

4. There are two basic methods of analysis: the first one is through a summary of the all

answers, the second one, more analytical, shows a personalized report of the answers

sent by each user.

5. It’s possible to insert within the survey several kind of questions, but also pictures,

graphs, scheme and text, in order to completely customize the contents. It’s also

possible to copy and paste text, with unlimited number of letters.

6. Filters, comparison, graphs and simple tables can be created.

7. Unlimited questions and answers could be included.

8. Collected data can be exported directly in .PDF, .XLS, .CSV format, for subsequent

elaborations.

9. Users can continuously update the answers (until the pre-defined deadline). It’s

possible to monitor who sent the answer and when in real-time.

10. It’s possible to control and limit the access to the survey thanks to the definition of

passwords and IP address management.

11. It’s possible to organize data collection in personalized folders.

12. Logo and colors can be personalized.

13. It is very user-friendly and allows user to easily develop his own survey.

14. Basic limited version is available for free (with a SurveyMonkey account creation).

For an advanced and more complete version, a fee is required.

The next Table 4 summarizes the main features of both, highlighting that SurveyMonkey is a

bit more intuitive to use, allows a more complex handling of data and results, a customization

of the form in terms of colors and logo, but requires the payment of a fee for the more

complete version:

SurveyMonkey Google Form

1. “Jump logics” Yes Yes

2. Data collection complexity Yes Partial

3. Text, picture and graphs incorporation Yes Yes

4. Sharing options alternatives Yes Yes

5. Filters and comparison on results Yes No

6. Unlimited questions and answers Yes Yes

7. Export data settings Yes Yes

8. Updating, control and security of sent data Yes Partial

9. Access control through PW or other methods Yes Yes

10. Organization of data collected within folders Yes Yes

11. Logo and colors personalization Yes No

12. Ease of use and intuitive development Yes Partial

13. Availability for free Not full version Yes

Table 4 SurveyMonkey vs Google Form

2.3.2 Data Elaboration and Presentation

After the Data Collection phase, it is necessary to elaborate data, calculate KPIs, query and

present results. As seen, different methodologies are available and have been used in previous

projects. The technological solution can be represented by a sort of database which allows to



26


store and elaborate data, providing also more complex calculations and views. Here some

simple but useful tools are analyzed and compared according to the needs.

MS Access

Mainly allows to store data creating a relational database and provides queries of

them; not suitable for data elaboration and calculations of KPIs.

Complex queries can be executed.

It is included in the Microsoft Office Suite, therefore available for the largest part of

users.

It is more suitable for a desktop use, and not suggested for database-driven web sites.

It has a low scalability: a limited number of users can simultaneously access on that

database.

It can manage only few Giga bytes of data (1 o 2).

MySQL

Mainly allows to store data creating a relational database and provides queries of

them; not suitable for data elaboration and calculation of KPIs.

Complex queries can be executed.

It is the most used and documented software to create web-based DMBS.

It has a high scalability: it is suitable for the access of hundreds or thousands of people

in the same time.

It is suitable to be used in the support of data-base web sites.

It can manage Tera bytes of data.

It requires an extra fee for the complete version (a basic limited version is available

for free).

MS Excel

Mainly allows to store data creating simple databases and provides simple views of

them; it doesn’t allow the creation of complex logical relations between fields (on the

contrary of other tools just described), but it can easily collect, elaborate and visualize

data.

Elaborations and calculations of KPIs can be done using advanced functions.

Presentation of data and different views can be done using filters and pivot, but not

complex query of data, as in the previous cases.

It is very easy and intuitive to use.

It is included in the Microsoft Office Suite, therefore available for the largest part of

users.

It can manage only few Giga bytes of data (1 o 2).

Directly linked with the form: when data are gathered through the web-based form

(the tool considered in the first part of Data collection), it allows to directly export

data on Excel spreadsheet, where they can be elaborated.

It has a very low scalability: only one user can access to that file.

The next Table 5 summarizes the main features of the three tools:



27


MS Access My SQL MS Excel

1. Storage, elaboration/calculation of

KPIs and presentation of data

Storage and

presentation

Storage and

presentation

Storage,

calculation

and

presentation

2. Ease of use Partial Partial Yes

3. Big quantity of data No (few

Giga)

Yes No (few

Giga)

4. Scalability Low High Very low

5. Availability MS Office

Suite

Fee MS Office

Suite

6. Data selection Complex

query

Complex

query

Views

through filters

and pivot

7. Data presentation No Via plug-in Built-in graph

functionalities

8. Link with Data collection form No No Yes

Table 5 Comparison between MS Access, My SQL and MS Excel

Probably the first two solutions (MS Access and MySQL) can provide a more complex and

complete set of options from the data storage and presentation perspective, because they

allows to create relational databases and user can create complex query of data. Anyway,

from the elaboration perspective these two solutions are a bit more limited for the calculation

of KPIs, compared with MS Excel. Furthermore, MS Excel can be directly linked with the

form, which automatically generate the spreadsheet.

2.4 V&V SotA findings We have extensively seen in this Chapter that common and well-established solutions for

software V&V Packages development and implementation are not present nor in literature nor

in practice. In fact, even if the complex and articulated themes of verification, validation,

assessment and evaluation are strongly addressed, they seem to be often confused and even if

some systems are normally used with success, they clearly remain linked to the specific

contexts or situations for which they have been designed.

In spite of all these considerations, in the literature review that has been conducted some

specific elements have emerged as possible interesting solutions that could be customized and

combined according to the FITMAN V&V specific objectives. From a methodological point

of view, we are thinking of questionnaires, interviews, Balanced Scorecard (KPIs) and

requirement tables, while from a technological one of SurveyMonkey for Data Collection and

to MS Excel and MySQL for Data Elaboration and Presentation.

Anyway, the results of the present SotA represent only simple suggestions that will be

eventually elaborated and customized according to the specific needs and restrictions of the

generic FITMAN V&V Package. As a consequence, in the rest of the document the main

methodological and technological points of the generic FITMAN V&V Package will be

gradually addressed, in order to outline at the end of the discussion an extremely detailed

explanation of its contents, features and technical functionalities.



28


3. FITMAN V&V Package

The generic FITMAN V&V Package has been developed by taking in inputs and putting

together in a coherent and effective framework the main results of D2.1 and D2.2, i.e. the

FITMAN 7-Steps V&V Methodology (D2.1) [1], the FITMAN Business Validation Criteria

and the IT Verification Criteria (D2.1) [1], the Recommendations for the FITMAN V&V

Package functions (D2.1) [1], the FITMAN Technical (IT) Indicators (D2.2) [2], the

FITMAN Business Performance Indicators (D2.2) [2], the ECOGRAI Simplified

Methodology (D2.2), the methodologies and the modalities to analyze and elaborate the

abovementioned Performance Indicators (D2.2). As a consequence, starting from the V&V

Package non-functional requirements as provided in D2.1, the complete and analytic

description of the V&V Package’s methodological foundations, framework and specific tools

and technologies is provided. Afterwards, the practical instructions for the subsequent

instantiation of the generic V&V Package in the FITMAN eleven Trials (to be performed in

T2.4) are showed.

3.1 V&V Package functional and non-functional requirements As highlighted in Paragraph 1.6, several inputs for the development of the generic FITMAN

V&V Package have been taken into account, according to the different results of the previous

Deliverable D2.1 [1] and D2.2 [2]. Among them, it is useful now to remember the

Recommendations developed in the Deliverable D2.1 [1], because all the other elements

(please see Paragraph 1.6) will be properly considered in the rest of the discussion, and so it is

in this moment really important to show in detail this general preliminary indications.

The Recommendations of the Deliverable D2.1 [1] are divided in functional and non-

functional requirements. Both the categories of suggestions have been properly taken into

account in the design and implementation of the generic FITMAN V&V Package, as it will be

possible to note later. Anyway, it’s important to notice that as the project evolved and the

Trials became acquainted, the requirements have been consequently updated, re-adapted and

customized according to the specific issues and situations addressed.

In few words, the functional requirements developed in the Deliverable D2.1 [1] are:

1. Integrated package of generic information: The FITMAN V&V Package should

describe all the information regarding the criteria, methodologies and indicators as

defined in D2.1 [1] and D2.2 [2].

2. Support for trial-specific instantiation: The FITMAN V&V Package is used to

instantiate the generic Criteria and Indicators for a specific Trial’s Business Scenario.

3. V&V assessment support and guidance: The FITMAN V&V Package should guide

the users through the different phases of V&V assessment (Phase II).

4. Data input and collection; potential crowd inclusion: Different indicators require

different kinds of data which are collected from different types of sources. The

FITMAN V&V Package should support the collection and input of data.

5. Aggregation and analysis of data to calculate indicators

6. Trial result visualization: The objective of visualization is to offer the possibility to

understand and interpret the Trials’ Business Scenarios results, also through

comparison:

- At the Trial’s Business Scenario level the V&V assessment results should be

visualized and allow comparison for example between different user/expert

groups, different time points or to target values. The visualization may depend on

Indicator type.



29


- At the FITMAN level the visualization should support comparison between Trials’

Business Scenarios, Trial groups (Smart, Digital, Virtual), architecture levels or

between GEs.

7. Documentation: The FITMAN V&V Package should document all the raw data, query

etc. results and the process: which methods were used, who participated in the

assessment, when it was performed etc. to allow tracing back.

8. Assessment status indication: This allows the identification of the status (the phases

completed or currently active) of the Trial’s Business Scenario assessment both to

support the Trial management and the Use Case management at the FITMAN level:

- The Trial’s Business Scenario dashboard should visualize the current state of the

Trial’s Business Scenario assessment: which phases have been completed, which

stakeholders have completed their assessment etc. (depending on the phases and

methodology).

- The FITMAN level dashboard should give an overview of the status of all the

trials V&V status and offer links to the Trial’s Business Scenario’s dashboards.

9. Search functions: The V&V Package could include a search function to search all

FITMAN V&V assessment information to identify for example: which Trials’

Business Scenarios have assessed which GEs, which Trials’ Business Scenarios have

used specific Indicators, which Trials’ Business Scenarios have reached their target

values etc.

While the non-functional requirements are:

1. Simple, self-explanatory, easy to use tool: The V&V Package should be as simple as

possible to use.

2. Easy to take into use and instantiate: The tools should not require any software

installations, or be dependent on specific non-standard environment. From this point

of view either web-based of excel-based tool is recommended.

3. On-line or/and off-line usage

4. Assessment data openness

5. Modular Implementation/ Operation

A deeper explanation of all the above-mentioned points is available in the Deliverable D2.1

[1].

3.2 V&V Package methodological foundations In Paragraph 1.6 of the present document has been already presented the logical process

which has lead us to the development for the FITMAN V&V Package of a 3-Sections Model

(i.e. the “Technology Section”, the “General Information Section” and the “Instructions and

Support Section”), whose “Technology Section” is in its turn organized in three different

kinds of forms, i.e. the “Self-certification Form”, the “PIs Measurement Form” and the

“Community-based Form”. Moreover, another “lifecycle” dimension of analysis of the

FITMAN V&V Package’s Technology Section has been taken into account, considering for

its practical exploitation other three distinct chronological phases, i.e. “Data Collection”,

“Data Elaboration” and “Data Presentation”. Finally, ECOGRAI Simplified Methodology has

been also properly fully addressed for Business Performance Indicators definition and

development.



30


While the definition of the three Sections and of the three chronological phases of exploitation

of the FITMAN V&V Package’s Technology Section embeds evidently basic concepts and

will be self-explained in the following discussion of the V&V Package framework, it is on the

other hand necessary to clearly motivate the choice of the three different kinds of Forms that

will be used in the V&V Process for Data Collection. This preliminary presentation will lead

in fact to a simpler and more intuitive analysis of the whole FITMAN V&V Package in the

next Paragraph 3.3.

First of all, according to the results of D2.1 [1], a general table resuming the links among

Trials’ Business Scenarios, Steps, Criteria Categories and Criteria has been produced:



31


STEPS CRITERIA CATEGORIES CRITERIA'S NATURE

Business Criteria IT Criteria

1. Code Verification Functionality Correctness GENERIC

"" Interoperability GENERIC

Maintainability Code Consistency GENERIC

"" Analizability GENERIC

2. Model Verification Functionality Correctness GENERIC

Reliability Software Maturity GENERIC

"" Fault Tolerance GENERIC

"" Recoverability GENERIC

Maintainability Analizability GENERIC

"" Code Consistency GENERIC

3. Backlog Verification Functionality Correctness GENERIC




Efficiency Time Behavior GENERIC

"" Resource Behavior GENERIC

Portability Hardware Independence GENERIC



4. Release Verification Functionality Correctness GENERIC


"" Security GENERIC








"" Changeability GENERIC

"" Stability GENERIC

"" Testability GENERIC


FITMAN-relevant Openness GENERIC

"" Versatility GENERIC

5. Product Validation Functionality Correctness GENERIC


"" Security GENERIC

Usability Understandability GENERIC

"" Ease of learning GENERIC

"" Attractiveness GENERIC



Portability Adaptability GENERIC

"" Replaceability GENERIC

"" Hardware Independence GENERIC






"" Traceability GENERIC



CRITERIA



32


Table 6 Links among Trials’ Business Scenarios, Steps, Criteria Categories, Criteria and their nature

For a generic Trial’s Business Scenario, to each Step of the V&V Methodology, the proper

Criteria Categories have been assigned and then, to each Criteria Category the proper Criteria

have been associated. In the list of the Criteria, all the different kinds have been considered

(i.e. Generic and Trial Specific Business Validation Criteria, Generic and Trial Specific IT

Verification Criteria). It has been highlighted in particular first the clear distinction between

Business and IT Criteria and then between Generic and Trial Specific Criteria. The distinction

between the first five Product-specific Steps and last two Trial’s Business Scenario-specific

Steps has been depicted respectively by means of the rose and of the blue colour.

As it is possible to note in the figure, the Criteria of the Product-specific Steps are all

correctly “Generic”, in agreement with the fact that in this first part the different GEs, SEs,

TSCs and TICs (i.e. the “Products”) have to be verified and validated separately as

independent Software Components. On the other hand, the Criteria of the Trial’s Business

Scenario-specific Steps can be obviously Generic or Trial-specific, in agreement with the fact

that in this second part are the different integrated Trials’ Business Scenarios solutions that

have to be validated as overall packages (i.e. combinations of different GEs, SEs, TSCs and

TICs). The Table 6 is hence evidently nothing more than the synthetic resume of the findings

of D2.1 [1].

The first choice about the practical solution to adopt for Data Collection has been taken here

in order to find a practical and simple solution for the Verification and Validation of the

6. Trial Solution Functionality Interoperability GENERIC

"" Security GENERIC

Usability Understandability GENERIC

"" Ease of learning GENERIC

"" Operability TRIAL SPECIFIC

"" Communicativeness TRIAL SPECIFIC

"" Attractiveness TRIAL SPECIFIC

Efficiency Time Behavior TRIAL SPECIFIC

"" Resource Behavior TRIAL SPECIFIC

Portability Adaptability TRIAL SPECIFIC

"" Installability GENERIC

"" Coexistence GENERIC

"" Replaceability TRIAL SPECIFIC

Maintainability Changeability TRIAL SPECIFIC


"" Traceability TRIAL SPECIFIC



7. Business Validation Customer Reliability Customer Order Fulfillment Quality GENERIC

"" Customer Order Fulfillment Quantity GENERIC

"" Documentation Accuracy GENERIC

"" Customer Order Fulfillment Cycle Time GENERIC

"" Source Cycle Time GENERIC

Responsiveness Make Cycle Time GENERIC

"" Flexibility GENERIC

"" Adaptability GENERIC

Agility Supply Chain Value at Risk (VAR) GENERIC

"" Total Suppy Chain Management Costs GENERIC

"" Costs of Goods Sold GENERIC

Cost Cash-to-Cash Cycle Time GENERIC

"" Inventory Optimization GENERIC

Assets Sales Receivable Optimization GENERIC

"" Payable Optimization GENERIC

"" Supply Chain Fixed Assets Optimization GENERIC

"" Working Capital Optimization GENERIC

STEEP Safety/Healthy in Working Environment TRIAL SPECIFIC

"" People Motivation and Empowerment TRIAL SPECIFIC

"" Market Positioning and Market Share TRIAL SPECIFIC

"" Quality TRIAL SPECIFIC



33


single GEs, SEs, TSCs and TICs. The five Product-specific Steps seem to be very different

from the subsequent two, in which Criteria are applied in order to get feedbacks about the

overall Trial’s Business Scenarios solutions implemented in the eleven Use Cases.

Furthermore, some other not negligible aspects, such as:

the will not to prevent the different consolidated methods and procedures followed by

the different Development Teams in the creation of the different GEs, SEs, TSCs and

TICs;

the extreme difficulty to create a general common procedure for the Verification and

Validation of extremely various and complex components;

the will to make the V&V Package as simpler as possible and at the same time a real

useful tool for the correct and effective collection of different kinds of data;

have led to the adoption for the five Product-specific Steps of a Self-certification Form with

whom the different Development Teams will be able to certify, driven by specific procedures,

the achievement of the abovementioned IT Criteria.

Second of all, for the last Product-specific Step and for the first Trial’s Business Scenario-

specific Step, the IT Indicators defined in D2.2 [2] have been linked to the related IT Criteria

previously showed (always according to the results of D2.2 [2]), in order to obtain an

overview of all the Technical Indicators defined for the Validation phase:



34




35


Table 7 PIs associated to each Criteria and classified by their own nature

As it is depicted in Table 7, the P5 and T1 Steps has been filled with the proper IT Indicators.

Then, for each Indicator, a further annotation depending on its own nature has been done:

Quantitative: if the Indicator implies an “objective” measure (e.g. Turnaround Time,

Throughput, Cycle Time);

Interoperability GENERIC Interoperability Maturity Level QUANT

Security GENERIC Authentication Mechanism Integrity QUANT

Understandability GENERIC Required IT Background QUAL

Ease of learning GENERIC Learning Time QUAL

"" "" Time to Expertise QUAL

Operability TRIAL SPECIFIC Operation Time QUANT

Communicativeness TRIAL SPECIFIC QUAL

Attractiveness TRIAL SPECIFIC Users' Attraction Level QUAL

"" TRIAL SPECIFIC Users' Engagement Time QUAL

"" TRIAL SPECIFIC Users’ satisfaction level regarding system’s attractiveness QUAL

Time Behavior TRIAL SPECIFIC Response Time QUANT

"" "" Turnaround Time QUANT

"" "" Throughput QUANT

"" "" Waiting Time QUANT

Resource Behavior TRIAL SPECIFIC I/O Utilization QUANT

"" "" I/O Utilization Message Density QUANT

"" "" Memory Utilization QUANT

"" "" Transmission Utilization QUANT

"" "" I/O Related Errors QUANT

"" "" User Waiting Time of I/O Devices Utilization QUANT

Adaptability TRIAL SPECIFIC Adaptability of Data Structures QUANT

"" "" System Sofware Environmental Adaptability QUANT

"" "" Porting User Friendliness QUAL

Installability GENERIC Installation Time QUANT

"" "" Ease of Setup Retry QUANT

"" "" Installation Flexibility QUANT

"" "" Ease of Installation QUANT

Coexistence GENERIC Internal Coexistence QUANT

"" "" External Coexistence QUANT

Replaceability TRIAL SPECIFIC Continued Use of Data QUANT

"" "" Function Inclusiveness QUANT

"" "" Use Support Functional Consistency QUAL

Changeability TRIAL SPECIFIC Change Recordability QUANT

"" "" Modification Complexity QUANT

"" "" Software Change Control Capability QUANT

Stability GENERIC Change Impact QUANT

"" "" Change Success Ratio QUAL

"" "" Modidication Impact Localization QUANT

Traceability TRIAL SPECIFIC QUANT

Openness GENERIC Openness Level QUAL

Versatility GENERIC Versatility GE Package Usage Index QUANT

"" "" Versatility Average GE Usage/Trial QUANT

Customer Order Fulfillment Quality GENERIC

Customer Order Fulfillment Quantity GENERIC

Documentation Accuracy GENERIC

Customer Order Fulfillment Cycle Time GENERIC

Source Cycle Time GENERIC

Make Cycle Time GENERIC

Flexibility GENERIC

Adaptability GENERIC

Supply Chain Value at Risk (VAR) GENERIC

Costs of Goods Sold GENERIC

Cash-to-Cash Cycle Time GENERIC

Inventory Optimization GENERIC

Sales Receivable Optimization GENERIC

Payable Optimization GENERIC

Supply Chain Fixed Assets Optimization GENERIC

Working Capital Optimization GENERIC

Safety/Healthy in Working Environment TRIAL SPECIFIC

People Motivation and Empowerment TRIAL SPECIFIC

Market Positioning and Market Share TRIAL SPECIFIC

Quality TRIAL SPECIFIC



36


Qualitative: if the Indicator implies the “subjective” opinion of a human being (e.g.

Required IT Background, Users’ Attraction Level, Porting User Friendliness).

According to the different nature of the Indicators involved in the P5 and T1 Steps of the

V&V Methodology, a choice about two different solutions for the correct Data Collection has

been done. For the Quantitative Indicators a “PIs Measurement Form” has been ideated in

order to collect for each Software Component and for each Trial’s Business Scenario the

objective “measurable” data useful for Validation. For the Qualitative Indicators a

“Community-based Form” has been ideated in order to collect for each Trial’s Business

Scenario the opinions and judgements of the human beings for the subsequent consolidation

in proper synthetic Indicators.

At this point the use of three different kinds of Forms for Data Collection, i.e.:

Self-certification Form

PIs Measurement Form

Community-based Form

has been well-defined.

Third of all, the ECOGRAI Simplified Methodology established in Deliverable D2.2 [2] and

resumed in Paragraph 1.5 will be applied to Business Performance Indicators, i.e. to the

second and last of the Trial’s Business Scenario-specific Steps. Unlike the Technical

Indicators, the Business Performance Indicators have to be developed “ad hoc” for each

Business Scenario of the Trial (see Table 7). As a consequence, the proper Business Scenario-

specific Business Performance Indicators will be defined during the Instantiation Process (see

Paragraph 4.4) by means of the three-phases ECOGRAI Simplified Methodology already

explained, covering in this way also the T2 Step.

However, as for the IT Indicators, also the Business Performance Indicators will be obviously

distinguished between Quantitative and Qualitative and subsequently completely integrated

respectively in the proper Trial’s Business Scenario-specific PIs Measurement and

Community-based Forms. The complete explanation of the integration of the ECOGRAI

Simplified Methodology for Business Performance Indicators’ Data Collection will be

addressed in the proper Paragraph 3.3.1.1.

The following Table 8 briefly resumes all the concepts described until now:

.



37


Table 8 Kinds of Forms linked to the 7 Steps of the V&V Methodology

1. TRIAL 2. STEPS 3. CRITERIA CATEGORIES CRITERIA'S NATURE 5. PI PI NATURE

Business Criteria IT Criteria

1. Code Verification Functionality Correctness GENERIC


Maintainability Code Consistency GENERIC

"" Analizability GENERIC

2. Model Verification Functionality Correctness GENERIC






3. Backlog Verification Functionality Correctness GENERIC









4. Release Verification Functionality Correctness GENERIC


"" Security GENERIC














5. Product Validation Functionality Correctness GENERIC Fault Detection QUANT

"" "" "" Module Fault Density QUANT

"" "" "" Data Integrity QUANT

"" Interoperability GENERIC Interoperability Maturity Level QUANT

"" Security GENERIC Authentication Mechanism Integrity QUANT

Usability Understandability GENERIC Required IT Background QUAL

"" Ease of learning GENERIC Learning Time QUAL

"" "" "" Time to Expertise QUAL

"" Attractiveness GENERIC Users' Attraction Level QUAL

"" "" "" Users' Engagement Time QUAL

"" "" "" Users’ satisfaction level regarding system’s attractiveness QUAL

Efficiency Time Behavior GENERIC Response Time QUANT

"" "" "" Turnaround Time QUANT

"" "" "" Throughput QUANT

"" "" "" Waiting Time QUANT

"" Resource Behavior GENERIC I/O Utilization QUANT

"" "" "" I/O Utilization Message Density QUANT

"" "" "" Memory Utilization QUANT

"" "" "" Transmission Utilization QUANT

"" "" "" I/O Related Errors QUANT

"" "" "" User Waiting Time of I/O Devices Utilization QUANT

Portability Adaptability GENERIC Adaptability of Data Structures QUANT

"" "" "" System Sofware Environmental Adaptability QUANT

"" "" "" Porting User Friendliness QUAL

"" Replaceability GENERIC Continued Use of Data QUANT

"" "" "" Function Inclusiveness QUANT

"" "" "" Use Support Functional Consistency QUAL

"" Hardware Independence GENERIC Hardware Dependencies QUANT

Maintainability Analizability GENERIC Activity Recording QUANT

"" "" "" Diagnostic Functions Operability QUANT

"" "" "" Audit Trail Capability QUANT

"" "" "" Diagnostic Function Report QUANT

"" Changeability GENERIC Change Recordability QUANT

"" "" "" Modification Complexity QUANT

"" "" "" Software Change Control Capability QUANT

"" Stability GENERIC Change Impact QUANT

"" "" "" Change Success Ratio QUAL

"" "" "" Modidication Impact Localization QUANT

"" Testability GENERIC Built-in Test Functions QUANT

"" "" "" Autonomy of Testability QUANT

"" "" "" Availability of Built-in Test Function QUANT

"" "" "" Re-test Efficiency QUANT

"" Code Consistency GENERIC Cohesion Ratio QUANT

"" Traceability GENERIC QUANT

FITMAN-relevant Openness GENERIC Openness Level QUAL

"" Versatility GENERIC Versatility GE Package Usage Index QUANT

"" "" "" Versatility Average GE Usage/Trial QUANT

6. Trial Solution Functionality Interoperability GENERIC Interoperability Maturity Level QUANT

"" Security GENERIC Authentication Mechanism Integrity QUANT

Usability Understandability GENERIC Required IT Background QUAL

"" Ease of learning GENERIC Learning Time QUAL

"" "" "" Time to Expertise QUAL

"" Operability TRIAL SPECIFIC Operation Time QUANT

"" Communicativeness TRIAL SPECIFIC QUAL

"" Attractiveness TRIAL SPECIFIC Users' Attraction Level QUAL

"" TRIAL SPECIFIC Users' Engagement Time QUAL

"" TRIAL SPECIFIC Users’ satisfaction level regarding system’s attractiveness QUAL

Efficiency Time Behavior TRIAL SPECIFIC Response Time QUANT

"" "" "" Turnaround Time QUANT

"" "" "" Throughput QUANT

"" "" "" Waiting Time QUANT

"" Resource Behavior TRIAL SPECIFIC I/O Utilization QUANT

"" "" "" I/O Utilization Message Density QUANT

"" "" "" Memory Utilization QUANT

"" "" "" Transmission Utilization QUANT

"" "" "" I/O Related Errors QUANT

"" "" "" User Waiting Time of I/O Devices Utilization QUANT

Portability Adaptability TRIAL SPECIFIC Adaptability of Data Structures QUANT

"" "" "" System Sofware Environmental Adaptability QUANT

"" "" "" Porting User Friendliness QUAL

"" Installability GENERIC Installation Time QUANT

"" "" "" Ease of Setup Retry QUANT

"" "" "" Installation Effort QUANT

"" "" "" Installation Flexibility QUANT

"" "" "" Ease of Installation QUANT

"" Coexistence GENERIC Internal Coexistence QUANT

"" "" "" External Coexistence QUANT

"" Replaceability TRIAL SPECIFIC Continued Use of Data QUANT

"" "" "" Function Inclusiveness QUANT

"" "" "" Use Support Functional Consistency QUAL

Maintainability Changeability TRIAL SPECIFIC Change Recordability QUANT

"" "" "" Modification Complexity QUANT

"" "" "" Software Change Control Capability QUANT

"" Stability GENERIC Change Impact QUANT

"" "" "" Change Success Ratio QUAL

"" "" "" Modidication Impact Localization QUANT

"" Traceability TRIAL SPECIFIC QUANT

FITMAN-relevant Openness GENERIC Openness Level QUAL

"" Versatility GENERIC Versatility GE Package Usage Index QUANT

"" "" "" Versatility Average GE Usage/Trial QUANT

7. Business Validation Customer Reliability Customer Order Fulfillment Quality GENERIC

"" Customer Order Fulfillment Quantity GENERIC

"" Documentation Accuracy GENERIC

"" Customer Order Fulfillment Cycle Time GENERIC

"" Source Cycle Time GENERIC

Responsiveness Make Cycle Time GENERIC

"" Flexibility GENERIC

"" Adaptability GENERIC

Agility Supply Chain Value at Risk (VAR) GENERIC

"" Total Suppy Chain Management Costs GENERIC

"" Costs of Goods Sold GENERIC

Cost Cash-to-Cash Cycle Time GENERIC

"" Inventory Optimization GENERIC

Assets Sales Receivable Optimization GENERIC

"" Payable Optimization GENERIC

"" Supply Chain Fixed Assets Optimization GENERIC

"" Working Capital Optimization GENERIC

STEEP Safety/Healthy in Working Environment TRIAL SPECIFIC

"" People Motivation and Empowerment TRIAL SPECIFIC

"" Market Positioning and Market Share TRIAL SPECIFIC

"" Quality TRIAL SPECIFIC

4. CRITERIA



38


3.3 V&V Package framework definition and description It is possible now to describe in detail the generic FITMAN V&V Package framework. As we

have already seen in Paragraph 1.6, the V&V Package is composed by three different

Sections, i.e.:

1. the Technology Section, including all the three above-mentioned kinds of Forms;

2. the General Information Section, explaining to all the possible beneficiaries the main

points related to all the different aspects of the FITMAN V&V Package;

3. the Instructions and Support Section, explaining in detail to all the possible

beneficiaries the modalities for the correct implementation and use of the FITMAN

V&V Package in the eleven Trials;

Moreover, the last two Sections, put together, contribute to form the so-called “FITMAN

V&V Communication Package”, in order to distinguish the pure “Forms-based” Section (i.e.

the Technology one) from its consequent “meta-structure”, i.e. all the information and

instructions needed to manage and support a correct and satisfying use of the Forms

themselves.

In this respect, the explanation will go on by taking into account the two different main parts

of the generic FITMAN V&V Package, i.e. the Technology Section and the Communication

Package, remembering also that for the discussion of the former another “lifecycle”

chronological dimension has to be considered:

i. Data Collection, i.e. the correct and efficient gathering of all the kinds of data by

means of different kinds of Forms. The ECOGRAI Simplified Methodology for

Business Performance Indicators is also integrated;

ii. Data Elaboration, i.e. all the methodological and technological aspects of data

analysis and aggregation (e.g. definition of the different dimensions of analysis,

ways to aggregate and compare the data, practical implementation in software

tools such as MS Excel);

iii. Data Presentation, i.e. the different modalities to report and discuss the final

results of the FITMAN V&V Process, both from a methodological and a

technological point of view (e.g. different ways to present the final data, selection

of the more representative findings, practical implementation in PDF files).

In this section all the methodological aspects of the V&V Package are addressed, while the

practical tools and technologies’ ones will be discussed in the following Chapter 4.

3.3.1 V&V Technology Section

It is useful and necessary to analyze the V&V Technology Section according to the three

different chronological phases supported by it.

3.3.1.1 Data Collection

The reference point for Data Collection, as already said in Paragraph 1.6, is represented by the

Business Scenario. In fact, each of the FITMAN eleven Use Case Trials has more Business

Scenarios, i.e. more specific selected cases within the Trial, which represent the correct basis

for the management of the FITMAN Verification and Validation operations.

As a consequence, to each Trial will be associated different Business Scenarios, each of them

constituted by different GEs, SEs, TSCs and TICs, i.e. the basic Software Components which

integrated and orchestrated will form the specific Business Scenario’s solution:



39


Figure 3 Example of Software Components for a Trial’s Business Scenario

The different Software Components and the integrated Business Scenarios’ solutions will be

verified and validated by means of the three different kinds of Forms already explained in

Paragraph 3.2, i.e. the Self-certification, the PIs Measurement and the Community-based

ones. The result of the integration of these two different point of views is the association,

respectively to each Trial’s Business Scenario’s Software Component and to each Trial’s

Business Scenario, of different specific modalities to collect the necessary data.

The Data Collection’s Solutions for a generic Trial’s Business Scenario are resumed in the

following Table 9:

Analysis Unit Data Collection’s Solutions

Business Scenario’s Software Components

(i.e. GEs, SEs, TSCs, TICs)

Self-certification Form + PIs Measurement

Form

Business Scenario

PIs Measurement Form + Community-based

Form

Table 9 Data Collection’s Solutions

Trial 1

Business Scenario 1

GE 1

GE 2

GE n

SE 1

SE 2

SE n

TSC 1

TSC 2

TSC n

TIC 1

TIC n

TIC 2

Business Scenario 2



40


As a consequence, for a generic Trial’s Business Scenario the different Data Collection’s

modalities will be logically structured as follows:

Figure 4 Data Collection’s Solutions and related Steps

For simplicity, in Figure 4 are considered only one GE, SE, TSC and TIC per Trial’s Business

Scenario.

Trial 1

Business Scenario 1

GE 1


Steps P1, P2, P3, P4, P5

PIs Measurement Form Step P5

SE 1 Self-certification

Form Steps P1, P2, P3,

P4, P5


Step P5

TSC 1




Step P5

TIC 1




Step P5


Steps T1, T2

Community-based Form

Steps T1, T2

Business Scenario 2



41


In the light of this, the Data Collection Setup Process for a generic Trial’s Business Scenario

will necessary involve four different sub-processes, according both to the abovementioned

considerations and to the different origin of Quantitative IT and Business Performance

Indicators, i.e.:

Quantitative Business Performance Indicators Sub-process

Quantitative IT Indicators Sub-process

Self-certification Sub-process

Community-based Sub-process

as it is clearly depicted in the following Figure 5:

Figure 5 Data Collection Setup Process

The four different Data Collection Setup Sub-processes for a generic Trial’s Business

Scenario have different origins and evolutions until their “Formalization” (see the red line in

Figure). Hereafter, they will present the same exact modalities, which can be resumed in three

main steps:

1. Formalization by means of the proper Data Collection Form

2. On-line Form Development

3. Data Collection by means of the proper kind of Form



42


Before the Formalization step, different sources can obviously bring to the identification of

the four Basic Elements of the V&V Process, i.e. Quantitative Business Performance

Indicators, Quantitative IT Indicators, Software Components (i.e. GEs, SEs, TSCs and TICs)

and Qualitative Indicators.

In this respect, as already explained in Paragraphs 1.5 and 3.2, it is important to note how the

first part of the Data Collection Setup Process for Quantitative IT and Business Performance

Indicators is different even if they will be finally gathered by means of the same kind of

Form, i.e. the PIs Measurement Form. The Quantitative IT Indicators have in fact already

been completely identified in the Deliverable D2.2 [2]. As a consequence, the instantiation in

the Trial’s Business Scenario will simply involve the selection of the proper Indicators

(formalized by means of the proper Data Collection Form) and the development of the related

on-line Form (see Chapter 4).

On the other hand, the Quantitative Business Performance Indicators have to be identified and

developed during the Instantiation Process itself. To do this, as suggested by the Deliverable

D2.2 [2], first the application of the three-phases ECOGRAI Simplified Methodology will be

necessary. The output of ECOGRAI will be then the definition of a list of instantiated

Quantitative Business Performance Indicators for each Trial’s Business Scenario. The specific

PIs will be subsequently formalized by means of the proper Data Collection Form and finally

implemented on the web by means of the related on-line Form (see Chapter 4).

The just mentioned situation is useful in order to clarify how in practice the real Instantiation

effort can be both methodologically and practically very different among the four Data

Collection Setup Sub-processes. For this reason, the boundaries of the real Instantiation

Process are clearly depicted and highlighted in Figure.

Finally, it is now useful to clearly explain the role of the “Data Collection Form” (see another

time Figure 5) in the whole Data Collection Setup Process.

The Data Collection Form has been designed in order to make simple, precise and efficient

the logically subsequent Instantiation Process. Its aim is in fact to define in a clear and unique

way the different Basic Elements of the V&V Process, i.e.:

Quantitative Business Performance Indicators

Quantitative IT Indicators

Software Components (i.e. GEs, SEs, TSCs, TICs)

Qualitative Indicators

For each of them, it is in practice defined a complete table containing all the related important

values and information, allowing in this way the subsequent creation of the related on-line

Forms (built on this structure) and also the efficient and ordered elaboration and presentation

of the data. The following Table 10 provides the reference structure of a generic Data

Collection Form for a Basic Element of the V&V Process:


30/09/2013 Deliverable D2.3 – V1.0


Part Field Explanation Example 1 Example 2

Iden

tifi

cati

on

in

form

ati

on

Trial Name The name of the Trial TRW TRW

Trial Type Smart, Virtual, Digital Domain D D

Company Size SME/Large Enterprise LE LE

Business Scenario Define the Business Scenario in the Trial 1-Risk Modelling 1-Risk Modelling

Software Component ID GE, SE, TSC, TIC GE1 NA

Form ID Allows to group Basic Elements in multiple on-line

Forms (This test is displayed on the top of the on-line

Form)

Self-certification

Form – TRW –

Scenario 1 – Risk

Modelling

PIs Measurement Form -

TRW - Scenario 1 - Risk

Modelling

Field Position Order of visualization in a specific on-line Form 1 2

V&V Methodology Step V&V Step the Basic Element refers to 4 6

Max number of answers

(Single/Community)

Collected just once or many times (if > 1 --> Community

based )

1 1

Indicator Name Name for the Indicator utilized internally / Automatic Risk System

Available

Indicator Category

(Technical/Business

Performance)

Technical/Business Performance Indicator / Technical

Qualitative/Quantitative Qualitative/Quantitative Indicator / Quantitative

Accepted Values Range or specific values accepted for the Basic Element Yes/Not/NA 0, 100

When collect data first time When the survey for this Basic Element is open the first

time

01/12/13 31/12/13

How many times For repetitive collection of the Basic Element 1 4

Frequency (week/day/month) For repetitive collection of the Basic Element / Month

Field acquisition type Alfa-Numerical, Text, Logical, Radio Button, Combo Radio Button Alfa-Numerical



44


Menu, etc.

Suggested Values Alternative proposed to the user (Radio Button or

Combo Menu)

Yes/Not/NA /

Mandatory/Optional Survey cannot be closed if the value is not specified Mandatory Mandatory

Info

rmati

on

to d

isp

lay

Label to Display Label in front of the Indicator / The automatic

“Intelligent” risk

identification system suits

the changing multiple

factors of the certain

worker

Formula/Method to calculate the

Indicator

Description of how to get the value of the Indicator / NA

Unit of measure If any, specify the unit of measure of the Indicator / NA

Notes Open text to display in which is possible for the user to

add specific comments and remarks

/ The system is able to

bridge the gap between

existed passive risk control

system and a new level of

system with the accurate

risk prediction and

complete work plan

generation

Table 10 Data Collection Form structure


30/09/2013 Deliverable D2.3 – V1.0 - Final I-VLab


The Data Collection Form for a Basic Element of the V&V Process is hence organized in two

different parts:

Identification Information

Information to Display

The former represents the “meta-structure” of the Basic Element, i.e. the information to

identify it in an univocal way (e.g. Trial Name, Form ID, Field Position, Max number of

answers), while the latter explains the information that will appear in the subsequent on-line

Form and that will be as a consequence visualized by the different users. In particular, the

Identification Information part will be the same for every V&V Basic Element, while the

Information to Display part will change according to the specific Basic Element taken into

account. In Table 10 is provided an example of the Data Collection Form for a Quantitative

Business Performance or IT Indicator.

The whole Data Collection Setup Process has been clearly explained. In the rest of the present

Paragraph will be then deeply analyzed the structure of the different kinds of on-line Forms,

i.e. Self-certification Form, PIs Measurement Form and Community-based Form.

a. Self-certification Form: Its aim it is to verify and validate the different single Software

Components developed, i.e. GEs, SEs, TSCs and TICs. According to the Deliverable D2.1

[1], the GEs will be involved in the following Steps:

P4

P5

while the SEs, the TSCs and the TICs in the following ones:

P1

P2

P3

P4

P5

For every single Software Component, a verification and validation form will be available in

order to assess the result of the specific tests applied to it. Following another time the

suggestions of the Deliverable D2.1 [1], for every Step of the V&V Methodology a specific

Verification or Validation Technique is recommended. As a consequence, for a particular GE

the Self-certification Form will be like the following:

GE x

STEPS

RECOMMENDED

TECHNIQUE

ALTERNATIVE

TECHNIQUE (if

any)

RESULT

(YES/NO/NA)

COMMENTS (if

any)

P4

Regression Testing

NA

YES

NA

P5

Black Box Testing for

Validation

NA

YES

NA



46


Table 11 Example of Self-certification Form for GEs

While for a particular SE, TSC or TIC the Self-certification Form will be like the following:

SEx

/TSCx

/TICx

STEPS RECOMMENDED

TECHNIQUE

ALTERNATIVE

TECHNIQUE (if

any)

RESULT

(YES/NO/WIP)

COMMENTS

(if any)

P1

White Box Testing

NA

YES

NA

P2

Traceability Analysis

NA

YES

NA

P3

Regression Testing

NA

NO

NA

P4

Regression Testing

NA

NO

NA

P5

Black Box Testing for

Validation

NA

YES

NA

Table 12 Example of Self-certification Form for SEs, TSCs and TICs

As it is evident, for every Step of the single GE/SE/TSC/TIC, a specific V&V Technique is

associated and recommended (“Recommended Technique”: according to the results of the

Deliverable D2.1 [1]), but also the use of another alternative V&V Technique is allowed

according to the specific procedures and needs of the particular Development Team

(“Alternative Technique”). Subsequently, in the “Result” column the Development Team is

allowed to synthetically express the outcome of the procedure and eventually to add some

specific comments about the application of the particular V&V Technique (e.g. issues

occurred, possible improvements, next tests planned) (“Comments”).

Because of the fact that every Trial’s Business Scenario is composed by different GEs, SEs,

TSCs and TICs and that each of them implies the filling of a dedicated form by the proper

Development Team, we have to concretely image the whole Self-certification Form of each

Trial’s Business Scenario as a composition of several different sub-forms, each of them

specific for a particular Software Component and updated by its own Development Team. An

example of this situation is showed in the following Figure 6:



47


Figure 6 Example of the Self-certification Form’s structure for a Trial’s Business Scenario

FITMAN Stakeholders:

Self-certification Steps Stakeholders

P1

Development Team

P2

Development Team, Sprint Master

P3

Development Team, Sprint Master, Product

Owner



48


P4

Development Team, Sprint Master, Product

Owner

P5

Product Owner (and Trial Supporting Team)

Table 13 Self-certification Form’s FITMAN Stakeholders

b. PIs Measurement Form: As already inferable, the PIs Measurement Form will be

practically associated both to each different Software Component of the specific Trial’s

Business Scenario (Step P5 of the V&V Methodology) and directly to the whole Trial’s

Business Scenario itself (Steps T1 and T2 of the V&V Methodology). Moreover, for the

Validation of the whole Trial’s Business Scenario, the PIs Data Collection will be further

divided into two Sub-forms, i.e. the IT Sub-form and the Business Performance Sub-form, in

order to clearly highlight and manage the two different areas of interest already identified in

the present Paragraph. As a consequence, the Data Collection of the Quantitative IT and

Business Performance Indicators for a Trial’s Business Scenario will be practically organized

as depicted in Figure 7:



49


Figure 7 Example of the PIs Measurement Form’s structure for a Trial’s Business Scenario

For simplicity, in Figure 7 are considered only one GE, SE, TSC and TIC per Trial’s Business

Scenario.

We have to precise that each PIs Sub-form will be formed by all the different IT Indicators

that will be instantiated for the related Software Component of the related Trial’s Business

Scenario. The particular IT Sub-form will then be formed by all the different IT Indicators

that will be instantiated for the related Trial’s Business Scenario, while the particular Business

Sub-form will be formed by all the different Business Performance Indicators that will be

instantiated always for the related Trial’s Business Scenario.

The structure of the PIs Measurement Form for a generic Quantitative Indicator (IT or

Business Performance) will be the following, as already previously showed during the

explanation of the “Information to display” part of the Data Collection Form:



50


Quantitative

IT/Business

Performance

Indicator

Label to display:

Formula/Method to calculate the

Indicator:

Unit of measure:

Notes:

Table 14 Logical structure of Data Collection for a generic Quantitative IT or Business Performance

Indicator


PIs Measurement Sub-forms Stakeholders

PIs Sub-forms Software Components

Product Owner (and Trial Supporting Team)

IT Sub-form

Trial Solution Owner, Trial Team

Business Sub-form

Trial Team

Table 15 PIs Measurement Form’s FITMAN Stakeholders

c. Community-based Form: The Community-based Form will be directly associated to each

different Trial’s Business Scenario (Steps T1 and T2 of the V&V Methodology). As a

consequence, the Data Collection of the Qualitative IT and Business Performance Indicators

for a Trial’s Business Scenario will be practically organized as depicted in Figure:

Figure 8 Example of the Community-based Form structure for a Trial’s Business Scenario



51


The particular Trial’s Business Solution’s Community-based Sub-form will be formed by all

the different Qualitative IT and Business Performance Indicators that will be instantiated for

the related Trial’s Business Scenario.

The structure of the Community-based Form for a generic Qualitative Indicator (IT or

Business Performance) will be defined directly during the Instantiation Process.


Community-based Form Stakeholders

Qualitative IT Indicators

Trial Solution Owner, Trial Team

Qualitative Business Performance

Indicators

Trial Team

Table 16 Community-based Form’s FITMAN Stakeholders

3.3.1.2 Supplementary Community-based Techniques

Beyond the mandatory generic Community-based Form just proposed for each Trial’s

Business Scenario, some other supplementary techniques are here proposed in order to

integrate and complete the Community-based (i.e. Crowdsourcing) V&V modalities for the

first five Product-specific Steps. They represent very significant general suggestions whose

practicability and concrete implementation will be evaluated case by case in the subsequent

Instantiation phase. The following contents are obviously taken from the previous Deliverable

D2.1 [1]. The suggestions for the Instantiation phase are hence clearly depicted in the

following Table 17:



52


TRIAL’S BUSINESS

SCENARIO

STEPS

Proposed Community-based (i.e. Crowdsourcing)

Technique

P5

Online deliberation and feedback tools, Open calls for

testing scenarios/ trials, Workshops and specific working

groups, Traditional online survey tools, Social networks

P4

Deliberation and feedback tools for ex-ante

crowdsourcing, Dedicated IT tools for ex-post

crowdsourcing

P3

Deliberation and feedback tools for ex-ante

crowdsourcing, Dedicated IT tools for ex-post

crowdsourcing

P2 Physical or online workshops, Readily-available

prototyping applications, Social deliberation platforms

P1

/

Table 17 Supplementary Community-based Techniques

The aforementioned supplementary techniques, if applied, will also lead for the specific

Trial’s Business Scenario to a more articulated Data Elaboration, as it is possible to see in

Paragraph 3.3.1.4.

3.3.1.3 Determination of the Business Performance Indicators and Examples of

application

The BPIs are defined using the ECOGRAI method. We have described the simplify

ECOGRAI method in Paragraph 1.5.2. In the next Paragraph an example of application with

each kind of factory, Smart, Digital, Virtual, will be presented. These examples are produced

from the information collected in Phase 1 and 2 of the Trial Handbook.

3.3.1.3.1 Determination of the elements of ECOGRAI

The application of the simply ECOGRAI method and his 3 phases will be developed for the

trials TRW, AUGUSTA WESTLAND, and GEOLOC SYSTEM (SEGEM MACBO) in the

following paragraphs. These trials has been chosen in order to have an example for each kind

of factory, Smart factory for TRW, Digital factory for AUGUSTA WESTLAND, and Virtual

for GEOLOC SYSTEM.

To apply ECOGRAI simplified method, the Trial is described as a system with its

characteristics in the first phase. In the second, according to the priority whished by the trial,

an objective is chosen in the list provided by the Trial in the Chapter 2 of the Trial Handbook

regarding to the criteria (quality, cost, delay, safety). Then the Decision variable or Action

variable (DV/AV) with the constraints (if any) which allows to reach this objective is

determined. In the third, a performance indicator is defined to measure the reaching of the

objective by using the DV/AV.

3.3.1.3.2 For Smart factory (TRW)

TRW Automotive is one of the world’s largest automotive suppliers, with 2011 sales of $16.2

billion. As leader in automotive safety, TRW produces active systems in braking, steering and

suspensions and sophisticated occupant safety systems, as seat belts, airbags and steering

wheels. The TRW factory seeks to improve the health and safety of workers in production

workplace through the adoption of FI-WARE technologies.



53


Simplify ECOGRAI, Phase 1:

This phase allows to collect the information concerning the system.

Elements of the system:

Prevention technician

H&S coordinator

Blue collar worker

Information systems technician

Manager of the company

Operation technician

Production line

Wharehouse

Functions (Static) and Processes (Dynamic):

TRW assemblies and manufactures power steering systems for passenger cars and

commercial vehicles

Boundary of the system:

Customers

Dynamic, evolution of the system

Objectives for scenario 1 (given by TRW):

Effective and consistent prevention strategy

Optimization of prevention costs

Objectives for scenario 2 (given by TRW):

Reduction of accidents and incidents

Increase of the productivity


This phase allows to determine the Decision Variable or the Action Variable implemented in

order to reach the objective.

Constraints in order to define the DV/AV

Criteria in order to choose the DV/AV:

Safety

Decision Variables / Action Variables:

To implement The FASyS system


This phase allows to choose the performance indicator allowing to measure the reaching of

the objective.

List of PIs for scenario 1 (given by TRW):

Decrease of events: Reduction of the number of accidents and incidents in the factory



54


Optimization of cost of accidents and incidents: Increase of the profitability of the

investment in preventive strategy

Decrease of errors in the prevention strategy: Reductions of the human errors in the

design of the planning

Increase of the modelled risks and active preventions: Number of risks that has been

defined using the new system

PIs for scenario 2 (given by TRW):

Decrease of events: Reduction of the number of accidents and incidents in the factory

Decrease the rate of absenteeism: Reduction in the average number of workers sick

leave

Increase the number of alarms and alerts: Rise in the risk detections, alarms and

warnings

Increase the number of H&S systems: Rise in the deployed monitoring systems

Decrease the number of workers with diseases: Reduction in occupational diseases

Increase the number of training sessions: Rise in the training sessions regarding H&S

Increase of the productivity: Rise in the produced units

Example of triplets Objectives, DV/AV, PI:

As there are 2 business scenarios in this case, the first objective of the second scenario is

chosen to simplify. This objective corresponds to the safety criteria that the trial seeks to

optimize among others. A performance indicator derivative of the 1st PI (for example: the

ratio) of the 2nd

scenario is defined to measure the reaching of the objective by using the

DV/AV.

Figure 9 Example of triplets Objectives, DV/AV, PI for TRW

* to be defined in the Instantiation Process

3.3.1.3.3 For Digital factory (Augusta Westland)

AgustaWestland, the Anglo-Italian helicopter company owned by Italy’s Finmeccanica, is a

total capability provider in the vertical lift market. Through its rotorcraft systems design,

development, production and integration capabilities, its experience in the training business

and its customer focused Integrated Operational Support solutions, the Company delivers

unrivalled mission capability to military and commercial operators around the world.

This expertise, backed by technological excellence and innovation, makes the Company a

leader in a number of the world’s most important helicopter markets offering the widest range

of advanced rotorcraft available for both commercial and military applications.

The trial will take place in the FALs (final assembly line) of some AUGUSTA WESTAND

plants

The correct use and management of tools is a very important point for the assembly and the

future maintenance of the H/C. So, safety and control of tools are one of priorities for the

Trial.

DV/AV: To

implement

The FASyS

system

PI-Ratio: Number of accidents and

incidents in the factory before and after

the DV/AV implementation during a

period*

Obj:

Reduction of

accidents and

incidents Formula : As is



55



This phase allows to collect the information concerning the system


FALs (final assembly line)

FAL plants collect parts

Technicians at the shop floor

Job Card

The logbook

Service Stations H/C maintenance

Smart toolbox (sensor)


To manufacture helicopter

To operate the helicopter maintenance


Other Augusta Westland plants

Vendors of parts

Commercial and military customers

Dynamic, evolution of the system:

To create e-logbook to be archived and accessible in the system for future search.

For each helicopter the system will create a serialized BoM that will be useful for a

better control of the helicopter configuration

Objectives:

Reduction of time of the preparation of the final version of logbook and relevant data

Improvement of H/C Configuration Control

Improved the tools tracking management

Support and improvement FOD prevention






Quality

Cost

Security


To create a digital logbook (electronic version of logbook starting from its paper

format)

To develop sensors and applications to control the tools used with the Toolbox




56



the objective.

List of PIs for scenario 1 (given by AUGUSTA WESLAND):

Reduction of time of logbook preparation and relevant data: Reduction of the average

time spent to get the logbook in its final version

Improvement of H/C Configuration Control, Support and improvement FOD:

Reduction of the number of time of errors in managing logbook not up to date

Improved the tools tracking management: Reduction of times spent in FOD prevention

Improved the tools tracking management: Time spent in monitoring the “tool story”


In this case, according to the priority of safety whished by the Trial, the second relevant

objective is chosen in the list corresponding to the safety criteria. The performance indicator

derivative of the 4th

PI (for example: the ratio) is chosen.

Figure 10 Example of triplets Objectives, DV/AV, PI for Agusta Westland


3.3.1.3.4 For Virtual factory SEGEM Macbo (Geoloc System) The activity of SEGEM-Macbo (SEM), an SME in the Aquitaine region, is the engineering and

production of individual machines and complete plants for the processing of logs, as well as

accompanying services and after-sales support. The customers are sawmills, paper-mills and

collectivities using wood-energy. In addition to an internal supplier delivering parts and machines,

some external suppliers provide parts, machines and complementary engineering.

The trial is faced with the necessity of reducing engineering delays (Time-to-market) and costs (Cost-

to-Market).


This phase allows to collect the information concerning the system


Provider

Buyer

Internal services

Collector

Data management,

production process,

after-sales,

design,

commercial

Obj: Improved

the tools tracking

management

DV/AV - To

develop sensors

and

applications to

control the

tools used

(Toolbox)

PI: Ratio: Time spent in

monitoring the “tool story” before

and after the implementation of the

DV/AV during a period*

Formula : As is



57



To develop special machinery for the wood industry


Suppliers, customers, co-traitors

Dynamic, evolution of the system

Objectives:

Reduce quoting lead time and increase accuracy

Reduce Call for Tender cycle

Automate Data management Process






Time


To use the FITMAN platform services



the objective.

List of PIs for scenario 1 (given by SEGEM MACBO):

Quoting lead time : Lead time between the customer’s request and quote

% of quoting lead time improvement : Ratio between new and past quoting lead time

Product cost and lead time: Relative difference between estimated and real values:

Measures the accuracy of quotes

Ratio between actual and planned product lead time: %

Number of iterations in the quoting process: Number of cycles between the company

and the customer to achieve the final version of the quotation

Number of partners answering to the tenders: Number of partners answering to the

tenders, to measure the attractivity of the offers

Number of tasks in data management process: Number of tasks in data management

process to show the complexity of the process

Number of manual tasks in the data management process: Number of manual tasks in

the data management process, measures the degree of automation

Number of tasks to ensure data interoperability: Number of tasks without added value

in the data management process

Time to market : Lead time between the idea of the new product validated by the CEO

and the date of commercialisation to the customer




58


In this case, according to the priority of the reduction of the “Time to market” whished by the

Trial, the first relevant objective in the list having an influence on the development time

reduction is chosen. The delay is the criteria to optimize among others. The performance

indicator derivative of the 1st PI (for example: the ratio) of the PIs list is chosen.

Figure 11 Example of triplets Objectives, DV/AV, PI for Geoloc System


3.3.1.4 Data Elaboration

After Data Collection, Data Elaboration is needed. In general, all the collected data will be

made available in order to be subsequently extracted, analyzed and re-used by the different

users as they prefer. In particular, the collected data will be intensively elaborated in WP7

and, as a consequence, only some suggestions and guidelines are provided in this Paragraph in

order to facilitate and support the task. In this respect, the following considerations have to be

considered as practical “examples” rather than real rigorous methodologies to respect.

a. Self-certification Form: One possibility to aggregate the final results of the Verification

and Validation of the different Software Components is to gather the different GEs, SEs,

TSCs and TICs of the specific Trial’s Business Scenario in a common framework, in order to

allow a rapid and efficient comparison among the different outcomes obtained:

STEPS RECOMMENDED

TECHNIQUE

ALTERNATIVE

TECHNIQUE (if any)

RESULT

(YES/NO/NA)

COMMENTS (if

any)

GE1 P4 P5 GE2 P4 P5 ….. GEn P4 P5 SE1 P1 P2 P3 P4 P5 SE2 P1 P2 P3 P4

Obj: To

reduce the

quoting lead

time

DV/AV - To use

the FITMAN

platform services

PI : Ratio: Average lead time

between the customer’s request and

quote before and after the

implementation of the DV/AV

during a period*

Formula: Total processing time of

customer’s request to quote / total

number of requests



59


P5 ….. SEm P1 P2 P3 P4 P5 TSC1 P1 P2 P3 P4 P5 TSC2 P1 P2 P3 P4 P5 ….. TSCk P1 P2 P3 P4 P5 TIC1 P1 P2 P3 P4 P5 TIC2 P1 P2 P3 P4 P5 ….. TICl P1 P2 P3 P4 P5

Table 18 Example of Views of Self-certification results

As it is depicted in Table 18, for each Software Component it will be possible to see the

related Steps, the Verification or Validation Recommended Technique, the eventual

Alternative Technique used, the final Result (Yes/No/NA) of the Verification or Validation

test and the eventual Comments of the users.

The previous aggregation will simply provide a complete and orderly list of all the Software

Components concerned. Moreover, another Trial’s Business Scenario-specific aggregation

could be done, in which all the Software Components verified are divided for a generic Trial



60


according to the specific Trial’s Business Scenario to which they belong, as it is depicted in

the following Table 19 (in which it is provided only a symbolic example):

STEPS RECOMMENDED

TECHNIQUE

ALTERNATIVE

TECHNIQUE (if any)

RESULT

(YES/NO/NA)

COMMENTS (if

any)

Trial’s

Business

Scenario 1

GE1 P4 P5 SE4 P1 P2 P3 P4 P5 TSC3 P1 P2 P3 P4 P5 TIC1 P1 P2 P3 P4 P5 Trial’s

Business

Scenario 2

GE2 P4 P5 GE3 P4 P5 SE6 P1 P2 P3 P4 P5 TSC2 P1 P2 P3 P4 P5 TIC1 P1 P2 P3 P4 P5 Trial’s

Business

Scenario 3

GE2 P4 P5 SE4 P1



61


P2 P3 P4 P5 SE5 P1 P2 P3 P4 P5 TSC8 P1 P2 P3 P4 P5 TIC4 P1 P2 P3 P4 P5

Table 19 Example of Views of Self-certification results for each Trial’s Business Scenario

Also in this other aggregation it will be possible to see for each Software Component the

related Steps, the Verification or Validation Recommended Technique, the eventual

Alternative Technique used, the final Result (Yes/No/NA) of the Verification or Validation

test and the eventual Comments of the users.

Obviously, in this second case each Software Component could be “repeated”, because e.g. a

GE can in general belongs to one or more Trial’s Business Scenario-specific integrated

solutions.

b. PIs Measurement Form: For this second kind of Form an elaboration could be done, i.e.

the collection of the values of different PIs according to the established temporal granularity.

On the base of this, different levels of aggregation can be organized. First of all, for a generic

Trial’s Business Scenario, the following check-list can be organized:

TRIAL’S

BUSINESS

SCENARIO

STEPS PIs VALUE P1 VALUE P2 ….. VALUE Pm

T1 T2

Table 20 View of PIs Measurement Form results

For the generic Trial’s Business Scenario, the values of each PI are collected and monitored

step by step according to the established temporal granularity.

It is well-known that the Basic Element which is going to be measured by means of this

specific kind of Form is the Quantitative Indicator but that, on the base of this, different levels

of consultation of the results can be used:

PI

Step

Trial’s Business Scenario

Trial



62


as well as:

Smart/Digital/Virtual Domains


as it has been promised in Paragraph 1.6.

c. Community-based Form: For the Community-based Form, for each Trial’s Business

Scenario the following kind of table could be developed:

TRIAL’S

BUSINESS

SCENARIO

STEPS Qualitative

Indicators

RESULT

T1 T2

Table 21 Example of view of Community-based results

Obviously, in this case the Indicators will be Qualitative and, as a consequence, the result will

not be represented by an “objective” value but by another kind of synthetic evaluation (e.g. a

mark, a qualitative judgment, a prevalent answer), that will result from the “average” of all

the answers collected from all the users of the specific Trial’s Business Scenario.

Also in this case, basing on the proper Basic Elements (i.e. the Qualitative Indicators),

different levels of consultation of the results can be used:

Qualitative Indicator

Step

Trial’s Business Scenario

Trial

as well as:

Smart/Digital/Virtual Domains


as it has been promised in Paragraph 1.6.

If the specific Trial’s Business Scenario will also include supplementary Community-based

(i.e. Crowdsourcing) Techniques (see Paragraph 3.3.1.2), further Data Elaboration should

obviously be addressed. In particular, for a generic Trial’s Business Scenario, the following

view could be developed:

TRIAL’S

BUSINESS

SCENARIO

STEPS

Proposed Community-

based (i.e.

Crowdsourcing)

Technique

Implemented

Community-based

(i.e.

Crowdsourcing)

Technique (if any)

RESULT

P5


feedback tools, Open calls

for testing scenarios/

trials, Workshops and

specific working groups,




63


tools, Social networks

P4


tools for ex-ante



crowdsourcing

P3


tools for ex-ante



crowdsourcing

P2

Physical or online

workshops, Readily-

available prototyping


deliberation platforms

P1

/

Table 22 Example of view of supplementary Community-based results

3.3.1.5 Data Presentation and Distribution

The tool provides a set of plotting functionalities to visualize the results. Users (provided with

appropriate privilege) can create charts to represent the collected data.

Data will be available in MS Excel and MySQL format for personalized plotting and

elaborations. These files will be the base for in-depth analysis by each individual Trial Owner

and the base for the analysis in WP7 (i.e. Lessons learned, Recommendations, Best Practices).

3.3.2 V&V Communication Package

As already extensively discussed, the V&V Communication Package represents the V&V

Package Technology Section’s “meta-structure”, i.e. all the information and instructions

needed to manage and support a correct and satisfying use of the Forms themselves.

The V&V Communication Package is composed by two different complementary parts:

1. the General Information Section, explaining to all the possible beneficiaries the main

points related to all the different aspects of the FITMAN V&V Package;

2. the Instructions and Support Section, explaining in detail to all the possible

beneficiaries the modalities for the correct implementation and use of the FITMAN

V&V Package in the eleven Trials;

Our aim is now to conduct a “meta-analysis” on the V&V Communication Package, i.e. an

explanation and description of all the contents that will be included in it.

In the next Paragraphs will be discussed the two Sections of the V&V Communication

Package only from a “content” point of view. For the “tools and technological” one please see

the proper Chapter 4.

3.3.2.1 General Information Section

In this section, V&V Package’s overall Definition and Methodology are presented. Finally, a

brief description of all the GEs, SEs, TSCs and TICs involved in the FITMAN Project is

exposed. In particular, the General Information Section will be structured as follows:

FITMAN V&V Methodology:



64


- Objectives

- Verification and Validation

- 7-Step Methodology

- Product-specific/Trial’s Business Scenario-specific Steps

V&V Package Presentation:

- Objectives (combination of methodological and technological components

for the collection, elaboration and presentation of the data)

- Organization (three Sections, three different scopes)

- Technology Section’s lifecycle (i.e. Data Collection, Data Elaboration,

Data Presentation)

V&V Package Methodology:

- V&V Package Methodological Passages

- Three kinds of Forms (i.e. Self-certification Form, PIs Measurement Form,

Community-based Form)

ECOGRAI Simplified Methodology

Indicators’ types:

- Technical (Quantitative, Qualitative)

- Business Performance (Quantitative, Qualitative)

Indicators’ list:

- Generic Technical Indicators

- Technical Indicators for FI-WARE (i.e. GE Versatility and GE

Openness)

- ENAPS Business Performance Indicators

GEs, SEs, TSCs ad TICs List

3.3.2.2 Instructions and Support Section

In this section, after the overall outline of the previous one, the specific instructions in order

to fill the different Forms of the V&V Package are clearly provided and analyzed. In

particular, the practical modalities and advices to fill the Forms, the updating and the timing

issues are addressed. Moreover, some other useful indications and support tools are provided.

Log-in and help

SurveyMonkey User’s Guide

Instructions for Data Collection: In this section, the modalities for Data

Collection according to the different forms of the V&V Package are described.

In particular:

i. Self-certification Form: In this first part, both methodological and

technological issues for Data Collection relevant to the users are clearly

explained.



65


ii. PIs Measurement Form: In this second part, both methodological and

technological issues for Data Collection relevant to the users are clearly

explained.

iii. Community-based Form: In this third and last part, both

methodological and technological issues for Data Collection relevant to

the users are clearly explained.

Privacy Issues: In this section, the issues about access management and

continuous updating of the data are addressed. In particular, the modalities for

the access of the different categories of users to the different specific kinds of

Forms (divided into the three aforementioned parts) for the first and then for

the following times is explained, taking obviously into account the specific

features of the particular software (i.e. SurveyMonkey, see Chapter 4) used for

the practical implementation of the V&V Data Collection phase.

Decalogue: In this section, a simple and exhaustive “Decalogue” of advices

and errors to avoid, as it has been already presented in the Deliverable D2.1

[1], is inserted. This Decalogue represents a useful and smart guide resumed in

some main points that constitutes, in addition to the previous indications, a

continuous support for all the categories of users involved in the collection of

the different kinds of data.

FAQs: In this section, in addition to the previous one (i.e. “Decalogue”), the

most common FAQs about the correct collection of the data (e.g. about the

accesses, the updating of the values, the deadlines to respect, the timing of the

insertion of the values), are presented. The different FAQs are organized in

five specific sub-sections:

- Self-certification Form

- PIs Measurement Form

- Community-based Form

- Privacy Issues

- Indicators

Glossary: In this section it is possible to find a wide resume of all the most

important terms used in the V&V Package (e.g. Verification, Validation,

Criterion, PI,…), as they have been defined in the Deliverable D2.1 [1].

GEs, SEs, TSCs and TICs used per Trial’s Business Scenario

Contacts: In this final section, the information (e.g. phone number, e-mail,…)

about the people to contact in order to have more explanations about the

different aspects of the V&V Package is provided.



66


4. V&V Package tools and technologies

Since all the content and methodological aspects of the generic FITMAN V&V Package have

been just properly addressed, it is now necessary to deep into the practical technological

solution adopted.

The generic FITMAN V&V Package is practically implemented on the FITMAN Website.

Access to the Forms is granted by the survey owner by sending a personalized e-mail to Trial

Stakeholders involved in each specific case.

Please refer to

Appendix 4 – SurveyMonkey tool [15] for the presentation of the key steps to setup a survey,

define and invite participants, collect data and consolidate them.

4.1 V&V Communication Package The V&V Package Home Page presents the starting buttons of the four main elements of the

V&V Communication Package that has already been described, as it is depicted in Figure 12:

Figure 12 FITMAN V&V Package support page

By clicking on the proper button, it will be possible to access the specific element.

At this link the implementation of the support page: http://www.fitman-fi.eu/intranet/wp-

folders/wp2-fitman-verification-validation/test-environment/fitman-support-page.

While it is useless to deep into the description of the simple static web pages of the V&V

Communication Package containing the related information, it is on the other hand important

to analyze the technological solution adopted for the Technology Section. In doing this, we

use the usual three-phases classification already extensively addressed.

4.2 Data Collection We decided to use for Data Collection the SurveyMonkey software, because if compared with

Google Form is more intuitive, user-friendly and customizable and allows a more complex

collection of data and results (for details please see Paragraph 2.3.1).

As it has been already explained, to every single Trial’s Business Scenario are assigned three

different kinds of Forms, i.e. the Self-certification, the PIs Measurement and the Community-

based Form. More practically, every kind of Form will be organized in different Sub-forms,

according to the specific Basic Element which has to be analysed (see Paragraph 3.3.1.1 for

details). The specific URL of the Trial’s Business Scenario will be provided to the single Trial





67


Owner, leading him to a Trial’s Business Scenario-specific Area where the different kinds of

Forms are integrated in a common solution. In this respect, the Trial Owner will be able to

practically choose by means of some options each time the proper Sub-form, as it is depicted

in Figure 13:

Figure 13 Data Collection Forms conceptual schema for a Trial’s Business Scenario

For simplicity, only one GE, SE, TSC and TIC is taken into account for a Trial’s Business

Scenario.

Since accessed by means of a specific Password the Trial’s Business Scenario-specific Area,

the Trial Owner can decide the specific Sub-form to access, fill it and then exit the Area.

Let’s see now in detail some examples of the specific layout of the three different kinds of

Forms for a generic Trial’s Business Scenario.



68


4.2.1 Self-certification Form

It is made by different sequential pages (i.e. different sequential Sub-forms), in which Data

Collection is organized according to the nature of the specific Software Component (i.e.

GE/SE/TSC/TIC). In particular, one and only one page (i.e. one and only one Sub-form) will

be assigned to every single Software Component. Within the single Sub-form, for every Step

involved a specific sub-part will be reserved, correlated with the particular Recommended

V&V Technique, a box in which is possible to put the eventual Alternative V&V Technique

used, three buttons for the choice of the Result of the test (i.e. “Yes”, “No”, “NA”) and

another box for the eventual writing of additional Comments. The Figure 14 shows the

situation for a generic SE taken as an example:

Figure 14 Example of a SE’s Self-certification Sub-form

4.2.2 PIs Measurement Form

It is made by different sequential pages (i.e. different sequential Sub-forms), in which Data

Collection is organized according to the nature of the Quantitative Indicators (IT or Business

Performance) if they refer to the whole Trial’s Business Scenario or alternatively to the

specific Software Component. In particular, one and only one page (i.e. one and only one

Sub-form) will be assigned to every area of interest of the Quantitative Indicators (i.e. IT and

Business Performance) and to each of the software Component. In the Figures below are

showed an example for a generic SE and for a Trial’s Business Scenario’s Business and IT

Sub-form:



69


Figure 15 Example of a PIs Sub-form for a SE

Figure 16 Example of a Business Sub-form for PIs Measurement



70


Figure 17 Example of an IT Sub-form for PIs Measurement

4.2.3 Community-based Form

One and only one page (i.e. one and only one Sub-form) will be assigned to the whole Trial’s

Business Scenario. In Figure 18 is showed an example for a Trial’s Business Scenario Sub-

form:

Figure 18 Example of a Community-based Sub-form



71


4.2.4 ICT SMEs’ effort to implement FITMAN V&V Process

The clear and precise Instantiation’s effort for each Trial will be deeply addressed

subsequently in the Instantiation phase. Anyway, the FITMAN web-based V&V tool has been

thought in order to reduce the eventual supplementary actions of end-users. In fact, the great

advantage is represented by the fact that all the Data collected will be made available by

means of efficient and advanced tools which, beyond the pure Data Collection’s tool (i.e.

SurveyMonkey), will include among others MS Excel and MySQL. On top of that, all the

supporting kinds of Forms have been set up in order to make as rapid as possible the specific

configuration which will be managed during the Instantiation phase. The issue of ICT SME’s

effort has been hence taken into account both from a technological and methodological point

of view.

4.3 Data Elaboration, Presentation and Distribution We decided to use MS Excel for the Data Elaboration phase, because, even if compared with

MS Access allows simpler and rawer elaborations, on the other hand is more suitable for the

numerical elaboration of values and can be directly linked with SurveyMonkey, which

automatically generates the necessary spread-sheets. Moreover, MS Excel allows to store the

data and to visualize and filter them according to simple selections and commands. In few

words, MS Excel can perfectly responds to every need we have by means of simpler and more

useful solutions, allowing furthermore the extremely direct integration with the Data

Collection’s software SurveyMonkey (for details please see Paragraph 2.3.1).

MS Excel enables the different needed Data Elaboration functionalities exposed in Paragraph

3.3.1.4 by means of its basic functionalities. Moreover, it allows the storage of the different

data collected serving as a simplified Database. The very strength of MS Excel remains

however the possibility to directly publish .xls files by means of SurveyMonkey itself, thanks

to the proper “Exportation” functionality:

Figure 19 MonkeySurvey Export function

The .xls files produced can be then stored directly on the web by means of the proper

SurveyMonkey functionality:

Figure 20 SurveyMonkey web storage function



72


or alternatively they can be “traditionally” shared on the web by means of other tools such as

Dropbox.

4.4 V&V Package instantiation approach We have presented and discussed in detail the whole generic FITMAN V&V Package both

from a methodological and technological point of view. The aim of the present Deliverable

D2.3 is in fact the development of a generic integrated V&V solution for the FITMAN

project’s eleven Trials. The following T2.4 will operate the necessary instantiation in order to

obtain the proper Trials’ Business Scenarios. In the light of this, we provide now some

guidelines for a correct instantiation of the present generic FITMAN V&V Package in the

eleven Trials.

The main result of this Deliverable D2.3 is the development of generic Forms, methodologies

and support framework in order to correctly drive the FITMAN V&V Process along its three

different chronological phases, i.e. Data Collection, Data Elaboration and Data Presentation.

As far as for all these three phases proper generic Forms and tools have been ideated, it is

now fundamental to clearly explain how in T2.4 people should operate in order to transform

all these frameworks in customized and directly usable tools for all the specific eleven Trials

of the FITMAN project.

4.4.1 V&V Package instantiation objectives and scope

The FITMAN V&V Package is a generic integrated V&V solution for the FITMAN project’s

eleven Trials. It includes all the methods and tools to select/define the Indicators, collect the

data using different kinds of Forms, elaborate and present the data.

In an individual Trial’s business Scenario not all Indicators and kinds of Forms are needed.

To keep the V&V assessment as simple as possible for the Trials the V&V package needs to

be instantiated for the Trials. The instantiation means specifying and adapting the generic

FITMAN V&V Package for each FITMAN Trial’s Business Scenario, including the selection

of Indicators and prefilling potential Indicators templates or sheets offered by the V&V

Package itself. The scope of instantiation is the Trials; thus the instantiation is performed for

V&V phases which are performed by/in the Trials. It means that the instantiation does not

necessarily include all those V&V Product-specific phases which are performed by the

software providers outside the Trials. As a result V&V Package instantiations will be

received, one for each of the Trials’ Business Scenarios defined in each of the eleven Trial

[D1.1] Each Trial’s Business Scenario is defined as “each of the specific selected cases within

the TRIAL, in which we can generically improve the current outcome by applying the Future

Internet Core Platform” (GE, SE, TSC and TIC). This is performed in the following WP2 task

T2.4.

The V&V Package offers generic Forms, methodologies and a support framework in order to

correctly drive the FITMAN V&V Process. In the first, instantiation phase supports the Trials

in the definition of the IT and Business Performance Indicators. For Business PIs definition

the FITMAN simplified ECOGRAI methodology described in D2.2 [2] is used. The V&V

Package also supports the three chronological phases in the PIs measurement, i.e. Data

Collection, Data Elaboration and Data Presentation. The data collection and measurement

takes place in the Trials in the experimentation phase (WP4-6 Smart, Digital, Virtual). The

instantiation phase means the preparation for the actual measurement: in the instantiation

phase the measuring objects and Forms are prepared, to be ready for the experimentation

phase.

4.4.2 Phases and roles in the instantiation

The instantiation phases are described in detail in D2.4. Roughly the plan is to support the

Trials in the instantiation through the following means:



73


Supporting material as offered by the V&V Communication Package;

Training events, hands-on training. This will happen in phase the Trials are prepared

for the training. The partners offering the training are WP2 task leaders, involved in

the methodology development: VTT, NTUA, VLAB, POLIMI. The training is

directed towards the Trial support organizations;

Final definition of PIs and continuous support: this is offered by the Trial support

partners. The Trial support partners are partners which participate in the Trials AND

which have resources reserved for Task 2.4. They are listed in the following Table 23:

Trial Support partner

1 Volkswagen IPK

2 TRW ATOS

3 AugustaWestland TXT

4 Whirlpool POLIMI

5 Piacenza Softeco

6 A.P.R. Lyon2

7 Consulgal Uninova

8 TANet Coventry

9 COMPlus IPK

10 Machinery for Wood UBX1

11 AIDIMA UPV

Table 23 Support partners for the Trials

4.4.3 Forms definition and instantiation

The whole general Data Collection Setup Process has been defined in Paragraph 3.3.1.1. As a

consequence, basing on the four specific Data Collection Setup Sub-processes, the practical

Instantiation Process will change according to their features. Anyway, the structure of the

Data Collection Form has been already completely defined and hence the Instantiation will

subsequently involves only the selection of the Trial’s Business Scenario’s specific

Quantitative and Qualitative Indicators and Software Components and finally their practical

composition and integration in the different kinds of Forms implemented by means of

SurveyMonkey.

On the other hand, for Data Elaboration and Presentation, the Instantiation Process will

simply involve the creation of the needed schemas in MS Excel (among the ones suggested)

and the customized extraction of the final results provided by means of MySQL.

In general, please refer to

Appendix 4 – SurveyMonkey tool [15] for examples of data plotting.



74


5. Confidentiality Aspects

Data confidentiality aspects are to be taken into consideration in WP2 where Trials give their

Trial’s Business Scenario-specific data input to the assessment. The confidentiality levels are

different in different steps of the assessment process. These are described in the following

Table 24:

Data Confidentiality level

Raw data from each trial, i.e. values given

for indicator calculation and answers given to

qualitative questions

Assumed to be the most sensitive data

The calculated indicator values Can still be sensitive

The comparison data, e.g. average values Not sensitive in case there has been enough

raw data input available

Table 24 Confidentiality level according to the kind of data

Confidentiality is an important and a challenging issue in FITMAN V&V assessment due to

the fact that the deliverables are public, and on the other hand the Trials are very different

from each other, so that the amount of data for each specific Indicator will not necessarily be

large. For anonymity at least three values would be needed in order to have the calculated

averages and other information anonymous. This will most probably not happen with all

Indicators and questions in the FITMAN V&V. However, for the quality of the results it

would be important not to lose the significance of the results. On the other hand, the

confidentiality issue probably will affect the quality of the data and the willingness to collect

the data. For these reasons we take the following actions in T2.3 and T2.4:

1) The quantitative and qualitative questions will be chosen so that they:

o do not include the most confidential issues of the companies, if

possible;

o where possible, information is asked to cover at least two periods or

two different dates in order to be able to measure the change. The

change (e.g. percentage) is not as sensitive as numeric values;

o where possible, information is asked using a scale, which might not

be found to be as sensitive as other types of quantitative or

qualitative data.

2) The Trials will be informed about this issue in the V&V Communication

Package, and a part in the presentation and guidance section will be

included to explain the confidentiality issues for the Trials;

3) Trials’ Business Scenarios-specific accounts and passwords are used in the

assessment tool;

4) The data aggregation will be done in a manner that it will keep as much

information anonymous as possible.

The privacy issues of the FITMAN project have been described as a whole in the Description

of Work. The FITMAN project will comply with applicable data protection legislation. All

partners in the consortium will comply with relevant EU Directives on data protection

(95/46/EC and 2006/24/EC) and will keep abreast with current proposed changes to the

European data protection framework. An External Ethics Board has been formed and is in

charge of supervising and analysing the privacy and personal information management issues.



75


6. Conclusions

The previous sections has addressed the definition and development of a viable approach for

implementing a V&V Methodology and related set of Indicators into a complex environment

like the FITMAN Project is. This Project addresses in the manufacturing space a variety of

production realities belonging to different sectors, dimensions and background. For each of

the 11 Trials we have also one or two Business Scenarios, addressing different objectives,

processes and technological implementation. Finally we have to keep in mind that all these

companies are dealing with the day-by-day business.

On the other side the actual measure of the effectiveness and efficiency of the FI-WARE

enablers implementation is a key take-away of this project. One of the major FITMAN results

is to establish in objective way what are the benefits of utilizing a complete new paradigm in

Application development. Again, only with a reliable measuring system in place any

conclusion drawn out from the Trials implementations is not or scarcely meaningful.

All these conditions challenged the definition of a generic V&V Package, in the sense that we

had to keep in mind the methodological foundation of the approach, but at the same time we

need that it is generic enough to be adopted for the different scenarios and light enough not to

be an additional burden for Trial Owners.

At the end, we can draw in few words the following conclusions:

First: the defined V&V Package is generic enough to be instantiated in any type of

Trial/Business Scenario. The way Indicators are defined and inserted in the survey

platform allows to collect data (Technical and Business Performance) at any level

defined by the methodology and for any type of Business Scenario.

Second: the selected technological platform ( [15]) ensures a simplified development

environment for the creation and modification of surveys (forms, links,

documentation). Built-in functions allow to create sophisticated Data Collection

forms, navigation, control of access and security;

Third: a great attention has been devoted to the creation of a complete V&V

Communication Package (including informational and training material) to support the

instantiation of all the requested information and to train people at different levels and

with different responsibilities. A proper section in the FITMAN Website provides an

on-line tool for information distribution and support;

Fourth and last: we think that this V&V Package (and the related Methodology) can

be beneficial not just for FITMAN Project’s Trials, but can be leveraged also by other

FI-PPP Phase II and Phase III Projects and eventually constitute a standard for V&V

of FI-PPP Projects and implementations.



76


7. Appendixes

7.1 Appendix 1 - Glossary

Self-Certification Once a specific Software Development phase (15) is carried out by

development team according recognized standards, it is assumed that the

module is compliant to Verification standards described in D2.1. For

such reason the V&V Package simply collect the declaration from the

module/phase owner.

Performance The performance of an organization ( enterprise, system, trial)

measures the evolution of this organisation towards the objectives,

under the influence of external or internal factors

Indicator An indicator can be defined as an entity that helps to understand a

situation or to give a state . Therefore it can be a sign, a number, a

graphic,…..

Performance

Indicator

PI (Performance Indicator) is a quantified data which measures the

efficiency of action variables or decision variables, in the frame of the

achievement of an objectives defined for this system

Key Performance

Indicator

A KPI is a Performance Indicator which allows to define the

Performances of a system (Enterprise or trial) at the strategic level

(global performance).

Objective Objectives allow to define the results that the global company, or a part

of the company (trials) must reach.

Decision variable

(DV)

Action variable

(AV)

A Decision variable is an action determine by a decision maker for

reaching the objectives of an enterprise, a sytem, a trial, …..

An Action variable is an action which allow to reach an objective

defined for an enterprise, a sytem, a trial,….

In fact, the 2 variables represent very similar concepts, the difference

coming from the human decision

Constraints The constraints are the limitation on DV or AV to reach the objectives.

Criterion A standard on which a judgment or decision both at business and at IT

level may be based.

In D2.1, a criterion reflects the diverse parameters that are involved and

influence the V&V activities of the trials.

In D2.2, a criterion allows to choose the right variable to reach the

objectives of an entreprise, a system, a trial….

7.2 Appendix 2 – A practical example

Illustration with example “Capacity planning”:

The capacity often called “load” is the quantity of work to be realized by a load center for a

given period. It is expressed generally in hours machines, in hours workforce or in rate of

production.

A load center can be a manufacturing unit, constituted by one or several persons and/or

machines which can be considered as an entity from the point of view of the planning of

needs in capacity and of the scheduling.

The load plan determines the matching between the amount of work allocated and the

nominal of production resources capacity. It is the tool for the shop floor control.

It happens that this capacity is exceeded and obliges to attempt an action to balance it.



77


It is the object of the example by the application of the ECOGRAI simplified method that

follows:

Objective: To smooth the loads

Decision/Action variables (DV/AV):

Four choices are possible:

To increase the capacity of certain resources by hiring staff

To turn to the subcontracting

To negotiate the delivery date

To suggest overtime

Criteria:

Three choices to optimize are possible:

The cost

Regarding the cost, one can choice between the hiring, overtime or subcontracting.

The delay

Generally, one can always negotiate the delay with the customer without request of

increase of price.

The quality

It depends if the enterprise decides to make it itself or to trust the subcontracting.

Constraint:

The number of hours that an employee is authorized to perform according the social

law.

Performance Indicators:

% Appropriateness (adequation) rate of the load with the capacity before and after the

(DV/AV) implementation for a period* (Total of load used/Total production capacity)

% Rate of use of production capacities before and after the (DV/AV) implementation

for a period* (real production / Production capacity)

% Cost linked to the subcontracting before and after the (DV/AV) implementation for

a period*(Number of hours subcontracted / Number of hours produced)

Obj : To smooth

the loads

DV/AV:

-To increase the capacity

of certain resources by

hiring staff

-To turn to the

subcontracting

-To negotiate the delivery

date

PI: % Rate of use of

production capacity before

and after the (DV/AV)

implementation for a period*

Formula: Real production /

Production capacity

Formula: Total of load

used/Total production

capacity

Obj : To smooth

the loads

DV/AV:

-To increase the capacity of

certain resources by hiring

staff

-To turn to the

subcontracting

-To negotiate the delivery

date

-To suggest overtime

PI: % Appropriateness rate

of the load with the

capacity before and after

the (DV/AV)

implementation for a

period*



78


Technical

Indicator

A Technical indicator is an entity which evaluate the technical

performance of a technical system.

Example: GEs, SEs, TSCs.

Validation The process of providing evidence that the software and its associated

products satisfy system requirements allocated to software at the end of

each life cycle activity, solve the right problem (e.g., correctly model

physical laws, implement business rules, use the proper system

assumptions), and satisfy intended use and user needs.

Verification The process of providing objective evidence that the software and its

associated products conform to requirements (e.g., for correctness,

completeness, consistency, accuracy) for all life cycle activities during

each life cycle process (acquisition, supply, development, operation, and

maintenance); satisfy standards, practices, and conventions during life

cycle processes; and successfully complete each life cycle activity and

satisfy all the criteria for initiating succeeding life cycle activities (e.g.,

building the software correctly).

7.3 Appendix 3 – Acronyms and abbreviations

GE Generic Enabler

SE Specific Enabler

TSC Trial Specific Component

TIC Trial Interface Component

TI Technical Indicator

BPI Business Performance Indicator

7.4 Appendix 4 – SurveyMonkey tool examples Here following few examples of SurveyMonkey features and functionalities divided in four

categories:

Obj : To smooth

the loads

DV/AV:

-To turn to the subcontracting

PI: % Cost linked to the

subcontracting before and

after the (DV/AV)

implementation for a

period*

Formula: Number of hours

subcontracted / Number of

hours produced



79


1. Form creation

2. User involvement

3. Data Collection

4. Data visualization and utilization (calculation and distribution)

7.4.1 Form creation

Here below some useful examples about the functionalities to correctly create Forms on

SurveyMonkey.



80




81


7.4.2 User involvement

Users can be involved, providing the direct URL to survey:

or sending a customized email to selected groups of users:

Ad hoc functions allow generation, management and tracking of involved people:



82


Specific settings allows to control the way the survey is carried out.

In any case a password (provided in person) is required to access the survey:



83


All answer to survey are directly associated and identified with the user, the IP address and

the time it is done.

Example of a message to a survey participant:



84


Before entering a password is required:

Example of data entering please refer to Paragraph 4.2.

7.4.3 Data Collection

Here below some useful examples about the functionalities to correctly collect data on

SurveyMonkey.



85


7.4.4 Data visualization and utilization (calculation and distribution)

SurveyMonkey tool provide a variety of building functions for data plotting and aggregation

based on sophisticated filtering and conditioning functions. Here following few examples of

possible diagram generated by the tool.



86




87


8. References

[1] “Deliverable D2.1 - FITMAN V&V Generic Method and Criteria Identification”.

[2] «Deliverable 2.2 - FITMAN V&V Business and Technical Indicators Definition».

[3] A. S. Jadhav and R. M. Sonar, “Framework for evaluation and selection of the software

packages:A hybrid knowledge based system approach,” The Journal of Systems and

Software, no. 84, pp. 1394-1407, 2011.

[4] I. Samoladas, G. Gousios, D. Spinellis and I. Stamelos, “The SQO-OSS quality model:

measurement-based open source software evaluation,” in Open Source Development,

Communities and Quality - OSS 2008: 4th International Conference on Open Source

Systems, Boston, 2008.

[5] T. L. Saaty, «How to make a decision: The Analytic Hierarchy Process,» European

Journal of Operational Research, n. 48, pp. 9-26, 1990.

[6] R. Harrison, S. Counsell e R. Nithi, «An Evaluation of the MOOD Set of Object-

Oriented Software Metrics,» IEEE Transactions on Software Engineering, vol. XXIV, n.

6, pp. 491 - 496 , 1998.

[7] M. S. Bandor, «Quantitative Methods for Software Selection and Evaluation,» Software

Engineering Institute, Carnegie Mellon University, Technical Note CMU/SEI-2006-TN-

026, 2006. http://www.sei.cmu.edu/library/abstracts/reports/06tn026.cfm, Pittsburgh,

Pennsylvania, 2006.

[8] J. P. Chin, V. A. Diehl e K. L. Norman, «Development of an Instrument Measuring User

Satisfaction of the Human-Computer Interface,» in CHI '88 Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems, Washington, DC, USA, 1988.

[9] H. Vieira and R.-J. Goncalves, “VALTE – Methodology for VALidation and TEsting of

Supply Chain Software Components,” in 15th International Conference on Concurrent

Enterprising - ICE Conference Proceedings , Leiden,Netherlands, 2009.

[10] G. Boloix and P. N. Robillard, “A Software System Evaluation Framework,” Computer,

vol. XXVIII, no. 12, pp. 17 - 26 , 1995.

[11] M. Rosemann e J. Wiese, «Measuring the Performance of ERP Software – a Balanced

Scorecard Approach,» in 10th Australasian Conference on Information System

Proceedings , Wellington, New Zealand, 1999.

[12] I. Vlahavas, I. Stamelos, I. Refanidis e A. Tsoukiàs, «ESSE: an Expert System for

Software Evaluation,» Knowledge-Based Systems, vol. XII, n. IV, p. 183–197, 1999.

[13] R. Oppermann e H. Reiterer, «Software evaluation using the 9241 evaluator,» Behaviour

and Information Technology, vol. XVI, n. 4/5, pp. 232 - 245, 1997.

[14] «Google Drive - Google Form,» [Online]. Available:

http://www.google.com/drive/apps.html.

[15] Surveymonkey, «http://www.surveymonkey.com,» [Online]. Available:

http://www.surveymonkey.com.

D2.3 FITMAN Verification & Validation Generic Assessment Package · 2017-04-24 · FITMAN Verification & Validation Generic Assessment Package 20/09/13 – V1.0 Document Owner: Giacomo

Documents