Deliverable No: D7.3 Final Evaluation Report - CORDIS...Deliverable No: D7.3 Final Evaluation Report i-Treasures ICT-600676 Filename: D7.3_deliverable_v8.docx Page 6 of 91 7.10.4 Evaluation

Project Title: i-Treasures: Intangible Treasures – Capturing the Intangible Cultural Heritage and Learning the Rare Know-How of Living Human Treasures

Contract No: FP7-ICT-2011-9-600676

Instrument: Large Scale Integrated Project (IP)

Thematic Priority: ICT for access to cultural resources

Start of project: 1 February 2013

Duration: 48 months

Deliverable No: D7.3

Final Evaluation Report

Due date of deliverable:

31 March 2017

Actual submission date:

31 May 2017

Version: 8th version of D7.3

Main Authors: Athanasios Manitsaris (UOM), Anastasios Katos (UOM), Eleni Katsouli (UOM), Maria Lazaridou (UOM), Dimitrios Theodorou (UOM), Evanthia Kalampoka (UOM), Leontios Hadjileontiadis (AUTH), Vasileios Charisis (AUTH), Stelios Hadjidimitriou (AUTH)

Project funded by the European Community under the 7th Framework Programme for Research and Technological Development.

Deliverable No: D7.3 Final Evaluation Report i-Treasures ICT-600676

Filename: D7.3_deliverable_v8.docx Page 2 of 91

Project ref. number ICT-600676

Project title i-Treasures - Intangible Treasures – Capturing the Intangible Cultural Heritage and Learning the Rare Know-How of Living Human Treasures

Deliverable title Final Evaluation Report

Deliverable number D7.3

Deliverable version Version 8

Previous version(s) Version 7

Contractual date of delivery 31 March 2017

Actual date of delivery 31 May 2017

Deliverable filename D_7_3_Deliverable_v8.docx

Nature of deliverable R

Dissemination level PU

Number of pages 91 (plus Appendices)

Workpackage WP 7

Partner responsible UOM

Author(s) Athanasios Manitsaris (UOM), Anastasios Katos (UOM), Eleni Katsouli(UOM), Maria Lazaridou (UOM), Dimitrios Theodorou (UOM), Evanthia Kalampoka (UOM), Paraskevi Kritopoulou(UOM), Christina Volioti(UOM), Leontios Hadjileontiadis (AUTH), Vasileios Charisis (AUTH), Stelios Hadjidimitriou (AUTH), Nikos Grammalidis (CERTH), Kosmas Dimitropoulos (CERTH), Filareti Tsalakanidou (CERTH),Giannis Chantas (CERTH), Francesca Maria Dagnino (CNR), Francesca Pozzi (CNR), Lise Crevier-Buchman (CNRS), Pierre Roussel (CNRS), Sohaib Laraba (UMONS), Sotiris Manitsaris (ARMINES), Alina Glushkova (ARMINES), Marius Cotescu (ACAPELA)

Editor Athanasios Manitsaris (UOM)

EC Project Officer Marcel Watelet

Abstract The purpose of the final evaluation report is to access the technical performance of the i-Treasures systems and its modules in laboratory conditions and to evaluate the performance of the i-Treasures platform and its components, and the use-cases in real conditions.

Keywords Technical performance assessment, Web-Platform performance evaluation, Use-case evaluation, Priority for components improvement



Signatures

Written by Responsibility- Company Date

Anastasios Katos (UOM), Athanasios Manitsaris (UOM)

Responsible of D7.3 (UOM) 11/5/2017

Verified by

Athanasios Manitsaris (UOM)

Responsible of D7.3 (UOM), Task T7.3 Leader

23/5/2017

Leontios Hadjileontiadis

WP7 leader (AUTH) 23/5/2017

Approved by

Nikos Grammaliis

Coordinator (CERTH) 30/5/2017

Kosmas Dimitropoulos

Quality Manager (CERTH) 30/5/2017



Table of Contents 1. Executive summary ..................................................................................... 7

2. Introduction ................................................................................................. 8

3. Methodology ............................................................................................... 9

4. Laboratory Testing of Modules .................................................................. 12

4.1 Introduction ............................................................................................... 12

4.2 iTreasuresEvaluator tool ........................................................................... 12

4.3 Laboratory testing results .......................................................................... 15

5. Technical Assessment of the system ........................................................ 21

5.1 Methodology ............................................................................................. 21

5.2 Overall Assessment of the system ............................................................ 21

5.2.1 Introduction ........................................................................................ 21

5.2.2 Collecting the data ............................................................................. 21

5.2.3 Examining the data ............................................................................ 22

5.2.4 Evaluation of Results ......................................................................... 22

5.3 Assessment of the Main Technical Indices ................................................ 23

5.3.1 Introduction ........................................................................................ 23

5.3.2 Collecting the data ............................................................................. 24

5.3.3 Examining the data ............................................................................ 25


6. Evaluation of the Web Platform's performance .......................................... 27

6.1 Introduction ............................................................................................... 27

6.2 Methodology ............................................................................................. 27

6.3 Results ...................................................................................................... 28

6.4 Evaluation of Results ................................................................................ 31

7. Use Cases Evaluation ............................................................................... 33

7.1 Contemporary Music Composition use case evaluation ............................ 33

7.1.1 Introduction ........................................................................................ 33

7.1.2 Methodology ...................................................................................... 33

7.1.3 Results ............................................................................................... 34

7.2 Tsamiko dance use case evaluation.......................................................... 36

7.2.1 Introduction ........................................................................................ 36

7.2.2 Methodology ...................................................................................... 37

7.2.3 Results ............................................................................................... 38

7.2.4 Evaluation of results ........................................................................... 39

7.3 Pottery use case evaluation ...................................................................... 40

7.3.1 Introduction ........................................................................................ 40



7.3.2 Methodology ...................................................................................... 41

7.3.3 Results ............................................................................................... 42


7.4 Salsa dance use case evaluation .............................................................. 44

7.4.1 Introduction ........................................................................................ 44

7.4.2 Methodology ...................................................................................... 45

7.4.3 Results ............................................................................................... 46


7.5 Walloon dance use case evaluation .......................................................... 48

7.5.1 Introduction ........................................................................................ 48

7.5.2 Methodology ...................................................................................... 49

7.5.3 Results ............................................................................................... 50


7.6 Human Beat Box use case evaluation ....................................................... 53

7.6.1 Introduction ........................................................................................ 53

7.6.2 Methodology ...................................................................................... 53

7.6.3 Results ............................................................................................... 54


7.7 Byzantine use case evaluation .................................................................. 57

7.7.1 Introduction ........................................................................................ 57

7.7.2 Methodology ...................................................................................... 57

7.7.3 Results ............................................................................................... 59


7.8 Canto a Tenore case evaluation ................................................................ 61

7.8.1 Introduction ........................................................................................ 61

7.8.2 Methodology ...................................................................................... 62

7.8.3 Results ............................................................................................... 63


7.9 Cantu in Paghjella use case ...................................................................... 67

7.9.1 Introduction ........................................................................................ 67

7.9.2 Methodology ...................................................................................... 67

7.9.3 Results ............................................................................................... 68


7.10 Generic Game Framework case evaluation............................................... 71

7.10.1 Introduction ........................................................................................ 71

7.10.2 Methodology ...................................................................................... 72

7.10.3 Results ............................................................................................... 73




8. Overall conclusions and proposals for the future ....................................... 77

8.1 Conclusions with respect to laboratory testing ........................................... 77

8.2 Findings with respect to the Technical Evaluation ..................................... 77

8.2.1 Overall Assessment of the System ..................................................... 77

8.2.2 Assessment of the Main Technical Indices ......................................... 78

8.3 Findings with respect to the Web-Platform evaluation ............................... 78

8.4 Findings with respect to the Use-Cases evaluation ................................... 79

8.4.1 Contemporary music composition use case ....................................... 79

8.4.2 Tsamiko dance use case.................................................................... 80

8.4.3 Pottery use case ................................................................................ 81

8.4.4 Salsa dance use case ........................................................................ 81

8.4.5 Walloon use case ............................................................................... 82

8.4.6 Human Beat-Box use case ................................................................. 83

8.4.7 Byzantine Music use case .................................................................. 83

8.4.8 Canto a Tenore use case .................................................................. 84

8.4.9 Cantu in Paghjella .............................................................................. 85

8.4.10 Generic Game use case..................................................................... 86

8.5 Overall Performance of i-Treasures .......................................................... 86

8.6 Limitations and further research ................................................................ 88

8.7 Conclusions .............................................................................................. 88

References .............................................................................................................. 91



1. Executive summary

The purpose of the final evaluation report is twofold. First, to assess the technical performance of the final version of i-Treasures systems and its modules in laboratory conditions through a set of trials and tests. In this last report, the Technical Assessment of the system is also enriched with a survey based on two structured questionnaires, one for the non-functional and one for the functional requirements. Second to evaluate the performance of the i-Treasures platform and its components in real conditions with real users, both experts and learners through the organization of trials and tests for all four ICH use cases. These activities took place during special demonstration events, running in Greece, France, Italy and Belgium (described in detail in the deliverable D6.2 “Demonstration report”).

For assessing the technical performance of the i-Treasures systems the fisEval tool is used as well as in the first report (D7.2), to fuse the various independent module assessment indices engaged and derive an overall performance index for each module, which characterizes the quality of the module.

For evaluating the overall technical performance, the performance of the i-Treasures platform and its components and the ICH use cases, the same methodology with the one applied in D.7.2 (structural equation modeling) is used. The analysis of the respondents’ results to questionnaires was based on descriptive statistics and confirmatory factor analysis. The results indicate the priority for changes/amendments that should be imposed to the components of the i-Treasures platform and the use cases in order to improve their performance.

Overall, the results with respect to the laboratory testing of modules, the technical performance, the web-platform and use cases evaluation were promising. The overall performance indices acquired during the second phase of laboratory testing exceeded those of the first phase, and all modalities surpassed the baseline criteria. Comparing the overall web-platform performance score with the overall score of last year’s evaluation, we see that the overall evaluation of the system has been increased by 7.6 percent. This notable increase was due to the fact that the priorities for improvement of components proposed in last year’s evaluation were followed during the second development phase.

Finally, it is concluded that the i-Treasures system is perceived by individuals to be of high quality (score 4.15 in Table 37) in terms of Web-Preservation & Transmission of ICH, thus contributing at an empirical level.



2. Introduction WP7 “Technical Assessment and Evaluation” deals with the evaluation of the proposed system and its functionalities in terms of complying with the user requirements and expectations. More specifically, the objectives of WP7 for the last reporting period are as follows:

Assessment of the technical performance of the i-Treasures system and its modules during the second development cycle. The assessment is performed in laboratory conditions through a set of trials and tests of the system and modules functionalities.

Evaluation of the performance of the final version of the i-Treasures platform and its components in real conditions with real users. Organization of trials/tests for all four ICH use cases.

Specifically, the last phase of Task 7.2 includes the assessment of the technical performance of the final version of i-Treasures system and its modules. The assessment is performed in laboratory conditions through a set of trials and tests of the system and modules’ functionalities, based on the assessment plan presented in deliverable D7.1 using the FisEva tool, already developed in the first phase (D7.2).

The laboratory testing of technical performance is critical in order for an optimized implementation of the technological modalities, such as sensors, processing algorithms and interfaces, to be achieved. The modules assessed are: a) Body & Gesture Recognition (for Tsamiko, Salsa and Walloon dances), b) Gesture Recognition for Contemporary Music Composition use case, c) Electroencephalography (EEG) Analysis, d) Body & Finger Recognition for Pottery use case, e) Text-to-Song, f) Vocal Tract Sensing & modeling, g) Facial expression analysis, and g) Multimodal Data Fusion and Semantic Analysis. In this vein, various technical performance categories and indices are used specifically for each module due to the great variance of their characteristics. Detailed description of the assessment indices can be found in the deliverable D7.1 "Assessment Plan". Furthermore, for each assessment index a benchmark value is introduced that acts as a reference level in order to assure the quality of the modules.

Within the last phase of Task 7.3, evaluation of the performance of the final version of the i-Treasures platform and its components in real conditions with real users, both experts and learners, took place. Tests were realized for selected sub-use cases of the four use cases: Salsa dance, Tsamiko dance, Walloon dance, Cantu in Paghjella, Canto a Tenore, Human Beat Box singing, Byzantine music, Contemporary Music Composition and Pottery. Towards this direction, the innovative evaluation methodology of structural equation modeling is used in order to trace the “good” and “poor” parts of the system. Additionally, the contribution of each component of the system towards the performance of other components, and, in turn, towards to the overall performance of the system is found.



3. Methodology

In evaluating the quality of the Web Platform and the Use-Cases the Structural Equations Model (SEM) methodology, or the Latent Variables Model, is used [8]. The methodology is described in detail in the First Evaluation Report (D7.2).

The instruments for collecting data were developed as structured questionnaires, based on the operational models proposed by experts and the people involved in each case. The survey instruments for the Technical Assessment of the System are the following structured questionnaires:

1. The questionnaire for the overall Assessment of the system. (Appendix XIII)

2. The questionnaire for the Assessment of the Main Technical Indices

(Appendix XIV)

The survey instruments for Web Platform and use cases evaluation are the following questionnaires:

1. Questionnaire for the evaluation of the Web Platform and Contemporary

Music Composition use case (APPENDIX III)

2. Questionnaire for the evaluation of the Web Platform and the Tsamiko Dance

use case (APPENDIX IV)

3. Questionnaire for the evaluation of the Web Platform and the Pottery use

case (APPENDIX V)

4. Questionnaire for the evaluation of the Web Platform and the Salsa Dance

use case (APPENDIX VI)

5. Questionnaire for the evaluation of the Web Platform and the Byzantine music

use case (APPENDIX VII)

6. Questionnaire for the evaluation of the Web Platform and the Byzantine music

use case

7. Questionnaire for the evaluation of the Web Platform and the Human Beat-

Box use case (APPENDIX VIII)

8. Questionnaire for the evaluation of the Web Platform and Cantu in Paghjella

use case (APPENDIX IX)

9. Questionnaire for the evaluation of the Web Platform and the Canto a Tenore

use case (APPENDIX X)

10. Questionnaire for the evaluation of Walloon Game (APPENDIX XI)

11. Questionnaire for for the evaluation of the ITGD tool and the Generic Game

Framework (APPENDIX XII)

Each questionnaire consists of two substantial parts, plus a part with respect to the demographic characteristics of the respondent (Section 1). In all questionnaires, Part A (Section 2-6, 8-9, see Appendix II) refers to the Web Platform evaluation. The aim of this part is to collect as many as possible data from respondents having different orientation in terms of country and interests (e.g., some people are more interested in contemporary music, but some others are more interested in pottery). Part B (Section 1,7) is concentrated on the specific use case that is evaluated (e.g., Contemporary music composition, Tsamiko dance, Pottery, Salsa dance, Byzantine music, Human beat-box, Cantu in Paghjella, Canto a Tenore and Walloon dance). All questionnaires are available in Appendices II-XIV)

All questionnaires were prepared in English and translated by the use/sub-use case leaders into the mother tongue of the respondents. As a pilot study, the translated questionnaires were examined for their proper and correct translation by experts and



other users knowing both languages (i.e., English and the mother tongue of the respondents), for examining the understandability of the questions included. Finally, the translated questionnaires were transformed into e-questionnaires. The link used for e-questionnaires is:

http://i-treasures.multimedia.uom.gr/survey2/index.php/admin/

Additionally, two evaluation protocols (i.e., one for the technical assessment of the system and another for the web platform and use cases evaluation) have been sent to all partners, indicating specific steps and tips for all relevant activities (e.g., preparation, evaluation workshops, questionnaires data processes). These protocols are also included in Appendix I.

All questionnaires and protocols were made available via a dedicated page1 in the i-Treaures platform (Figure 1).

Figure 1: Evaluation Questionnaires in i-Treasures page

Before using the collected data in examining the performance of the system, the consistency of the survey instrument was investigated [1]. The relevant methodology is described in detail in the First Evaluation Report (D7.2). Additionally, in this report, we present the methodology for assessing the overall fitting of the operational

1 http://i-treasures.multimedia.uom.gr/drupalprivate/node/200

http://i-treasures.multimedia.uom.gr/survey2/index.php/admin/

http://i-treasures.multimedia.uom.gr/drupalprivate/node/200



models. This methodology follows Bollen’s [2] recommendation to examine multiple indices, since it is possible for a model to be adequate on one fit index but inadequate on many others [9]. These indices are presented in the aforementioned First Evaluation Report (D7.2).



4. Laboratory Testing of Modules

4.1 Introduction

This Section presents the final results of the technical assessment of the i-Treasures modules in laboratory conditions, within the context of Task 7.2 – Laboratory Testing of Modules (M13-M36). In particular, the assessment results of the following modules are described:

Body & Gesture Recognition (regarding Tsamiko sub use-case)

Body & Gesture Recognition (regarding Salsa sub use-case)

Body Recognition (regarding Walloon sub use-case)

Gesture Recognition (regarding Contemporary Music Composition use-case)

EEG Analysis (regarding Contemporary Music Composition use-case)

Body & Finger Analysis (regarding Pottery use-case)

Text-to-Song (regarding Singing use-case)

Vocal Tract Sensing and Modeling (regarding Singing use-case)

Facial Expression Analysis (regarding Singing use-case)

Multimodal Data Fusion & Semantic Analysis (regarding Tsamiko and Salsa sub use-cases).

The performance index (PI), reflecting the overall quality of each module was estimated using a fuzzy inference system (FIS)-based approach, as explained in D7.1. Towards this direction, the fisEval tool was developed to serve as a cascading FIS structure software. In particular, the core functionality of the tool allows the user to create FIS structures that include FISs of different levels in a cascading order, i.e., outputs of low level FISs serving as inputs to higher level FISs. The fisEval tool engaged at the first phase of Task 7.2 was also used during the second phase and the same FIS structures and technical performance categories/indices were employed. More details about fisEval tool can be found in deliverable D7.2 2 “First Evaluation Report”.

A new tool, namely iTreasuresEvaluator, was developed, in order to facilitate the presentation of the assessment and evaluation results, not only of the aforementioned modules, but also of the other entities of the platform at any structural depth (i.e., modules, sub use-cases, use cases etc.). More details about the iTreasuresEvaluator can be found in Section 4.2 of this document.

4.2 iTreasuresEvaluator tool

Apart from the fisEval tool, a new MATLAB tool named iTreasuresEvaluator was developed, in order to facilitate the presentation of the assessment and evaluation results of the i-Treasures platform at any structural depth. More specifically, iTreasuresEvaluator permits the user to retrieve the evaluation results of individual modules, sub-use cases, use cases, the entire platform and other entities, i.e., data fusion and semantic analysis, Web-platform, and Generic game.

The functionality of the iTreasuresEvaluator is based on the existence of FIS structures created with fisEval tool. The important feature of iTreasuresEvaluator is that it does not simply present static assessment results, calculated by fisEval for



each module, in a collective way; instead, every time it is executed, the outputted assessment results are calculated based on the current input values of the modules. The reason for such functionality is to facilitate the user to instantly obtain the updated evaluation results when changing one or more input values (values for the assessment indices). Thus, what is actually needed for the iTreasuresEvaluator in order to be employed are the FIS structures and a set of input values for each module. It should be highlighted that the aforementioned functionality concerns only the laboratory assessment results of the standalone modules, namely, Body & Gesture Recognition (regarding Tsamiko sub use-case), Body & Gesture Recognition (regarding Salsa sub use-case), Body Recognition (regarding Walloon sub use-case), Gesture Recognition (regarding Contemporary Music Composition use-case), EEG Analysis, Body & Finger Analysis (regarding Pottery use-case), Text-to-Song, Vocal Tract Sensing and Modeling, Facial Expression Analysis, and Multimodal Data Fusion & Semantic Analysis, as well as the technical assessment of the sub-use cases. On the contrary, the evaluation outcome of the usability of sub-use cases and use cases, the web platform and the entire system are difficult to be calculated on the spot, due to the complex procedure required. Consequently, these results are just presented to the user. It should be noted that the results of the use-cases are derived by averaging the evaluation results of the corresponding sub-use cases since the entire evaluation procedure is sub-use case wise.

Another interesting characteristic of iTreasuresEvaluator is its robust design and implementation that permits to: i) easily introduce or delete new or existing modules, sub-use cases and use cases, and ii) easily adjust or change the modules engaged in each sub-use case. For more convenient inspection of the output, the assessment values are depicted in a barplot. The results include, apart from the final output, the output of the intermediate FISes, in the case of module evaluation, and the output of relevant modules in the case of sub-use case. Figure 2 presents some screenshots of the iTreasuresEvaluator tool showing how the user can select between the different types of assessment. Figure 3 and Figure 4 present an example of the technical assessment results of the EEG Analysis module and the Canto a Tenore sub-use case (considering that for Canto a Tenore sub-use case, the Vocal Tract sensing & Modeling and the Text-To-Song modules are engaged), respectively.



Figure 2: iTreasuresEvaluator tool exemplary screenshots showing how the user can select the assessment results to be presented: (a) Selection between modules, sub-use cases, use-cases, i-Treasures platform and other entities, (b) selection of the module of inter

Figure 3: An example of using iTreasuresEvaluator for presenting the technical assessment results of EEG Analysis module.



Figure 4: An example of using iTreasuresEvaluator for presenting the technical assessment results of Canto a Tenore sub-use case.

4.3 Laboratory testing results

This section presents an overview of the major results of the laboratory testing of the modules above. More detailed description about the tests performed, variances, the values of the assessment indices etc. can be found in the technical assessment report of each module in Appendix XV.

Module Body & Gesture Recognition (regarding Tsamiko sub use-case)

Test Leader CERTH

Major results

The performance of this module is satisfactory. The step features error rates are less than 4% while the step segmentation error rate does not exceed 5.5%. A major advantage of this module, apart from the high detection rate, is that it employs a fast, real-time approach. Moreover, it offers high granularity detection (at step level) and the step recognition is unsupervised. On the contrary, the major limitation of this module is that it is specifically tailored for the specific dance (Tsamiko).

Module Body & Gesture Recognition (regarding Salsa sub use-case)

Test Leader CERTH

Major results

The accuracy of this module is just above 70%, which denotes a rather effective behaviour considering the 20-class problem (fourteen times higher accuracy than the chance level that is 5%). The main advantages of this module are the real-time performance and scalability. Additionally, this approach requires just a simple annotation procedure for the training to take place. On the other hand, a major drawback of this module is that it depends on the precision of temporal annotation, something difficult to achieve. Moreover, the step class is quite coarse



information.

Module Body Recognition (regarding Walloon sub use-case)

Test Leader UMONS

Major results

The performance if this module is very satisfactory since it is able to recognise Walloon dance steps (captured by Kinect 2 sensor) with almost 85% accuracy. Additionally, its most promising aspect is its ability to recognize gestures captured by different motion capturing systems that the one used for training. However, the performance of this module is highly sensitive to noise. Another limitation is the need for preliminary information on the type of the recorded gestures for the feature selection procedure.

Module Gesture Recognition (regarding Contemporary Music Composition use-case)

Test Leader UOM

Major results

This module can efficiently and effectively predict body and finger movements with high precision and recall metrics that exceed 95%, supporting real-time applications and leading to fluid and immediate gesture sonification. Another strength of this module is the implementation of the confidence bounds that enable the recognition of expressive variations that might take place between the learner’s and expert’s performance. However, the computation of the confidence bounds requires an offline pre-processing step before the real-time and online recognition takes place. Moreover, distinct gestures have to be selected each time in order to be recognized by the system.

Module EEG Analysis

Test Leader AUTH

Major results

This module can predict four general affective states (High Valence/High Arousal; Low Valence/High Aousal; Low Valence/Low Arousal; and High Valence/Low Arousal) with 74% classification accuracy. Although no discrete emotions recognition takes place and the prediction accuracy is not ideal, the performance can be considered satisfactory given the highly complex problem. The major advantage of this module is that its functionality is subject-independent and supports near real-time applications. Moreover, it is based on a user-friendly recording device. On the contrary, the classification results are susceptible to EEG contamination by noise, such as movement artefacts and/or blinking.



Module Body & Finger Analysis (regarding Pottery use-case)

Test Leader ARMINES

Major results

This module can detect with high accuracy (more that 90%) the body of the potter and the wheel. On the contrary, the thumbs and the fingers can be identified with less that 62% accuracy due to the fact that these body parts are most of the time occluded by the clay, or they are self occluded. The major strength of this module, apart from the support of real-time applications, is that it provides accurate enough identification performance for all the segmented regions of interest in a complicated scene, as the one of pottery. On the contrary, the major drawbacks are the heave pre-processing procedure (filtering and RDFs training), the occlusion dependent results and the instability issues due to the noise of the camera.

Module Text-To-Song

Test Leader ACAPELLA

Major results

This module has effectively and efficiently covered the user requirements regarding supported inputs, musical aspects and computational time. The main advantages of the signing voice synthesis module are: i) the ability to explore new melodies and lyrics in the studied styles, ii) natural expressiveness through pitch unit-selection, iii) easy extension to new languages and singing styles through the open API, and iv) ability to read and parse Byzantine scores. However, there are also weaknesses relating to the limited naturalness for Canto a Tenore (mainly due to inexact phonetics), the fact that the quality and expressiveness are limited by the size of the corpus and the lack of variation in Canto a Tenore melodic lines.

Module Vocal Tract Sensing and Modeling

Test Leader UPMC

Major results

The performance of the tongue contour extraction module is reasonably satisfactory since the extracted contours are, in general, very close to the manually extracted contours, although the score (mean sum of distances) is lower than expected due to outliers. The main advantage of the implemented approach is that no training phase is required and is efficiently speaker independent. However, the computational time is quite high (almost 1 sec.), but it can be significantly reduced in case of robustness reduction. It should also be highlighted that UPMC has evaluated alternative contour extraction algorithms and the comparison results have shown that the proposed approach is the most efficient to drive the 3D model of the Hman Beat Box game.



Module Facial Expression Analysis

Test Leader CERTH

Major results

The performance of the facial feature tracking algorithm is satisfactory since the mean localization error is just 2.6 mm. The correct classification of facial action units reaches 85%, while the false positive rate is just 12%. The increased facial feature tracking accuracy, especially in the presence of open mouth expressions, is attributed to the use of local detectors. Moreover, the recognition of new action units does not require additional training thanks to the adoption of a rule-based approach for action unit detection based on a set of descriptors/facial measurements. On the other hand, the local detectors and gradients require increased computational effort and are highly affected by noisy depth data.

Module Multimodal Data Fusion & Semantic Analysis

Test Leader CERTH

Major results

This module exhibits efficient and effective performance in semantic analysis considering that all classification metrics have values that exceed 95%. More specifically, this module: i) can handle any number of detected events (such as dance steps and beats), ii) is highly efficient at recognising dance styles and, in particular, choreographies and iii) can efficiently assess the synchronization between the dancer steps and beats. As a result, there is efficient discrimination between experts and amateurs. Moreover, the generative models engaged are sufficient to learn from a limited amount of training data, in contrast with discriminative models (e.g., SVMs). However, there are also some limitations. This approach depends heavily on the accuracy of the medium level features given as input and on the timing precision of the detected events that given as input with their attributes to the MEBN model. Last but not least, it cannot be applied in real-time for very large scale problems.

Figure 5 depicts the final normalised (to [0, 1]) PIs of the above modules, along with the PIs of the first assessment round and the baseline PIs, as calculated by the FIS structures (using the fisEval tool) created by each test leader. It should be noted that the normalised PIs are depicted since each FIS structure has a different output range, depending on the membership functions and rule base settings. It is clear that the performance of all modules surpasses the one of corresponding baseline systems. It should also be noted that there is a certain improvement to the technical aspects of the modules compared to first round of assessment, corresponding to the first development cycle. To this end, among the most effective modules are Body & Gesture Recognition (Tsamiko), Gesture Recognition (CMC), Text-to-Song, Facial Expression Analyisis and Multimodal Data Fusion and Semantic Analysis, while ample room for improvement remains for the Body & Finger Recognition (Pottery) and Vocal Tract Sensing & Modeling modules.



According to the assessment plan, apart from the PIs for each module, technical general performance indices (tGPI) have been estimated for each sub use case (i.e., Tsamiko, Salsa, Walloon, Calus, Byzantine, Canto a Tenore, Cantu in Paghjella, Human Beat Box, Contemporary Music Composition-CMC and Pottery) via dedicated FIS, taking into account the modules that are utilised in each sub use case. The normalised tGPIs for all sub use cases (for both assessment rounds), along with the corresponding baseline tGPIs are presented in Figure 6. The performances of all sub use cases are rather solid and surpass the corresponding baseline indices by 13% at least. By comparing the two assessment phases it can be noted that the second development cycle played a crucial role towards the amelioration of the technical performance of the sub use cases. To this end, Pottery is the least efficient sub use case, which was expected since the clay renders the finger detection procedure rather difficult. On the contrary, the most successful sub use cases include Contemporary Music Composition and Byzantine music. Furthermore, although Walloon dance and Calus dance exhibit high tGPIsuc, it should be noted that they entail only one module (i.e., body recognition) each, thus, their tGPIsuc is identical to the PIs of the corresponding body recognition modules.

Based on the sub use cases results, tGPIs for the overall use cases (tGPIuc), i.e., Dancing, Singing, CMC and Pottery, have been calculated via averaging (Figure 7). It should be noted that weighted averaging has been utilised for the Dancing use case in order to obtain more solid results since Tsamiko and Salsa dances engage more modules than Walloon and Calus dances. Moreover, for the Pottery and CMC use cases that contain only one sub use case each, the tGPIuc and tGPIsuc are identical. From Figure 7 it can be deducted that all use cases, even the most complicated ones (i.e., Dancing and Singing), exhibit an effective behaviour from a technological point of view.

At last, taking one step further the laboratory technical assessment, an overall tGPI (tGPIo) for the entire i-Treaures system has been calculated via weighted averaging the tGPIuc of the four use cases. In this direction, the tGPIo is 0.8 while the baseline index is 0.58. Thus, it can be inferred that overall developed system exhibits solid technical performance. However, there are still improvement opportunities for the future.

Figure 5: Normalised PIs (2nd assessment round, 1st assessment round and baseline) as calculated by the FIS structures.



Figure 6: Normalised technical general performance indices for all sub use cases (tGPIsuc) for the two rounds of assessment along with the corresponding baseline tGPIsuc.

Figure 7: Normalised technical general performance indices for all use cases (tGPIuc) for the two rounds of assessment along with the corresponding baseline tGPIuc.

To conclude, the above technical assessment results denote that the modalities of the i-Treasures platform, considered as standalone working units, are of high quality, from a technological point of view, exhibiting rather satisfactory performance and behaviour and covering the majority of user requirements and system specifications. Although they have received effective improvements over the course of development phases, there are certain limitations that should be addressed in the future. However, these limitations do not hinder the overall functionality and quality of the i-Treasures platform.



5. Technical Assessment of the system

5.1 Methodology

The methodology followed in the technical assessment of the system depends on the information collected. This information is distinguished between two parts: the overall assessment of the system, which is based on non-functional requirements, and the assessment of the main technical indices, which is based on functional requirements. In both cases, descriptive statistics are used and in tracing the contribution of each concept on the assessment findings confirmatory factor analysis (CFA2) are employed. For evaluating the results the Normed Chi-squared and the RMR indices are used.

5.2 Overall Assessment of the system

5.2.1 Introduction

For the overall assessment the system the non-functional requirements presented in [5] were used, supporting, thus, the content validity of the items used in the analysis. The general titles of these requirements are shown in Table 1. The specific items used under each title are presented in Appendix XIII.

Table 1: Characteristics of non-functional requirements (N=24)

Section Construct Mean Standard deviation

Items Cronbach’s Alpha

1 Cost optimality / Energy efficiency

3.38 0.67 5 0.677

2 Accessibility / Usability 3.63 0.80 12 0.922 3 Documentation / Support 3.92 0.66 7 0.565 4 Interoperability / Portability 3.74 0.61 4 0.612 5 Extensibility / Scalability 3.68 0.69 4 0.842 6 Auditing 3.56 1.08 3 0.839 7 Security / Privacy 4.05 0.64 5 0.708 8 Fault tolerance / Recoverability 3.58 0.97 6 0.854 9 Licensing / Copyright 4.16 0.87 5 0.814

OVERALL 9 0.909

5.2.2 Collecting the data

According to the concepts presented in [5], structured questionnaire was developed (see Appendix XIII), which was distributed to and administered by the case studies leaders. The demographics of the respondents to the questionnaire are presented in Table 2.

2 https://en.wikipedia.org/wiki/Confirmatory_factor_analysis



Table 2: Overall Assessment of the System - Demographics (N = 24)

Number Percentage (%)

Gender: Male Female

17 7

70.8 29.2

Age: Up to 30 More than 30

10 14

41.7 58.3


Education: College University Post Grad

1 6 17

4.2 25.0 70.8

Perceived familiarity in using computers: Good Very Good

1 23

4.2 95.8

Respondent’s ID: Expert Member of Focus Group

13 11

54.2 45.8

5.2.3 Examining the data

Before any estimation we examined the internal consistency of the items (specific questions) which constituted the general constructs (titles) of the assessment, using Cronbach’s Alpha [4]. Internal consistency measures the extent to which respondents rate items of a construct in a similar manner. The results of this examination are shown in Table 1. Most Alphas are greater than 0.70 indicating constructs internal consistency. However, for three constructs (Cost optimality / Energy efficiency, Documentation / Support, and Interoperability / Portability) the Alphas were less than 0.70, but because they were greater than 0.50, they were still included in the analysis.

5.2.4 Evaluation of Results

Considering that the sample of respondents used in the analysis was rather small (N = 24), but taking also into consideration that the validity of the general constructs was acceptable, we aggregated all items in each construct. The descriptive statistics (means and standard deviations) of these constructs are shown in Table 1. It is seen that the means of all constructs are above “level 3”, indicating that the system has been assessed to be above average. However, in ranking its components according to their means, we see that Cost optimality / Energy efficiency (3.38) and Auditing (3.56) are at the lowest levels and Security / Privacy (4.05) and Licensing / Copyright (4.16) are the highest levels.

Figure 6 presents the results of CFA. In general, these results are acceptable because the Normed Chi-squared (2.027) [with critical level not more than 3, 5, and 7 to indicate excellent, good, and mediocre fit, respectively] and RMR (0.047) [with critical level not more than 0.05, 0.08, and 0.10 to indicate excellent, good, and mediocre fit, respectively] indices are significantly below their critical levels, respectively [2]. The standardized coefficients assigned to each arrow are all significant. According to these coefficients Accessibility / Usability (0.92) and Auditing



(0.88) have the highest contribution in the overall assessment of the system, whilst Interoperability / Portability (0.59) and Licensing / Copyright (0.52) have the lowest.

Taking into consideration the “component ranking rule” (i.e., the lower the ratio of the mean of a component by its loading is, the higher the priority for changes / amendments for this component), we found the following ratios: Cost optimality / Energy efficiency (5.12), Accessibility / Usability (3.95), Documentation / Support (4.96), Interoperability / Portability (6.34), Extensibility / Scalability (4.54), Auditing (4.05), Security / Privacy (5.00), Fault tolerance / Recoverability (5.11), Licensing / Copyright (8.00). In other words, future changes / amendments of the system should give priority to Accessibility / Usability and Auditing.

Overall Assessment of

the System

based on

Non-Functional

Requirements

Cost optimality /

Energy efficiency

Accessibility /

Usability

Documentation /

Support

Interoperability /

Portability

Extensibility /

Scalability

Auditing

Security / Privacy

Fault tolerance /

Recoverability

Licensing /

Copyright

0.66

0.92

0.79

0.59

0.81

0.88

0.81

0.70

0.52

Figure 8: Overall assessment of the system based on non-functional requirements

5.3 Assessment of the Main Technical Indices

5.3.1 Introduction

For the assessment of the main technical indices we used the functional requirements presented in [5], supporting, thus, the content validity of the items used in the analysis. The general titles of these requirements are shown in Table 3. The specific items used under each title are presented in Appendix XIV.



Table 3: Characteristics of functional requirements

Section Construct Mean Standard deviation

N Items Cronbach’s Alpha

1 Facial expression analysis 3.91 0.73 18 3 0.939 2 Human body motion and

gesture recognition 4.01 0.60 22 4 0.874

3 Electroencephalography analysis

4.15 0.69 9 3 0.935

4 Vocal tract sensing and modeling – Ultrasound analysis

3.79 0.91 14 2 0.862

5 Sound processing 3.89 0.62 10 7 0.927 6 Text-to-song 3.95 0.70 19 3 0.782 7 Ontology engineering 3.89 0.73 16 12 0.960 8 Educational platform 3.52 0.87 26 5 0.891

5.3.2 Collecting the data

According to the concepts presented in [5], we developed a structured questionnaire (see Appendix XIV) which was distributed to and administered by the case studies leaders. The demographics of the respondents to the questionnaire are presented in Table 4.

Table 4: Main Technical Indices - Demographics (percentages)

Facial expression analysis

(%)

Human body motion and

gesture recognition

(%)

Electroencephalography analysis

(%)

Vocal tract sensing

and modeling

– Ultrasound analysis

(%)

Sound processin

g

(%)

Text-to-song

(%)

Ontology

engineering

(%)

Educational platform

(%)

Gender:

Male

Female

72.2

27.8

77.3

23.7

77.8

22.2

78.6

21.4

70.0

30.0

68.4

31.6

68.8

31.3

73.1

26.9

Age:

Up to 30

More than 30

38.9

61.1

50.0

50.0

66.7

33.3

57.1

42.9

50.0

50.0

52.6

47.4

56.3

43.7

38.5

61.5

Education:

University

Post Grad

83.3

16.7

77.3

23.7

77.8

22.2

71.4

28.6

80.0

20.0

73.7

26.3

75.0

25.0

69.2

26.8

Perceived familiarity in

using computers:

Good

Very Good

100.0

4.5

95.5

100.0

7.1

92.9

100.0

5.3

94.7

100.0

7.7

92.3

Respondent’s ID:

Expert

Member of Focus Group

55.6

44.4

45.5

54.5

55.6

44.4

50.0

50.0

60.0

40.0

47.4

52.6

50.0

50.0

53.8

46.2



5.3.3 Examining the data

Table 3 reports the Cronbach’s Alphas for examining the internal consistency of the items included in the general constructs reflecting each technical index. All Alphas are much greater than 0.70 indicating constructs internal consistency.


We must note here that in contrast to the overall assessment of the system where all respondents involved answered the same questions, in the assessment of the main technical indices, the respondents involved focused on specific indices only. As a result the sizes (N) of the samples were different between indices, and they are reported in Table 3. Considering that these sample sizes were small and taking also into consideration that the validity of the general constructs was acceptable, we aggregated all items in each construct. The descriptive statistics (means and standard deviations) of these constructs are shown in Table 3. It is seen that the means of all constructs are above “level 3”, indicating that the main technical indices have been assessed to be above average. However, in ranking these indices according to their means, we see that Educational platform (3.52) and Vocal tract sensing and modeling – Ultrasound analysis (3.79) are at the lowest levels and Human body motion and gesture recognition (4.01) and Electroencephalography analysis (4.15) are the highest levels. Figure 7 presents the results of CFA. We must also note here that in contrast to the overall assessment of the system where we performed one CFA including all constructs, in this case we performed eight individual CFAs, one for each technical index. Then, we averaged the loadings of the items included in each CFA, producing eight averages, respectively. These averages are reported as standardized coefficients and are assigned in the arrows presented in Figure 7. We accept these results to be satisfactory, because the average values of the Normed Chi-squared (2.734) and RMR (0.054) indices are below their critical levels, respectively. According to the coefficients presented in Figure 7 Facial expression analysis (0.92) and Electroencephalography analysis (0.91) have the highest contribution in the assessment of the main indices, whilst Human body motion and gesture recognition (0.79) and Text-to-Song (0.75) have the lowest. Taking into consideration the “component ranking rule”, we found the following ratios: Facial expression analysis (4.25), Human body motion and gesture recognition (5.08), Electroencephalography analysis (4.56), Vocal tract sensing and modeling – Ultrasound analysis (4.26), Sound processing (4.80), Text-to-song (5.27), Ontology engineering (4.74) and Educational platform (4.40). In other words, future changes / amendments should give priority to Facial expression analysis and Vocal tract sensing and modeling – Ultrasound analysis.



Assessment of the Main

Technical Indices

based on

Functional Requirements

Facial Expression

Analysis

Human Body Motion

and Gesture

Recognition

Electroencephalo-

graphy Analysis

Vocal Tract Sensing

and Modelling /

Ultrasound Analysis

Sound Processing

Text-to-Song

Ontology

Engineering

Educational Platform

0.92

0.79

0.91

0.89

0.81

0.75

0.82

0.80

Figure 9: Assessment of the main technical indices based on functional requirements



6. Evaluation of the Web Platform's performance

6.1 Introduction

Figure 8 presents the operational model that is used in evaluating the web platform. Each “box” refers to a general construct, which is constituted by “dots” referring to sub-constructs. Both, constructs and sub-constructs have been proposed by experts, thus, verifying content validity. In terms of its structure the model follows the general structure used in the previous evaluation, although the current version has been augmented with almost all major components of the web platform. In particular, further to the “perceived performance” construct the model includes now the construct of “Preservation and Transmission of ICH”. The ultimate objective of the model is to be used for the evaluation of the overall web platform perceived performance, and the perceived performance with respect to the preservation and transmission of intangible cultural heritage.

6.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the Web Platform (see Appendix II). Following the evaluation protocol, various scheduled workshops have been organized by use case leaders in Greece, France, Italy and Belgium. The demographics of the respondents to the questionnaire are presented in Table 5.

Table 5: Web Platform - Demographics (N = 127)


Gender: Male Female

70 52

55.1 44.9

Age: Up to 20 21 – 30 31 – 40 41 - 50 More than 50

57 23 19 18 10

44.9 18.1 15.0 14.2 7.9

Education: Primary Secondary College University Post Grad

15 7 2 72 31

11.8 5.5 1.6 56.7 24.4

Perceived familiarity in using computers: Not much Adequate Good Very Good

8 30 53 36

6.3 23.6 41.7 28.3

Respondent’s ID: Expert Simple user

15 112

11.8 88.2



INFORMATION

Purpose

Quality

Difficulty

Organization

DESIGN

Accessibility

Navigation

Interface

Interaction

LEARNING

MANAGEMENT

SYSTEM

Design

Courses

Resources

Usability

FUNCTIONALITIES

Semantic

Text-to-Song

COMPLEXITY &

USABILITY

Complexity

Usability

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

PRESERVATION &

TRANSMISSION OF ICH

Contribute in preservation and

transmission

Easily adopted technologies

Useful tool for organizations

Recommend to colleagues

Useful for capturing and

transmitting expressions

Does not undermine

spontaneous expressions

Respects tradition

Innovative platform

CONTROLS: Gender, Age, Education, Familiarity with computers, Type of respondent

(Experts / Simple users)

Figure 10: The web platform operational model.

6.3 Results

Before we proceed to the actual estimation of the operational model through Structural Equations Modeling (SEM), via AMOS, we examined the characteristics of



the constructs involved. Table 6 presents the means, standard deviations, Cronbach’s Alphas and correlation coefficients between all constructs.

Most Alphas are greater than 0.70 indicating constructs internal consistency. However, for two constructs (Information and Complexity & Usability) the Alphas were less than 0.70, but because they were greater than 0.50, we still included in the analysis. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are between the levels of ‘3’ and ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of the Web platform is above ‘average’ but less than “good”.

Table 6: Means, standard deviations and correlation coefficients of the web platform constructs

Constructs

Means (standard

deviations)

Cronbach’s Alpha

Correlation coefficients Information Design LMS Functionalities Complexity

& Usability Perceived

performance Information 3.62

(0.48) 0.570 1

Design 3.97 (0.62)

0.821 0.607 1

LMS 3.90 (0.63)

0.859 0.351 0.606 1

Functionalities 3.66 (0.71)

0.825 0.385 0.601 0.572 1

Complexity & Usability

3.84 (0.77)

0.515 0.364 0.450 0.473 0.685 1

Perceived performance

4.00 (0.70)

0.868 0.462 0.600 0.421 0.615 0.724 1

ICH 4.15 (0.64)

0.883 0.513 0.589 0.435 0.547 0.479 0.601

Note: All correlation coefficients are significant at the 0.01 level (2-tailed)

We applied the Maximum Likelihood (ML) estimation methodology for estimating the operational model presented in Figure 8, having first examined the necessary conditions that should be fulfilled for using this estimation method. The estimation results of this operational model are presented in Figure 9.

Considering the fit indices of the web platform operational model (Chi-squared = 821.938, df = 422, p = 0.000, Normed Chi-squared = 1.948, RMR = 0.059, GFI = 0.711, NFI = 0.693, CFI = 0.819, RMSEA = 0.087) we see that many fit indices are acceptable, indicating that the estimated model presented in Figure 9 is acceptable. In fact, the estimated model in Figure 9 supports the theoretical model presented in Figure 8. Additionally, it is seen in Figure 9 that “design” and “LMS”, “functionalities”, and “complexity & usability” serially and fully mediate the relationship between “information” and “perceived performance” and “ICH”.



0.80

INFORMATION

Purpose Quality Difficulty Organization

DESIGN LMS

Access

Navigation

Interface

Interaction

Design

Course

Resources

Usability

FUNCTIONALITIESSemantic Text-to-Song

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

COMPLEXITY &

USABILITY

Complexity

Usability

ICH

Contribution

Adoption

Usefulness

Recommendation

Transmission

Spontaneous

Respects tradition

Innovative

0.310.83

0.53 0.35

0.96

0.74

-0.100.69

0.79 0.81 -0.27 0.74

0.82

0.63

0.70

0.81

0.74

0.79

0.84

0.75

0.86 0.80

0.55

0.78

0.78

0.72

0.83

0.73

0.74

0.80

0.77

0.71

0.72

0.59

0.68

0.66

Respondent

ID

-0.09

Gender

-0.17

Age0.35

Age

0.29

Age

0.08

-0.34

Figure 11: Estimation results of the Web platform operational model



6.4 Evaluation of Results

The above results highlight that:

• The model rather acceptably predicts performance (all standardized coefficients are significant and positive, and fit indices are in large acceptable).

• Design and LMS fully mediate the relationship between information and functionalities.

• The “difficulty of the informational content” has a negative influence on “performance”.

• “Quality” is the most important informational factor that predicts performance.

• “Accessibility” and “Navigation” are the most important design factors that predict performance.

• “Usability in LMS” is the most important factor of LMS that predicts performance.

• “Semantic” and “Text-to-song” functionalities are important in predicting performance.

• “Usability” in accomplishing activities is most important compared to “non-complexity” of the web platform in influencing performance.

• “Satisfaction” is the most important factor in determining perceived performance.

• The i-Treasures platform as a “useful tool” is the most important factor in determining preservation and transmission of ICH.

• Information, Perceive performance and ICH are positively influenced by “age”, meaning that older people assign higher values to these constructs. In contrast, “age” negatively influences LMS, meaning that younger people find the learning management system functionalities better designed and understandable than older people.

• “Gender” negatively influences “Design”, meaning that female find accessibility, navigation, interface design, and interaction less attractive than male.

• “Respondent ID” negatively influences “Perceived performance”, meaning that simple users feel that performance is at lower levels than it is believed by experts.

• Considering that many constructs are influenced by the demographic characteristics of the users, it is accepted that the web platform belongs to the so-called “Contingency Systems”, which support the view that the system maximizes its performance according to the specific context within which it is operating [6].

In detail, Table 7 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of DESIGN on effectiveness is 0.381. That is, due to both direct (unmediated) and indirect (mediated) effects of DESIGN on effectiveness, when DESIGN goes up by 1 standard deviation, effectiveness goes up by 0.381 standard deviations (for further discussion of direct, indirect and total effects, see [10].



Table 7: Standardized Total Effects of the Web Platform operational model

Constructs / items Information Design LMS Functionalities Complexity & Usability

Information - - - - - Design 0.822 - - - - LMS 0.799 - - - - Functionalities 0.719 0.564 0.319 - - Complexity & Usability

0.586 0.460 0.260 0.815 -

Perceived performance Innovation Satisfaction Efficiency Effectiveness

0.551

0.405 0.456 0.398 0.486

0.443

0.317 0.358 0.312 0.381

0.245

0.180 0.203 0.177 0.216

0.767

0.563 0.634 0.554 0.676

0.941

0.690 0.778 0.679 0.830

ICH Ich8

3

Ich7 Ich6 Ich5 Ich4 Ich3 Ich2 Ich1

0.410

0.270 0.279 0.243 0.295 0.314 0.328 0.291 0.303

0.322

0.212 0.219 0.190 0.231 0.246 0.258 0.229 0.238

0.182

0.120 0.124 0.108 0.131 0.139 0.146 0.129 0.135

0.570

0.376 0.388 0.338 0.410 0.437 0.457 0.405 0.421

0.700

0.461 0.476 0.414 0.502 0.536 0.560 0.497 0.517

The figures in Table 7 are very important because they can guide the designers of the web platform to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.

3 Note: Ich from 1 to 8 correspond to the specific questions used in SECTION 9:

PRESERVATION AND TRANSMISSION OF INTANGIBLE CULTURAL HERITAGE (ICH) of the

questionnaire



7. Use Cases Evaluation

7.1 Contemporary Music Composition use case evaluation

7.1.1 Introduction

Figure 10 presents the operational model that is used in evaluating the Contemporary Music Composition (CMC). The ultimate objective of the model is to be used for the evaluation of the relationship between CMC Game and the Intangible Musical Instrument (IMI), and Perceived Performance, through the mediating mechanism of Learning Experience.

CMC GAME

Satisfactory

avatars

Clear objectives

Informative getting

started

CONTROLS: Gender, Age, Education, Familiarity with computers, Music literacy, Type of

respondent (Experts / Simple users)

IMI

Simple to use

Easy to remember

LEARNING

EXPERIENCE

Enough time

Real time

recognition results

Satisfactory Final

Challenge

CMC & GAME

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

Figure 12: Operational model of the Contemporary Music Composition

7.1.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the CMC Game and IMI (see Appendix III). Following the evaluation protocol, various scheduled demos have been organized in different countries. More specifically:

at the Multimedia, Security and Networking Lab (MSN Lab) of UOM in Thessaloniki (Greece), on 16-25 January 2017 (organized by UOM and AUTH)

at Concert Hall in Thessaloniki (Greece) on July 2016, in the context of the International Symposium on Movement and Computing (MOCO16) (organized by ARMINES, UOM and AUTH)

at Griffith University in Brisbane (Australia) on 11-15 July 2016, in the context of the International Conference on New Interfaces for Musical Expression (NIME’16) (organized by UOM)

at SomeTimeStudio (creative ideas and projects incubator) in Paris (France) on 19 September 2016 (organized by ARMINES). More details on demonstrations can be found in D6.2.



The purpose of the aforementioned demonstrations/workshops was firstly to present the web platform, the CMC game as well as the Intangible Musical Instrument to the users, users then to experiment with these functionalities, and finally to collect data through the e-questionnaire. In total, 64 people took part in these demonstrations/ workshops. However, 36 questionnaires were complete to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 8.

Table 8: CMC - Demographics (N = 36)


Gender: Male Female

22 14

61.1 38.9

Age: Up to 20 21 – 30 31 – 40 41 - 50

12 13 8 3

33.3 36.1 22.2 8.3


1 21 14

2.8 58.3 38.9


1 3 13 19

2.8 8.3 36.1 52.8

Music Literacy Not at all Not much Adequate Good Very good

7 11 6 6 6

19.4 30.6 16.7 16.7 16.7


3 33

8.3 91.7

7.1.3 Results

Before we proceed to the actual estimation of the operational model through Structural Equations Modeling (SEM), we examined the characteristics of the constructs involved. Table 9 presents the means, standard deviations, Cronbach’s Alphas and correlation coefficients between all constructs.

Most Alphas are greater or close to 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are between the levels of ‘3’ and ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of CMC is above ‘average’ but less than “good”.

We must note here that in the questionnaire, 15 items (questions) constitute the “CMC game” construct, 7 items the IMI construct, and 14 items the Learning experience construct. Considering the small size of the evaluation exercise we



aggregated these items into factors, using exploratory factor analysis (EFA). For example, factor 1 in the CMC Game, constituted by items 5, 6, 7, 8, 9, 11, 12 and 15 in the questionnaire. To this factor we gave the name of “satisfactory avatars” (first dot in the CMC Game construct), due to the highest loading of item 7 in the EFA. Similarly, we found all factors for all constructs in the analysis (see operational model in Figure 10), except of course the actual dimensions for perceived performance.

Table 9: Means, standard deviations and correlation coefficients of the CMC constructs

Constructs

Means (standard

deviations)

Cronbach’s Alpha

Correlation Coefficients

CMC GAME

IMI LEARNING EXPERIENCE

CMC GAME 3.65 (0.54)

0.607 1

IMI 3.79 (0.65)

0.718 0.280* 1

LEARNING EXPERIENCE

3.69 (0.55)

0.687 0.147* 0.781**

CMC & GAME PERCEIVED PERFORMANCE

4.18 (0.55)

0.737 0.229* 0.476** 0.722**

* Correlation coefficient is significant at the 0.05 level (2-tailed)

** Correlation coefficient is significant at the 0.01 level (2-tailed)

We applied the Maximum Likelihood (ML) estimation methodology for estimating the operational model presented in Figure 10, having first examined the necessary conditions that should be fulfilled for using this estimation method. The estimation results of this operational model are presented in Figure 11.

Considering the fit indices of the CMC operational model (Chi-squared = 196.368, df = 106, p = 0.000, Normed Chi-squared = 1.853, RMR = 0.370, GFI = 0.671, NFI = 0.457, CFI = 0.600, RMSEA = 0.156) we see that some fit indices are acceptable, indicating that the estimated model presented in Figure 11 is acceptable. In fact, the estimated model in Figure 6supports the theoretical model presented in Figure 10. Additionally, it is seen in Figure 11 that “learning experience” fully mediates the relationship between “CMC Game” and “IMI”, and “perceived performance”.



0.45CMC & GAME

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satsfaction

Innovation

LEARNING

EXPERIENCE

Real time

recognition

results

Enough timeSatisfactory

Final Challenge

IMI

Simple to use

Easy to

remember

CMC GAME

Satisfactory

avatars

Clear objectives

Informative

getting started

0.73

0.54

0.53

0.85

0.68

0.68 0.580.84

0.90

0.70

0.82

0.27

0.15

0.85

0.81

Age:

CMC GAME = -0.41

IMI = 0.37

LEARNING

EXPERIENCE = 0.24

Educational level:

IMI = -0.06

CMC & GAME

PERCEIVED

PERFORMANCE = -0.14

Familiarity in using

computers:

LEARNING

EXPERIENCE = 0.10

CMC & GAME

PERCEIVED

PERFORMANCE = -0.12

Music Literacy:

CMC GAME = 0.10

IMI = -0.33

CMC & GAME

PERCEIVED

PERFORMANCE = -0.08

Respondent’s ID:

CMC GAME = -0.24

IMI = -0.10

LEARNING

EXPERIENCE = 0.16

CMC & GAME

PERCEIVED

PERFORMANCE = -0.09

Figure 13: Estimation results of the CMC operational model

7.2 Tsamiko dance use case evaluation

7.2.1 Introduction

Figure 12 presents the operational model that is used in evaluating the Tsamiko dance use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Informational characteristics (i.e., informational content for Tsamiko dance and courses with respect to Tsamiko in LMS) and Perceived Performance, through the mediating mechanism of Interactional characteristics of the Tsamiko dance game-like application.

INFORMATIONAL

CHARACTERISTICS

Content

Courses

CONTROLS: Gender, Age, Education, Familiarity with computers, Experience in dancing,

Type of respondent (Experts / Simple users)

INTERACTIONAL

CHARACTERISTICS

Game scenario

Visualization

User performance

evaluation

Sensor set-up

Usability

Learning experience

TSAMIKO DANCE

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

Figure 14: Operational model of Tsamiko Dance



7.2.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the Tsamiko use case. Following the evaluation protocol, a session for the Tsamiko dance use case has been organized at the Department of Physical Education and Sports Science (TEFAA) of the Aristotle University of Thessaloniki. This activity involved a structured educational path for university students attending the “Didactics of Greek Traditional Dances” course in this department. The students followed the learning path of the corresponding LMS course, which also included a training session with the Tsamiko dance game. Finally, they explored the functionalities of the i-Treasures Web Platform (more details can be found in D6.2).

The purpose of the aforementioned demonstrations/workshops was firstly to present the web platform, the Tsamiko dance game-like application to the users, users then to experiment with these functionalities, and finally to collect data through the e-questionnaire (see Appendix IV). In total, 30 questionnaires were complete to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 10.

Table 10: Tsamiko Dance - Demographics (N = 30)


Gender: Male Female

15 15

50.0 50.0

Age: Up to 20 21 – 30 31 – 40 41 - 50

28 1 0 1

93.3 3.3 0.0 3.3

Education: University Post Grad

27 3

90.0 10.0


4 11 14 1

13.3 36.7 46.7 3.3

Experience in dancing Not at all Not much Adequate Good Very good

7 9 8 6

23.3 30.0 26.7 20.0

Experience in dancing Greek traditional dances Not at all Not much Adequate Good Very good

8 0 12 7 3

26.7 0.0 40.0 23.3 10.0

Experience in dancing Tsamiko Not at all Not much Adequate Good

12 0 9 5

40.0 0.0 30.0 16.7



Very good 4 13.3


1 29

3.3 96.7

7.2.3 Results

Before we proceed to the actual estimation of the operational model through Structural Equations Modeling, we examined the characteristics of the constructs involved. Table 11 presents the means, standard deviations, Cronbach’s Alphas and correlation coefficients between all constructs.

All Alphas are greater than 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are greater than ‘3’ but smaller than ‘4’, indicating that in general the perception of individuals about the quality of the various characteristics of the Tsamiko dance use case is above ‘average’ but less than ‘good’.

We must note here that in the structured questionnaire although each construct is individually presented, in the operational model we treated content and courses as two related sub-constructs under the general informational construct, and we have also treated game scenario, visualization, user performance evaluation, sensor set-up, usability and learning experience as six related sub-constructs under the general interactional characteristics construct. We did this aggregation of sub-constructs into higher level constructs, due to the fact that the sample size was rather small and the Cronbach Alphas were greater than 0.70, for the items comprising each sub-construct.

Table 11: Means, standard deviations and correlation coefficients of the Tsamiko Dance constructs

Constructs Means (standard

deviations)

Cronbach’s Alpha

Correlation Coefficients INFORMATIONAL

CHARACTERISTICS INTERACTIONAL

CHARACTERISTICS INFORMATIONAL CHARACTERISTICS

3.93 (0.62)

0.843 1

INTERACTIONAL CHARACTERISTICS

3.55 (0.48)

0.855 0.703 1

TSAMIKO PERCEIVED PERFORMANCE

3.81 (0.67)

0.868 0.690 0.865


We applied the Maximum Likelihood estimation methodology for estimating the operational model presented in Figure 12, having first examined the necessary conditions that should be fulfilled for using this estimation method. The estimation results of this operational model are presented in Figure 13.

Considering the fit indices of the Tsamiko dance operational model (Chi-squared = 72.700, df = 52, p = 0.031, Normed Chi-squared = 1.398, RMR = 0.028, GFI = 0.759, NFI = 0.771, CFI = 0.918, RMSEA = 0.117) we see that many fit indices are acceptable, indicating that the estimated model presented in Figure 13 is acceptable.



In fact, the estimated model in Figure 13 supports the theoretical model presented in Figure 12. Additionally, it is seen in Figure 13 that “Interactional characteristics” fully mediates the relationship between “Informational characteristics” and “Perceived performance”.

INFORMATIONAL

CHARACTERISTICS

INTERACTIONAL

CHARACTERISTICS

TSAMIKO

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

Content

Courses

Game scenario Visualization User performance

evaluationSensor set-up Usability

Learning

experience

0.82

0.94

0.83 0.91 0.22 0.64 0.70 0.82

0.83 0.99

0.80

0.86

0.81

0.72

Figure 15: Estimation results of the operational model of Tsamiko Dance

7.2.4 Evaluation of results


• The model rather acceptably predicts performance (all standardized coefficients are significant and positive, and fit indices are in general acceptable).

• Interactional characteristics fully mediate the relationship between Informational characteristics and Perceived performance.

• “Courses with respect to Tsamiko dance in the LMS” is the most important Informational characteristics factor that predicts performance.

• “Visualization” is the most important Interactional characteristics factor that predicts performance.

• “Efficiency” is the most important factor in determining perceived performance.

• The Informational – Interactional – Perceived performance relationship is not influenced by demographics of the respondents (see Figure 8). Although we used all demographics as uni-dimensional controls in estimation the results obtained were not significant.

• Considering that all constructs were not influenced by the demographic characteristics of the users, we conclude that Tsamiko dance belongs to the so-called “Universalistic Systems”, which support the view that the system maximizes its performance irrespectively to the specific context within which it is operating [6].

In detail, Table 12 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of Interactional characteristics on efficiency is 0.851. That is, due to both direct (unmediated) and indirect (mediated) effects of



Interactional characteristics on effectiveness, when Interactional characteristics goes up by 1 standard deviation, efficiency goes up by 0.851 standard deviations (for further discussion of direct, indirect and total effects, see [10].

Table 12: Standardized Total Effects of the Tsamiko operational model

Constructs / items INFORMATIONAL CHARACTERISTICS


INFORMATIONAL CHARACTERISTICS

- -


0.828 -

TSAMIKO PERCEIVED PERFORMANCE Innovation Satisfaction Efficiency Effectiveness

0.818

0.586 0.661 0.705 0.651

0.988

0.708 0.798 0.851 0.786

The figures in Table 12 are also very important because they can guide the designers of the Tsamiko dance case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.

7.3 Pottery use case evaluation

7.3.1 Introduction

Figure 14 presents the operational model that is used in evaluating the Pottery use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Informational characteristics (i.e., informational content for Pottery and courses with respect to Pottery in LMS) and Perceived Performance, through the mediating mechanism of Interactional characteristics of the Pottery game-like application.



INFORMATIONAL

CHARACTERISTICS

Content

Courses

CONTROLS: Gender, Age, Education, Familiarity with computers, Experience in pottery, Type

of respondent (Experts / Simple users)

INTERACTIONAL

CHARACTERISTICS

Game scenario

Visualization

User performance

evaluation

Sensor set-up

Usability

Learning experience

POTTERY

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

Figure 16: Operational model of Pottery

7.3.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the Pottery use case (see Appendix V). Following the evaluation protocol, a session for the Pottery use case has been organized in a cultural centre of the municipality of Pavlos Melas, in Thessaloniki by our pottery expert. This activity involved a structured educational path for adults that have no prior knowledge of pottery but want acquire basic knowledge and pottery making skills and was realized in the context of the pottery class taught by the expert in local residents. The learners followed the learning path of the corresponding LMS course, which also included a training session with the “Pottery master” game-like application. Finally, they also explored the i-Treasures Web Platform (more details can be found in D6.2).

The purpose of the aforementioned demonstrations/workshops was firstly to present the web platform, the Pottery game-like application to the users, users then to experiment with these functionalities, and finally to collect data through the questionnaire. In total, 7 questionnaires were complete to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 13.

Table 13: Pottery - Demographics (N = 7)

Number Percentage(%)

Gender: Male Female

4 3

57.1 42.9


3 4

42.9 57.1

Education: Secondary education University Post Grad

1 5 1

14.3 71.4 14.3

Perceived familiarity in using computers: Adequate Good

4 3

57.1 42.9

Experience in pottery Adequate

1

14.3



Good 6 85.7


1 6

14.3 85.7

7.3.3 Results


All Alphas are greater than 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are higher than ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of the Pottery use case is above ‘good’.

We must note here that due to the very small sample size we treated all constructs in their aggregate version. We did this aggregation into higher level constructs, due to the fact that Cronbach Alphas were greater than 0.70, for the items comprising each construct.

Table 14: Means, standard deviations and correlation coefficients of the Pottery constructs


deviations)

Cronbach’s Alpha





4.83 (0.32)

0.961 1


4.76 (0.24)

0.746 0.884** 1

POTTER PERCEIVED PERFORMANCE

4.82 (0.37)

0.936 0.873* 0.707*




Considering the fit indices of the Pottery operational model (Chi-squared = 13.384, df = 6, p = 0.037, Normed Chi-squared = 2.231, RMR = 0.036, GFI = 0.677, NFI = 0.529, CFI = 0.599, RMSEA = 0.453) we see that some fit indices are acceptable, indicating that the estimated model presented in Figure 15 is acceptable. In fact, the estimated model in Figure 15 supports the theoretical model presented in Figure 14. Additionally, it is seen in Figure 15 that “Interactional characteristics” fully mediates the relationship between “Informational characteristics” and “Perceived performance”.

However, due to the very small sample size, we must note here that the standardized coefficients referring to each sub-construct in a general construct (circle) in Figure 15 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis. Moreover, in each



general construct the reported “Total Variance Explained” represents the percentage of the information that has been not lost due to the dimension reduction (from the actual items of the questionnaire to the corresponding latent sub-constructs) via factor analysis [7].

INFORMATIONAL

CHARACTERISTICS

Content (0.997)

Courses (0.997)

Total variance explained =

99.373%

0.88 0.68

INTERACTIONAL

CHARACTERISTICS

Game scenario (0.865)

Visualization (0.991)

User performance evaluation

(0.840)

Sensor set-up (0.655)

Usability (0.907)

Learning experience (0.782)


71.618%

POTTERY PERCEIVED

PERFORMANCE

Effectiveness (0.986)

Efficiency (0.764)

Satisfaction (0.986)

Innovation (0.986)


87.500%

EducationGender

-0.29 0.24

Figure 17: Estimation results of the operational model of Pottery




• Interactional characteristics fully mediate the relationship between Informational characteristics and Perceived performance.

• “Content” and “Courses with respect to Pottery in the LMS” are equally important Informational characteristics factors that predict performance.


• “Effectiveness”, “Satisfaction” and “Innovation” are equally important factors in determining perceived performance.

• The Informational – Interactional – Perceived performance relationship is influenced by demographics of the respondents (see Figure 10). In particular, people with higher education perceive performance to be higher, and female perceive performance to be lower.

• Considering that the performance construct is influenced by some demographic characteristics of the users, we conclude that Pottery belongs to the so-called “Contingency Systems”, which support the view that the system maximizes its performance according to the specific context within which it is operating [6].

In detail, Table 15 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of Interactional characteristics on efficiency is 0.518. That is, due to both direct (unmediated) and indirect (mediated) effects of Interactional characteristics on efficiency, when Interactional characteristics go



up by 1 standard deviation, efficiency goes up by 0.518 standard deviations (for further discussion of direct, indirect and total effects, see [10].

Table 15: Standardized Total Effects of the Pottery operational model




- -


0.884 -

POTTERY PERCEIVED PERFORMANCE Innovation Satisfaction Efficiency Effectiveness

0.599

0.591 0.591 0.458 0.591

0.678

0.668 0.668 0.518 0.668

The figures in Table 15 are also very important because they can guide the designers of the Pottery dance case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.

7.4 Salsa dance use case evaluation

7.4.1 Introduction

Figure 11 presents the operational model that is used in evaluating the Salsa dancing use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Informational characteristics (i.e., informational content for Salsa dancing and courses with respect to Salsa in LMS) and Perceived Performance, through the mediating mechanism of Interactional characteristics of the Salsa game-like application.



INFORMATIONAL

CHARACTERISTICS

Content

Courses

CONTROLS: Gender, Age, Education, Familiarity with computers, Experience in dancing,

Experience in Latin, Experience in Salsa, Type of respondent (Experts / Simple users)

INTERACTIONAL

CHARACTERISTICS

Game scenario

Visualization

User performance

evaluation

Sensor set-up

Usability

Learning experience

SALSA PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

Figure 18: Operational model of Salsa dancing

7.4.2 Methodology

According to the operational model proposed by the group working with the Salsa dance use case, the Part B of the questionnaire, presented in Appendix VI, is developed. As we said previously, Part A of the questionnaire is common to all use cases.

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the Salsa dancing use case (see Appendix VI). A demonstration activity was organized in a local dance school in Thessaloniki, Greece and involved a structured educational path for adults that want to learn how to dance Salsa. The activity was realized in the context of a Salsa dance class in this school. The learners followed the learning path of the corresponding LMS course with the help of their teacher, which also included a training session with the Salsa dance game. Finally, they also explored the other functionalities of the i-Treasures Web Platform (more details can be found in D6.2).

The purpose of the aforementioned demonstrations/workshops was firstly to present the web platform, the Salsa dance game-like application to the users, users then to experiment with these functionalities, and finally to collect data through the e-questionnaire. In total, 11 questionnaires were complete to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 16.

Table 16: Salsa dancing - Demographics (N = 11)


Gender: Male Female

7 4

63.6 36.4


4 7

36.4 63.6


1 7 3

9.1 63.6 27.3

Perceived familiarity in using computers:



Good Very Good

6 5

54.5 45.5

Experience in dancing: Adequate Good Very good

5 4 2

45.5 36.4 18.2

Experience in dancing Latin dances: Not much Adequate Good Very good

3 5 2 1

27.3 45.5 18.2 9.1

Experience in dancing Salsa: Not at all Not much Adequate Good Very good

1 1 6 2 1

9.1 9.1 54.5 18.2 9.1


1 10

9.1 90.9

7.4.3 Results


All Alphas are greater than 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are higher than ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of the Salsa use case is above ‘good’.

We must note here that due to the very small sample size we treated the Informational characteristics and Interactional characteristics constructs in their aggregate version. We did this aggregation into higher level constructs, due to the fact that Cronbach Alphas were greater than 0.70, for the items comprising each construct.

Table 17: Means, standard deviations and correlation coefficients of the Salsa constructs


deviations)

Cronbach’s Alpha





4.52 (0.64)

0.876 1


4.13 (0.40)

0.884 0.801** 1

SALSA PERCEIVED PERFORMANCE

4.70 (0.35)

0.716 0.711* 0.625*






Considering the fit indices of the Salsa dance operational model (Chi-squared = 20.799, df = 19, p = 0.348, Normed Chi-squared = 1.095, RMR = 0.045, GFI = 0.697, NFI = 0.603, CFI = 0.926, RMSEA = 0.097) we see that many indices are acceptable, indicating that the estimated model presented in Figure 17 is acceptable. In fact, the estimated model in Figure 17 supports the theoretical model presented in Figure 16. Additionally, it is seen in Figure 17 that “Interactional characteristics” partially mediates the relationship between “Informational characteristics” and “Perceived performance”.

However, due to the small sample size, we must note here that the standardized coefficients referring to each sub-construct in a general construct (circle) in Figure 17 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis.

INFORMATIONAL

CHARACTERISTICS

Content (0.946)

Courses (0.946)


89.459%

0.81 0.18

INTERACTIONAL

CHARACTERISTICS




(0.605)


Usability (0.885)

Learning experience (0.788)


68.371%

SALSA PERCEIVED

PERFORMANCE

Education

-0.27

Effectiveness Efficiency Satisfaction InnovationSalsa

0.18

0.82

0.88 0.53 0.51 0.27

Figure 19: Estimation results of the operational model of Salsa




• Interactional characteristics partially mediate the relationship between Informational characteristics and Perceived performance.

• “Content” and “Courses with respect to Salsa in the LMS” are equally important Informational characteristics factors that predict performance.


• “Effectiveness” is the most important factor in determining perceived performance.



• The Informational – Interactional – Perceived performance relationship is influenced by demographics of the respondents (see Figure 17). In particular, people with higher education perceive Informational characteristics to be lower, and people more familiar in Salsa perceive Interactional characteristics to be higher.

• Considering that the Informational and Interactional constructs are influenced by some demographic characteristics of the users, we conclude that Salsa belongs to the so-called “Contingency Systems”, which support the view that the system maximizes its performance according to the specific context within which it is operating [6].

In detail, Table 18 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of Informational characteristics on effectiveness is 0.847. That is, due to both direct (unmediated) and indirect (mediated) effects of Informational characteristics on effectiveness, when Informational characteristics go up by 1 standard deviation, effectiveness goes up by 0.847 standard deviations (for further discussion of direct, indirect and total effects, see [10].

Table 18: Standardized Total Effects of the Salsa operational model




- -


0.809 -

SALSA PERCEIVED PERFORMANCE Innovation Satisfaction Efficiency Effectiveness

0.965

0.265 0.494 0.511 0.847

0.179

0.049 0.092 0.095 0.157

The figures in Table 18 are also very important because they can guide the designers of the Salsa dance case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.

7.5 Walloon dance use case evaluation

7.5.1 Introduction

Figure 18 presents the operational model that is used in evaluating the Walloon dancing use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Informational characteristics and the Perceived Performance, through the mediating mechanism of the Learning Experience of the Walloon game-like application.



LEARNING

EXPERIENCE

Final Challenge

Fun

INFORMATIONAL

CHARACTERISTICS

Game scenario

Visualization

User performance

evaluation

Sensor set-up

Usability

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

CONTROLS: Gender, Age, Education, Familiarity with computers, Experience in dancing, Experience

in Walloon dancing

Figure 20: Operational model of Walloon dancing

7.5.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the Walloon dancing use case (see Appendix XI). Following the evaluation protocol, two scheduled demos have been organized in Belgium.

The first demonstration activity took place at the University of Mons. The presentations were made in the conference room of the university, while the live demonstration of the game-like application took place at the Numediart motion capture room. In this demonstration, 5 UMONS students, 1 dance expert and 12 citizens of Mons participated.

The second demonstration took place at the cultural centre of Beloeil in Quevaucamps, Belgium in the context of an event about traditional dances and songs in the Walloon region, namely “That’s Folk 2016”. In this event, 16 people participated. More details on demonstrations can be found in D6.2.

The purpose of the aforementioned demonstrations/workshops was firstly to present the web platform, the Walloon dance game-like application to the users, users then to experiment with these functionalities, and finally to collect data through the questionnaire. In total, 10 questionnaires were complete to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 19.



Table 19 : Walloon dancing - Demographics (N = 10)

Number Percentage

Gender: Male Female

5 5

50 50


6 4

60 40

Education: College University

4 6

40 60

Perceived familiarity in using computers: Adequate Good Very Good

2 4 4

20 40 40

Experience in dancing: Not at all Not much Good Very good

2 1 3 4

20 10 30 40

Experience in Walloon dance: Not much Adequate Good Very good

3 5 2 1

27.3 45.5 18.2 9.1

Experience in dancing Salsa: Not at all Adequate Good Very good

3 1 3 3

30 10 30 30

7.5.3 Results


All Alphas are greater than 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are between ‘3’ and ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of the Walloon use case is above ‘average’ but less than ‘good’.

We must note here that in the questionnaire, 11 items (questions) constitute the “Learning experience” construct. Considering the small size of the evaluation exercise we aggregated these items into factors, using exploratory factor analysis (EFA). For example, factor 1, constituted by items 1, 2, 3, 5, 6, 10 and 11 in the questionnaire, and factor 2, constituted by items 4, 7, 8, and 9. To factor 1 we gave the name of “Final challenge” (first dot in the Learning experience construct), due to the highest loading of item 1 in the EFA. To factor 1 we gave the name of “Fun” (second dot in the Learning experience construct), due to the highest loading of item 8 in the EFA.



Additionally, we must note that due to the very small sample size we treated the “Informational characteristics” and “Learning experience” constructs in their aggregate version. We did this aggregation into higher level constructs, due to the fact that Cronbach Alphas were greater than 0.70, for the items comprising each construct.

Table 20: Means, standard deviations and correlation coefficients of the Walloon constructs


deviations)

Cronbach’s Alpha



LEARNING EXPERIENCE


3.90 (0.48)

0.834 1

LEARNING EXPERIENCE

3.70 (0.59)

0.881 0.581 1

WALLOON PERCEIVED PERFORMANCE

3.98 (0.92)

0.861 0.450 0.618



Considering the fit indices of the Walloon dance operational model (Chi-squared = 9.514, df = 9, p = 0.391, Normed Chi-squared = 1.057, RMR = 0.063, GFI = 0.809, NFI = 0.774, CFI = 0.981, RMSEA = 0.080) we see that most indices are acceptable, indicating that the estimated model presented in Figure 19 is acceptable. In fact, the estimated model in Figure 19 supports the theoretical model presented in Figure 18. Additionally, it is seen in Figure 19 that “Learning experience” fully mediates the relationship between “informational characteristics” and “perceived performance”.

However, due to the small sample size, we must note here that the standardized coefficients referring to Informational Characteristics and Learning Experience in Figure 19 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis.

LEARNING EXPERIENCE

Final Challenge (0.840)

Fun (0.840)


70.751%

INFORMATIONAL

CHARACTERISTICS




(0.872)


Usability (0.856)


66.801%

WALLOON PERCEIVED

PERFORMANCE

Effectiveness Efficiency Satisfaction Innovation

0.79 1.00 0.87 0.52

0.58 0.64

Figure 21: Estimation results of the operational model of Walloon






• Learning experience fully mediates the relationship between Informational characteristics and Perceived performance.

• “Game scenario” is the most important Informational characteristics factor that predicts performance.

• “Final challenge” and “Fun” are equally important Learning experience factors that predict performance.

• “Efficiency” is the most important factor in determining perceived performance.

• The Informational characteristics – Learning experience – Perceived performance relationship is not influenced by demographics of the respondents (see Figure 19), indicating that the Walloon system belongs to the so-called “Universalistic Systems”, which support the view that the system maximizes its performance irrespectively to the specific context within which it is operating [6].

In detail, Table 21 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of Informational characteristics on efficiency is 0.369. That is, due to both direct (unmediated) and indirect (mediated) effects of Informational characteristics on efficiency, when Informational characteristics go up by 1 standard deviation, efficiency goes up by 0.369 standard deviations (for further discussion of direct, indirect and total effects see [10]).

Table 21: Standardized Total Effects of the Walloon operational model


LEARNING EXPERIENCE


- -

LEARNING EXPERIENCE 0.581 - WALLOON PERCEIVED PERFORMANCE Innovation Satisfaction Efficiency Effectiveness

0.370

0.191 0.323 0.369 0.291

0.636

0.329 0.556 0.635 0.500

The figures in Table 21 are also very important because they can guide the designers of the Walloon dance case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.



7.6 Human Beat Box use case evaluation

7.6.1 Introduction

Figure 20 presents the operational model that is used in evaluating the Human Beat Box (HBB) use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Informational characteristics of HBB learning platform and Perceived Performance, through the mediating mechanism of Learning experience.

QUALITY

CHARACTERISTICS

OF HUMAN BEAT

BOX LEARNING

PLATFORM

CONTROLS: Gender, Age, Education, Familiarity with computers, Experience in singing,

Experience in HBB, Experience in imitation of percussion, stringed instruments, wind instruments,

electronic sounds, and animals, Type of respondent (Experts / Simple users)

LEARNING

EXPERIENCE

PERCEIVED

PERFORMANCE OF

HUMAN BEAT BOX

LEARNING

PLATFORM

Figure 22: Operational model of HBB

7.6.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the HBB use case (see Appendix VIII). Following the evaluation protocol, various scheduled demos have been organized by CNRS. More specifically, following the evaluation protocol, sessions for the HBB game-like use case has been organized at the UPMC lab by CNRS and UPMC. After a short introduction to the project, the participants accessed the Web-platform and followed the HBB course of the LMS. The course contains various activities. The theoretical part focuses on the origins and history of the human beatbox, basics of rhythms and tempo, etc. A test activity was designed to evaluate the knowledge acquired by the students. Finally, a practice activity based on the use of the hyper-helmet and the HBB game-like application was also included. In the latter, the participants were asked to hear and try to imitate sounds previously recorded by experts. They, finally, completed an evaluation questionnaire. In total, 6 questionnaires were complete to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 22.



Table 22: HBB - Demographics (N = 6)


Gender: Male Female

3 3

50.0 50.0


3 3

50.0 50.0

Education: Post Grad

6

100.0

Perceived familiarity in using computers: Adequate Very good

1 5

16.7 83.3

Experience in singing: Not at all Not much Adequate

4 1 1

66.7 16.7 16.7

Experience in HBB: Not at all Not much

5 1

83.3 16.7

Experience in imitation of percussion Not at all Not much

5 1

83.3 16.7

Experience in imitation of stringed musical instruments Not at all

1

100.0

Experience in imitation of wind instruments Not at all Not much

5 1

83.3 16.7

Experience in imitation of electronic sounds Not at all Not much

4 2

66.7 33.3

Experience in imitation of animal sounds Not at all Not much Very good

3 2 1

50.0 33.3 16.7

Respondent’s ID: Simple user

6

100.0

7.6.3 Results


All Alphas are greater than 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are around ‘3’ indicating that in general the perception of individuals about the quality of the various characteristics of the HBB use case is around ‘average’.

We must note here that due to the very small sample size we treated all constructs in their aggregate version. We did this aggregation into higher level constructs, due to the fact that Cronbach Alphas were greater than 0.70, for the items comprising each construct.



Table 23: Means, standard deviations and correlation coefficients of the HBB constructs


deviations)

Cronbach’s Alpha


CHARACTERISTICS OF HBB

LEARNING EXPERIENCE


2.83 (0.84)

0.825 1

LEARNING EXPERIENCE

3.54 (0.76)

0.880 0.404* 1

PERCEIVED PERFORMANCE OF HBB

3.50 (0.61)

0.756 -0.065 0.240



Considering the fit indices of the HBB operational model (Chi-squared =0.888, df = 3, p = 0.828, Normed Chi-squared = 0.296, RMR = 0.054, GFI = 0.922, NFI = 0.846, CFI = na, RMSEA = 0.000) we see that many fit indices are acceptable, indicating that the estimated model presented in Figure 21 is acceptable. In fact, the estimated model in Figure 21 supports the theoretical model presented in Figure 20. Additionally, it is seen in Figure 21 that “Learning experience” fully mediates the relationship between “Characteristics of HBB” and “Perceived performance”.

However, due to the very small sample size, we must note here that the standardized coefficients referring to each aggregated (circle) in Figure 21 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis.

QUALITY


LEARNING PLATFORM

Helmet stabilization (0.927)

Helmet handling (0.846)

Applying ultra sound probe

(0.806)


74.143%

0.40 0.42

LEARNING

EXPERIENCE

Helmet improving voice

technique (0.823)

Evaluation improving voice

technique (0.890)

Modeling improving vocal

technique (0.973)

Visualization improving vocal

technique (0.547)


67.902%

HBB PERCEIVED

PERFORMANCE


Efficiency (0.890)


Innovation (0.865)


64.428%

Experience in BB

-0.67

Figure 23: Estimation results of operational model of HBB






• Learning experience fully mediates the relationship between Characteristics of HBB and Perceived performance.

• “Helmet stabilization” is the most important characteristic that predicts performance.

• “Modeling improving vocal technique” is the most important Learning experience that predicts performance.

• “Satisfaction” is the important factor in determining perceived performance.

• The Characteristics – Learning experience – Perceived performance relationship is influenced by demographics of the respondents (see Figure 21). In particular, people that have higher experience in HBB rate performance to be at lower levels.

• Considering that the performance construct is influenced by at least one demographic characteristic of the users, we conclude that HBB belongs to the so-called “Contingency Systems”, which support the view that the system maximizes its performance according to the specific context within which it is operating [6].

In detail, Table 24 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of the Learning experience on satisfaction is 0.378. That is, due to both direct (unmediated) and indirect (mediated) effects of the Learning experience on satisfaction, when Learning experience go up by 1 standard deviation, satisfaction goes up by 0.378 standard deviations (for further discussion of direct, indirect and total effects, see [10]).

Table 24: Standardized Total Effects of the HBB operational model

Constructs / items CHARACTERISTICS OF BM LEARNING EXPERIENCE

CHARACTERISTICS OF HBB - - LEARNING EXPERIENCE 0.404 - HBB PERCEIVED PERFORMANCE Innovation Satisfaction Efficiency Effectiveness

0.168

0.145 0.153 0.150 0.078

0.416

0.360 0.378 0.370 0.193

The figures in Table 24 are also very important because they can guide the designers of the HBB singing case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.



7.7 Byzantine use case evaluation

7.7.1 Introduction

Figure 22 presents the operational model that is used in evaluating the Byzantine music use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Quality Characteristics of Byzantine Music (BM) Learning Platform and Perceived Performance of the BM Learning Platform, through the mediating mechanism of Learning Experience.

QUALITY

CHARACTERISTICS

OF BYZANTINE

MUSIC LEARNING

PLATFORM

CONTROLS: Gender, Age, Education, Familiarity with computers, Role in BM, Experience in BM,

Experience in Choral and European music, and Stringed music instruments, Type of respondent

(Experts / Simple users)

LEARNING

EXPERIENCE

PERCEIVED

PERFORMANCE OF

BYZANTINE MUSIC

LEARNING

PLATFORM

Figure 24: Operational model of Byzantine music

7.7.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the BM use case (see Appendix VII). Following the evaluation protocol, various scheduled demos have been organized by UOM. More specifically: the first event took place in the Experimental School of university of Thessaloniki (AUTH) where 20 students participated. A second demonstration event took place in the Byzantine Chanter of Church of Saint Nikolaos in Larisa, where 5 Byzantine music chanters got involved. The final demonstration event took place in UOM, where 5 postgraduate students participated. More details on demonstrations can be found in D6.2.

After a short introduction to the project, the participants accessed the Web-platform and followed the Byzantine music course of the LMS. After the users got familiarized with the web platform and its content, they were invited through the learning path to chant and to get the sensorimotor feedback improving their singing techniques.They, finally, completed an evaluation questionnaire. In total, 25 questionnaires were complete to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 25.



Table 25: Byzantine music - Demographics (N = 25)


Gender: Male Female

15 10

60 40


15 10

60 40

Education: Basic Secondary College University Post Grad

15 1 1 5 3

60 4 4 20 12

Perceived familiarity in using computers: Not much Adequate Good Very good

2 10 8 5

8 40 32 20

Role in Byzantine music: Cantor Chanter Chorister Priest Other

4 7 7 5 2

16 28 28 20 8

Experience in Byzantine music: Not at all Not much Adequate Good

8 6 7 4

32 24 28 16

Experience in Choral music: Not at all Not much Adequate Good Very good

4 3 8 3 7

16 12 32 12 28

Experience in European music: Not at all Not much Adequate Good Very good

4 4 9 5 3

16 16 36 20 12

Experience in stringed musical instruments: Not at all Not much Adequate Good Very good

9 7 6 1 2

36 28 24 4 8


6 19

24 76



7.7.3 Results


All Alphas are greater than 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are between ‘3’ and ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of the BM use case is above ‘average’ but less than ‘good’.

We must note here that in the structured questionnaire although each construct is individually presented, in the operational model we treated the items of learning experience as nine related items under the general learning experience construct. We did this aggregation of items into higher level construct, due to the fact that the sample size was rather small and the Cronbach Alphas were more than 0.70, for the items comprising this construct.

Table 26: Means, standard deviations and correlation coefficients of the BM constructs


deviations)

Cronbach’s Alpha


CHARACTERISTICS OF BM

LEARNING EXPERIENCE

CHARACTERISTICS OF BM

3.75 (0.74)

0.931 1

LEARNING EXPERIENCE

3.66 (0.78)

0.908 0.768 1

PERCEIVED PERFORMANCE OF BM

3.77 (0.66)

0.797 0.680 0.732



Considering the fit indices of the BM operational model (Chi-squared = 515.635, df = 207, p = 0.000, Normed Chi-squared = 2.491, RMR = 0.148, GFI = 0.467, NFI = 0.356, CFI = 0.458, RMSEA = 0.249) we see that some indices are acceptable, indicating that the estimated model presented in Figure 23 is acceptable. In fact, the estimated model in Figure 23 supports the theoretical model presented in Figure 22. Additionally, it is seen in Figure 23 that “Learning experience” partially mediates the relationship between “Characteristics of BM” and “BM perceived performance”.

However, due to the rather small sample size, we must note here that the standardized coefficients referring to the Learning experience construct (circle) in Figure 23 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis.



0.40

LEARNING

EXPERIENCE

Useful (0.890)

Techniques

(0.897)

User friendly

(0.666)

Easy to learn

(0.895)

Improving

(0.603)

Correcting

(0.709)

Metronome

(0.696)

Fun (0.881)

Enjoyable

(0.791)

Total variance

explained =

62.143%

Effectiveness

Efficiency

Satisfaction

Innovation

Objectives

Game scenario

Getting started

Virtual tutor

3D environment

Avatars

appearance

Accuracy

Music score

Video

Visual feedback

Comfortable

Not confusing

Feedback

Activity

CHARACTERISTICS

OF BM LEARNING

PLATFORM

PERCEIVED

PERFORMANCE

OF BM

LEARNING

PLATFORM

0.490.76

0.80

0.91

0.92

0.76

0.80

0.71

0.70

0.63

0.66

0.61

0.78

0.43

0.43

0.60

0.93

0.64

0.85

0.77

Byzantine

music

Role

Respondent

ID

-0.25 -0.39

0.22

Figure 25: Estimation results of the operational model of Byzantine music




• Learning experience partially mediates the relationship between Characteristics of BM and Perceived performance.

• “I have easily understood what I have to do in each ‘activity’ of the game” and “I found the ‘game scenario’ clear and satisfactory” are almost equally the most important Characteristics of BM items that predict performance.

• “I think that comparing my performance with expert’s is a way to evaluate my chanting ‘techniques’” and “It is ‘easy to learn’ to perform/reproduce watching the expert’s musical gestures” are almost equally the most important Characteristics of Learning experience items that predict performance.

• “Effectiveness” is the most important factor in determining perceived performance.

• The Characteristics of BM – Learning experience – Perceived performance relationship is influenced by demographics of the respondents (see Figure 23). In particular, people with higher knowledge of BM perceive Characteristics of BM to be lower, people get different Learning experience according to their role in BM, and experts in BM perceive performance of the BM learning platform to be lower.

• Considering that all constructs are influenced by some demographic characteristics of the users, we conclude that BM belongs to the so-called



“Contingency Systems”, which support the view that the system maximizes its performance according to the specific context within which it is operating [6].

In detail, Table 27 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of Characteristics of BM on effectiveness is 0.775. That is, due to both direct (unmediated) and indirect (mediated) effects of Characteristics of BM on effectiveness, when Characteristics of BM go up by 1 standard deviation, effectiveness goes up by 0.775 standard deviations (for further discussion of direct, indirect and total effects, see [10]).

Table 27: Standardized Total Effects of the BM operational model

Constructs / items CHARACTERISTICS OF BM LEARNING EXPERIENCE

CHARACTERISTICS OF BM - - LEARNING EXPERIENCE 0.759 - BM PERCEIVED PERFORMANCE Innovation Satisfaction Efficiency Effectiveness

0.771

0.599 0.427 0.432 0.775

0.491

0.376 0.417 0.314 0.455

The figures in Table 27 are also very important because they can guide the designers of the Byzantine music case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.

7.8 Canto a Tenore case evaluation

7.8.1 Introduction

Figure 24 presents the operational model that is used in evaluating the Canto a Tenore use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Design of LMS and Activities usefulness, through the mediating mechanisms of Course Contents in LMS and the Technologies (resources and tools) used during the course.



DESIGN

of the LMS

Accessibility

Navigation

Difficulty

Organization


Experience in Canto a Tenore singing, Roles, Type of respondent (Experts / Simple users)

CONTENTS

of Course

Organization

Clarity

Completeness

Adequateness

TECHNOLOGY

used during course

Easy to use

Useful

ACTIVITIES

(usefulness based

on)

Presentations

Practical activities

Discussion

Tests

Figure 26: Operational model of Canto a Tenore

7.8.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the Canto a Tenore use case (see Appendix X). Following the evaluation protocol, various scheduled demos have been organized by CNR. Data were collected during two demonstrations.

The first one was carried out at the upper secondary school “A. Volta” in Nuoro, Sardinia (Italy). The demonstration took place as an extra-curricular activity and involved 10 students (age 14-19) with different levels of competence in Canto a Tenore, participating spontaneously. Students completed the course on Canto a Tenore in 3 weeks (May 18th-June 6th 2016) and at the end of the course filled in the part of the questionnaire oriented to the LMS and Canto a Tenore course evaluation.

The second demonstration was carried out as an online workshop for experts. A group of 10 experts in musicology and Canto a Tenore browsed the platform and followed the Online Course for Canto a Tenore between November and December 2016. At the end of the period, experts filled in all parts of the questionnaire. In total, 14 questionnaires were properly completed to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 28. (More details on demonstrations can be found in D6.2.).

Table 28: Canto a Tenore - Demographics (N = 14)

Number Percentage



(%)

Gender: Male Female

14 0

100.0 0.0

Age: 14 - 20 46 - 60 More than 60

6 7 1

42.9 50.0 7.1

Education: Secondary University

7 5

50.0 50.0


1 7 4 2

7.1 50.0 28.6 14.3

Experience in singing: Not at all Not much Adequate Good Very good

4 4 3 1 2

28.6 28.6 21.4 7.1 14.3

Experience in Canto a Tenore singing: Not at all Not much Adequate Good Very good

2 3 3 4 2

14.3 21.4 21.4 28.6 14.3

Role: Boghe Mesu boghe Bassu Contra

2 2 5 5

14.3 14.3 35.7 35.7


8 6

57.1 42.9

7.8.3 Results


All Alphas are greater than 0.70 indicating constructs internal consistency. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are between ‘3” and ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of the Canto a Tenore use case is above ‘average’ but less than ‘good’.

We must note here that in the structured questionnaire although each construct is individually presented, in the operational model we treated the items of all constructs to be associated. We did this aggregation of items into higher level construct, due to



the fact that the sample size was rather small and the Cronbach Alphas were more than 0.70, for the items comprising this construct.

Table 29: Means, standard deviations and correlation coefficients of the Canto a Tenore constructs


deviations)

Cronbach’s Alpha


DESIGN CONTENT TECHNOLOGY

DESIGN 3.89 (0.87)

0.894 1

CONTENT 3.95 (0.72)

0.784 0.626*

1

TECHNOLOGY 3.92 (0.75)

0.801 0.670** 0.307* 1

ACTIVITIES 3.50 (1.20)

0.948 0.653* 0.553* 0.615*




Considering the fit indices of the Canto a Tenore operational model (Chi-squared = 9.358, df = 9, p = 0.405, Normed Chi-squared = 1.040, RMR = 0.132, GFI = 0.846, NFI = 0.763, CFI = 0.985, RMSEA = 0.055) we see that many indices are acceptable, indicating that the estimated model presented in Figure 25 is acceptable. In fact, the estimated model in Figure 25 supports the theoretical model presented in Figure 24. Additionally, it is seen in Figure 25 that “Content” and “Technology” fully mediate the relationship between “Design” and “Activities”.

However, due to the rather small sample size, we must note here that the standardized coefficients referring to all constructs (circles) in Figure 25 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis.



DESIGN

of the LMS

Accessibility (0.594)

Navigation (0.974)

Difficulty (0.974)

Organization (0.924)

Total variance

explained

= 77.628%

CONTENTS

of Course


Clarity (0.772)

Completeness

(0.491)

Adequateness

(0.889)

Total variance

explained

= 61.370%

TECHNOLOGY

used during course

Easy to use (0.916)

Useful (0.916)

Total variance

explained

= 83.882%

ACTIVITIES

(usefulness based

on)

Presentations (0.896)

Practical activities

(0.905)

Discussion (0.980)

Tests (0.956)

Total variance

explained

= 87.388%

0.63

0.68

0.31

0.48

Computers

Role-0.20

-0.44

Figure 27: Estimation results of the operational model of Canto a Tenore



• The model rather acceptably predicts activities (all standardized coefficients are significant and positive, and fit indices are in general acceptable).

• Contents and Technology fully mediate the relationship between Design and Activities.

• “Navigation” and “Difficulty” are equally the most important characteristics of Design items that predict Activities.

• “Organization” is the most important characteristic of Contents items that predict Activities.

• “Easy to use” and “Usefulness” are equally the most important characteristics of Technology items that predict Activities.

• “Discussion” is the most important factor in determining Activities.

• The Design – Contents & Technology – Activities relationship is influenced by demographics of the respondents (see Figure 25). In particular, people with higher familiarity in using computers perceive characteristics of Technology to be lower, and people perceive differently Activities usefulness according to their role in Canto a Tenore.

• Considering that some constructs are influenced by some demographic characteristics of the users, we conclude that Canto a Tenore belongs to the so-called “Contingency Systems”, which support the view that the system



maximizes its performance according to the specific context within which it is operating [6].

In detail, Table 30 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of Design on discussion is 0.515. That is, due to both direct (unmediated) and indirect (mediated) effects of Design on discussion, when Design go up by 1 standard deviation, discussion goes up by 0.515 standard deviations (for further discussion of direct, indirect and total effects, see [10]).

Table 30: Standardized Total Effects of the Canto a Tenore operational model

Constructs / items DESIGN CONTENT TECHNOLOGY

DESIGN - - - CONTENT 0.626 - - TECHNOLOGY 0.684 - - ACTIVITIES Presentations Practical activities Discussion Tests

0.525

0.470 0.475 0.515 0.502

0.313

0.280 0.283 0.307 0.299

0.481

0.431 0.435 0.471 0.460

The figures in Table 30 are also very important because they can guide the designers of the Canto a Tenore use case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.



7.9 Cantu in Paghjella use case

7.9.1 Introduction

Figure 26 presents the operational model that is used in evaluating the Cantu in Paghjella use case. The ultimate objective of the model is to be used for the evaluation of the relationship between the Design of LMS and Usability, through the mediating mechanisms of Course Contents in LMS and the Technologies (resources and tools) used during the course.

DESIGN

of the LMS

Accessibility

Navigation

Difficulty

Organization


Experience in Cantu in Paghjella, Voice, Type of respondent (Experts / Simple users)

CONTENTS

of Course

Organization

Clarity

Completeness

Adequateness

TECHNOLOGY

used during course

Tools

Resources

USABILITY

(ease of)

Understanding

Learning

Handling

Figure 28: Operational model of Cantu in Paghjella

7.9.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the Cantu in Paghjella use case (see Appendix XI). Following the evaluation protocol, two scheduled demos have been organized by CNRS.

The first event took place in the Museum of Corsica, where the audience had very enriching exchanges after assessing the web platform but no questionnaires were filled in. The second demonstration event took place in the Phonetics lab of Paris where 13 participants filled in the questionnaires: 10 postgraduate students and 3 university participants with different levels of competence in Cantu in Paghjella got involved. Data were collected during this second demonstration.

After a short introduction to the project, the participants accessed the Web-platform and followed the Cantu in Paghjella course of the LMS.



In total, 13 questionnaires were properly completed to be used in the evaluation exercise. The demographics of the respondents to the questionnaire are presented in Table 31.

Table 31 Cantu in Paghjella - Demographics (N = 13)


Gender: Male Female

4 9

30.9 69.2


5 8

38.5 61.5

Education: University Postgraduate

3 10

23.1 76.9


1 2 5 5

7.7 15.4 38.5 38.5

Experience in singing: Not at all Not much Adequate Good Very good

2 3 3 4 1

15.4 23.1 23.1 30.8 7.7

Experience in Cantu in Paghjella: Not at all Not much Adequate Good

8 3 1 1

61.5 23.1 7.7 7.7


1 12

7.7 92.3

7.9.3 Results


All Alphas (except technology) are greater than 0.70 indicating constructs internal consistency. The Alpha for the technology construct is much less than 0.70, and thus we treated in estimation individually its dimensions. Moreover, we see that all correlation coefficients between pairs of constructs are much less than unity, verifying construct discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are around ‘4’ indicating that in general the perception of individuals about the quality of the various characteristics of the Cantu in Paghjella use case is rather ‘good’.

We must note here that in the structured questionnaire although each construct is individually presented, in the operational model we treated the items of all constructs to be associated. We did this aggregation of items into higher level construct, due to



the fact that the sample size was rather small and the Cronbach Alphas were more than 0.70, for the items comprising this construct.

Table 32: Means, standard deviations and correlation coefficients of the Canto a Tenore constructs


deviations)

Cronbach’s Alpha


DESIGN CONTENT TECHNOLOGY

DESIGN 3.13 (0.65)

0.737 1

CONTENT 4.27 (0.59)

0.849 0.101 1

TECHNOLOGY 4.04 (0.59)

-0.200 -0.041 0.414* 1

USABILITY 3.67 (0.73)

0.772 0.563* 0.706** 0.415*




Considering the fit indices of the Cantu in Paghjella operational model (Chi-squared = 42.750, df = 24, p = 0.011, Normed Chi-squared = 1.781, RMR = 0.101, GFI = 0.712, NFI = 0.541, CFI = 0.672, RMSEA = 0.255) we see that some indices are acceptable, indicating that the estimated model presented in Figure 7.10.2 is acceptable. In fact, the estimated model in Figure 27 supports the theoretical model presented in Figure 26. Additionally, it is seen in Figure 27 that “Content” and “Technology” partially mediate the relationship between “Design” and “Usability”.

However, due to the rather small sample size, we must note here that the standardized coefficients referring to all constructs (circles) in Figure 27 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis.



TECHNOLOGY

DESIGN

of the LMS

Accessibility (0.916)

Navigation (0.937)

Difficulty (0.553)


Total variance

explained

= 58.362%

CONTENTS

of Course


Clarity (0.884)

Completeness

(0.714)

Adequateness

(0.896)

Total variance

explained

= 70.975%

0.40

0.96

Tools

USABILITY

Easy of:

Understanding

Learning

Handling

0.76

0.48

Resources

0.10 0.78

0.09 0.29

-0.15 -0.25

Cantu in

Paghjella

Education0.09

-0.07

Figure 29: Estimation results of the operational model of Cantu in Paghjella



• The model rather acceptably predicts activities (all standardized coefficients are significant, and fit indices are in general acceptable).

• Contents and Technology partially mediate the relationship between Design and Usability.

• “Navigation” and “Accessibility” are the most important characteristics of Design items that predict Usability.

• “Adequateness” and “Clarity” are the most important characteristic of Contents items that predict Usability.

• “Tools” and “Resources” are two characteristics of Technology that independently predict Activities. We note here that although the links referring to Resources are negative, the impact of Design to Usability through Recourses is positive.

• “Understanding” is the most important factor in determining Usability.

• The Design – Contents & Technology – Usability relationship is influenced by demographics of the respondents (see Figure 27). In particular, people with higher familiarity in Cantu in Paghjella perceive the characteristics of Usability to be at lower levels, and people with higher education feel that the Content of courses is at higher levels.

• Considering that some constructs are influenced by some demographic characteristics of the users, we conclude that Cantu in Paghjella belongs to the so-called “Contingency Systems”, which support the view that the system maximizes its performance according to the specific context within which it is operating [6].



In detail, Table 33 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized total (direct and indirect) effect of Design on understanding is 0.516. That is, due to both direct (unmediated) and indirect (mediated) effects of Design on understanding, when Design go up by 1 standard deviation, understanding goes up by 0.516 standard deviations (for further discussion of direct, indirect and total effects, see [10].

Table 33: Standardized Total Effects of the Cantu in Paghjella operational model

Constructs / items

DESIGN CONTENT TECHNOLOGY Resources

TECHNOLOGY Tools

DESIGN - - - - CONTENT - - - - TECHNOLOGY-Resources

-0.145

-

-

-

TECHNOLOGY-Tools

0.094

-

-

-

USABILITY Handling Learning Understanding

0.538

0.255 0.407 0.516

0.776

0.369 0.588 0.745

-0.247

-0.117 -0.187 -0.237

0.289

0.137 0.219 0.278

The figures in Table 33 are also very important because they can guide the designers of the Cantu in Paghjella use case to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.

7.10 Generic Game Framework case evaluation

7.10.1 Introduction

Figure 28 presents the operational model that is used in evaluating the ITGD tool and the Generic Game Framework. The ultimate objective of the model is to be used for the evaluation of the relationship between the Activities of recording dance courses using the tool design and creation of educational gaming ITGD & the Pedagogical Planner, and Perceived performance & Preservation and Transmission of Intangible Cultural Heritage, though the mediating mechanism of the functions of educational game and 3D environment generated by the ITGD tool.



FUNCTIONS OF

PEDAGOGICAL

GAME

Scenario

Visualization

Usefulness

Learning

experience

Types of dance

Types of exercises

ACTIVITIES

Installation and

sensor adjustment

Recording

procedures of

dance courses

Functions of

pedagogical

planner

PERCEIVED

PERFORMANCE

Effectiveness

Efficiency

Satisfaction

Innovation

CONTROLS: Gender, Age, Education, Familiarity with computers, Familiarity with dancing, Familiarity

with traditional Greek dancing, Respondent ID

ICH

Contribution

Usefulness

Adoption

Recommendation

Transmission

Spontaneous

Tradition

Innovative

Figure 30: Generic game framework

7.10.2 Methodology

Based on the operational model a structured questionnaire was developed by experts and the group of people working in the development of the of the Generic game use case (see Appendix XII). A demonstration activity was organized in the Department of Physical Education and Sports Science of the Aristotle University of Thessaloniki in cooperation with our Greek traditional dances expert and involved undergraduate students at the last year of their studies that their major/specialty was “Dance didactics”. The students used i-Treasures tools to designed and deliver a set of educational tools for teaching different Greek traditional dances. More specifically, they used a) the Pedagogical Planner to produce pedagogical plans for each dance and automatically create corresponding courses in the LMS, b) the LMS functionalities to develop/deliver these courses, and c) the generic framework for the development of dance games to design and develop game-like applications for different dances, using as an example the plans, courses and games delivered for Tsamiko and Salsa (more details can be found in D6.2).

The purpose of the aforementioned demonstrations/workshops was firstly to present the web platform, the ITGD tool, the Generic framework to the users, users then to experiment with these functionalities, and finally to collect data through the e-questionnaire. Although many users took part in the exercise and responded to the



questionnaire, finally, only 5 questionnaires were adequately answered. The demographics of the respondents to the questionnaire are presented in Table 34.

Table 34: Generic framework - Demographics (N = 5)


Gender: Male Female

1 4

20.0 80.0


4 1

80.0 20.0

Education: University Post Grad

2 3

40.0 60.0

Perceived familiarity in using computers: Adequate Good Very good

2 2 1

40.0 40.0 20.0

Experience with dancing: Adequate Good

2 3

40.0 60.0

Experience with traditional Greek dancing: Adequate Very good

3 2

60.0 40.0


1 4

20.0 80.0

7.10.3 Results


All Alphas (except one) are greater than 0.70 indicating constructs internal consistency. Moreover, we see that some correlation coefficients between pairs of constructs are greater than 0.90, producing worries about constructs discriminant validity. Furthermore, we see that the mean values of all constructs at the 5-level quality scale are between “3” and “4” indicating that in general the perception of individuals about the quality of the various characteristics of the Generic use case is between ‘average’ and ‘good’.

We must note here that in the structured questionnaire although each construct is individually presented, in the operational model we treated the items of all constructs to be associated. We did this aggregation of items into higher level construct, due to the fact that the sample size was very small and the Cronbach Alphas were more than 0.70, for the items comprising this construct.



Table 35: Means, standard deviations and correlation coefficients of the Generic framework constructs


deviations)

Cronbach’s Alpha


ACTIVITIES FUNCTIONS PERFORMANCE

ACTIVITIES 3.45 (0.48)

0.571 1

FUNCTIONS 3.46 (0.78)

0.913 0.964** 1

PERFORMANCE 3.78 (0.85)

0.937 0.806* 0.807* 1

ICH 3.63 (0.71)

0.911 0.961** 0.922** 0.794*




Considering the fit indices of the Generic framework operational model (Chi-squared = 0.464, df = 2, p = 0.992, Normed Chi-squared = 0.233, RMR = 0.001, GFI = 0.948, NFI = 0.986, CFI = 1.000, RMSEA = 0.000) we see that many indices are acceptable, indicating that the estimated model presented in Figure 29 is acceptable. In fact, the estimated model in Figure 29 supports the theoretical model presented in Figure 28. Additionally, it is seen in Figure 29 that “Functions of Pedagogical Game” fully mediates the relationship between “Activities” and “Perceived Performance” & “ICH”.

However, due to the very small sample size, we must note here that the standardized coefficients referring to all constructs (circles) in Figure 29 have not been estimated directly using the SEM methodology, but instead, they have been estimated individually through confirmatory factor analysis.



ACTIVITIES

Installation and sensor

adjustment (0768)

Recording procedures

of dance courses

(0.562)

Functions of

pedagogical planner

(0.832)

Total variance

explained

= 54.306%

FUNCTIONS OF

PEDAGOGICAL

GAME

Scenario (0.923)


Usefulness (0.929)

Learning experience

(0.874)

Types of dance (0.485)

Types of exercises

(0.859)

Total variance

explained

= 73.992%

PERCEIVED

PERFORMANCE


Efficiency (0.830)


Innovation (0.958)

Total variance

explained = 86.676%

ICH

Contribution (0.974)

Usefulness (0.937)

Adoption (0.949)

Recommendation

(0.937)

Transmission (0.807)

Spontaneous (-0.223)

Tradition (0.791)

Innovative (0.845)

Total variance

explained

= 70.605%

0.98

0.82

0.99

-0.18

Figure 31: Estimation results of the operational model of Generic framework


The above results, which should be considered with caution due to the very small sample size, highlight that:

• The model rather acceptably predicts activities (all standardized coefficients are significant, and fit indices are in general acceptable).

• Functions of pedagogical game fully mediate the relationship between Activities and Perceived performance & ICH.

• “Functions of the pedagogical planner” are the most important characteristics of Activities that predict Performance & ICH.

• “Visualization” is the most important characteristic of Functions of pedagogical game factors that predict Performance & ICH.

• “Effectiveness” is the most important item in determining Perceived performance.

• “Contribution” is the most important item in determining ICH.

• Fear of new technologies that may negatively affect “spontaneous” issues of ICH is supported by the negative coefficient of the “spontaneous” item in the ICH construct.

• Due to the very small sample size it was impossible to test any influence of demographics of the respondents on the Generic framework.

In detail, Table 36 presents the total standardized effect of each column variable on each row variable after standardizing all variables. For example, the standardized



total (direct and indirect) effect of Activities on effectiveness is 0.782. That is, due to both direct (unmediated) and indirect (mediated) effects of Activities on effectiveness, when Activities go up by 1 standard deviation, effectiveness goes up by 0.782 standard deviations (for further discussion of direct, indirect and total effects, see [10]).

Table 36: Standardized Total Effects of the Generic framework operational model

Constructs / items ACTIVITIES FUNCTIONS OF PEDAGOGICAL

GAME

ACTIVITIES - - FUNCTIONS OF PEDAGOGICAL GAME

0.980 -

PERFORMANCE 0.800 0.816 Innovation Satisfaction Efficiency Effectiveness

0.766 0.745 0.664 0.782

0.782 0.760 0.677 0.781

ICH Contribution Usefulness Adoption Recommendation Transmission Spontaneous Tradition Innovative

0.970 0.945 0.909 0.920 0.909 0.783 -0.216 0.767 0.820

0.990 0.964 0.928 0.939 0.928 0.799 -0.221 0.783 0.828

The figures in Table 36 are also very important because they can guide the designers of the Generic Framework to put effort in improving the various entities of the system in order to get better results, as perceived by the users of the system. However, any amendment should take into consideration the cost-benefit analysis of each change.



8. Overall conclusions and proposals for the future

8.1 Conclusions with respect to laboratory testing

Based on the results of the laboratory, technical assessment of the standalone components of the i-Treasures platform, it can be concluded that, despite the highly challenging targets, the performance of the individual modules of the developed platform is rather satisfactory, meeting the majority of the user requirements and system specifications set at the beginning of the projects and updated after the first development cycle. These effective and efficient module lead to competent sub use cases and use cases and a solid integrated i-Treasures platform. It should be highlighted that certain limitations have been identified that leave ample room for improvement in the future. Nevertheless, the performance of the developed modules surpasses the one of relevant baseline approaches and their impact on the corresponding scientific fields is undeniable. It is clear that all relevant responsible partners, i.e., CERTH, AUTH, UOM, UMONS, ARMINES, CNRS, UPMC and ACAPELLA have spared no effort to develop effective and efficient data analysis and singing voice synthesis modules that pave the way for a competent ICH resource platform, such as i-Treasures, that aims to enable knowledge exchange and contribute to transmission of the rare know-how from living human treaures to apprentices.

8.2 Findings with respect to the Technical Evaluation

The purpose of this section is to recall, extend and present the major findings in the previous sections, which were based on the “component ranking rule”. According to this rule we distinguish two steps: First, the scores for each component of the theme under investigation (i.e., overall system, technical indices, web platform, and use cases) are computed. Then, these scores are compared in order to find the priority for possible amendments/changes which should be taken for improving a specific component. We note here that “the smaller the score of a component, the higher its priority for amendments”.

8.2.1 Overall Assessment of the System

Figure 30 presents the scores of components with respect to the overall assessment of the system. It is seen that “Accessibility / Usability” and “Auditing” are the two components with highest priority for improvement.

Figure 32: Overall Assessment of the System component score



8.2.2 Assessment of the Main Technical Indices

Figure 31 presents the scores of components with respect to the overall assessment of the main technical indices. It is seen that “Facial expression analysis” and “Vocal tract sensing and modeling – Ultrasound analysis” are the two components with highest priority for improvement.

Figure 33: Assessment of the Main Technical Indices component scores

8.3 Findings with respect to the Web-Platform evaluation

Figure 32 presents the scores of components with respect to the web platform evaluation. It is seen that “FUNCTIONALITIES - Semantic” and “FUNCTIONALITIES – Text to song” are the two components with highest priority for improvement.



Figure 34: Web platform component scores

8.4 Findings with respect to the Use-Cases evaluation

8.4.1 Contemporary music composition use case

Figure 33 presents the scores of components with respect to the Contemporary Music Composition use case. It is seen that “LEARNING – Real time recognition results” and “IMI – Simple to use” are the two components with highest priority for improvement.



Figure 35: Contemporary music composition use case component scores

8.4.2 Tsamiko dance use case

Figure 34 presents the scores of components with respect to the Tsamiko dance use case. It is seen that “INTERACTIONAL – Learning experience” and “INTERACTIONAL – Visualization” are the two components with highest priority for improvement.

Figure 36: Tsamiko dance use case component scores



8.4.3 Pottery use case

Figure 35 presents the scores of components with respect to the Pottery use case. It is seen that “INFORMATIONAL – Content” and “INTERACTIONAL – Visualization” are the two components with highest priority for improvement.

Figure 37: Pottery use case component scores

8.4.4 Salsa dance use case

Figure 36 presents the scores of components with respect to the Salsa dance use case. It is seen that “INFORMATIONAL – Courses” and “INTERACTIONAL – Sensor set up” are the two components with highest priority for improvement.



Figure 38: Salsa dance use case component scores

8.4.5 Walloon use case

Figure 37 presents the scores of components with respect to the Walloon dance use case. It is seen that “INFORMATIONAL – User performance evaluation” and “LEARNING EXPERIENCE – Final challenge” are the two components with highest priority for improvement.

Figure 39: Walloon dance use case component scores

Γενικός τύπος Γενικός τύπος Γενικός τύπος Γενικός τύπος Γενικός τύπος Γενικός τύπος Γενικός τύπος Γενικός τύπος Γενικός τύπος Γενικός τύπος

INFORMATIONAL - Sensor set up

INFORMATIONAL - Game scenario

LEARNING EXPERIENCE - Fun

INFORMATIONAL - Visulaization

INFORMATIONAL - Usability

LEARNING EXPERIENCE - Final challenge

INFORMATIONAL - User performanceevaluation



8.4.6 Human Beat-Box use case

Figure 38 presents the scores of components with respect to the Human Beat Box use case. It is seen that “PLATFORM – Helmet stabilization” and “PLATFORM – Helmet handling” are the two components with highest priority for improvement.

Figure 40: Human Beat Box use case component scores

8.4.7 Byzantine Music use case

Figure 39 presents the scores of components with respect to the Byzantine music use case. It is seen that “PLATFORM – Activity” and “LEARNING EXPERIENCE – Techniques” are the two components with highest priority for improvement.



Figure 41: Byzantine music use case component scores

8.4.8 Canto a Tenore use case

Figure 40 presents the scores of components with respect to the Canto a Tenore use case. In the case of Canto a Tenore, the evaluation was carried out for the LMS platform. It is seen that “DESIGN – Difficulty” and “DESIGN – Navigation” are the two components with highest priority for improvement.



Figure 42: Canto a Tenore use case component scores

8.4.9 Cantu in Paghjella

Figure 41 presents the scores of components with respect to the Cantu in Paghjella use case. It is seen that “DESIGN – Accessibility” and “DESIGN – Navigation” are the two components with highest priority for improvement.

Figure 43: Cantu in Paghjella use case component scores



8.4.10 Generic Game use case

Figure 42 presents the scores of components with respect to the Generic Framework. It is seen that “FUNCTIONS – Usefulness” and “FUNCTIONS – Learning experience” are the two components with highest priority for improvement.

Figure 44 Generic framework component scores

8.5 Overall Performance of i-Treasures

Integrating most of the results produced in the previous sections we present in Table 37 an estimate for the overall performance of i-Treasures. This estimate depends on a step-by-step weighed average, based on the results of confirmatory factor analysis.

It is seen in the last column of Table 37 that the overall performance of i-Treasures, in a 5-level scale (i.e., ‘1’=very poor, ‘2’=poor, ‘3’=average, ‘4’=good, ‘5’=very good) is equal to ‘3.95’. This means that the current performance of i-Treasures is at the level of ‘good’. Comparing this overall performance score (3.95) with the overall performance score of last year’s evaluation (3.67) we see that the overall evaluation of the system has been increased by 7.6 percent. This notable increase was due to the fact that the priorities for improvement of components proposed in last year’s evaluation were followed. However, still this performance may increase in the future by focusing on amendments/changes of the system, based on the priorities for improvement of components proposed previously.



Table 37: Evaluation of the overall performance of i-Treasures

Means Loadings Weighted Means

Total Variance

Explained (%)

OVERALL PERFORMANCE

WEB PLATFORM 3.95

Effectiveness 3.97 0.896 4.00 71.759 Efficiency 3.94 0.806 Satisfaction 3.89 0.863 Innovation 4.22 0.821 WEB-PRESERVATION & TRANSMISSION OF ICH

Contribution 4.14 0.776 4.15 55.547 Adoption 3.97 0.736 Usefulness 4.15 0.812 Recommendation 4.04 0.768 Transmission 4.3 0.762 Spontaneous 4.03 0.666 Respects Tradition

4.21 0.729

Innovative 4.34 0.703 CMC

Effectiveness 4.08 0.908 4.11 57.665 Efficiency 4.00 0.770 Satisfaction 3.97 0.855 Innovation 4.67 0.397 TSAMIKO

Effectiveness 3.90 0.858 3.80 71.896 Efficiency 3.60 0.900 Satisfaction 3.73 0.862 Innovation 4.00 0.787 POTTERY

Effectiveness 4.86 0.986 4.83 87.500 Efficiency 4.71 0.764 Satisfaction 4.86 0.986 Innovation 4.86 0.986 SALSA

Effectiveness 4.64 0.724 4.71 54.519 Efficiency 4.73 0.859 Satisfaction 4.73 0.678 Innovation 4.73 0.678 WALLOON

Effectiveness 3.90 0.901 3.95 72.993 Efficiency 3.80 0.944 Satisfaction 3.90 0.867 Innovation 4.30 0.681 HBB

Effectiveness 3.17 0.463 3.55 64.428 Efficiency 3.50 0.890 Satisfaction 3.67 0.908 Innovation 3.67 0.865 BYZANTINE

Effectiveness 3.68 0.912 3.78 64.498 Efficiency 3.60 0.588 Satisfaction 3.72 0.881 Innovation 4.08 0.803 CANTO A TENORE

Presentations 3.36 0.896 3.50 87.388 Practical activities

3.64 0.905

Discussions 3.57 0.980 Tests 3.43 0.956 CANTU IN PAGHJELLA Understanding 3.85 0.920 3.68 69.893 Learning 3.85 0.804 Handling 3.31 0.777 GENERIC FRAMEWORK



Effectiveness 3.70 0.997 3.77 86.673 Efficiency 4.00 0.830 Satisfaction 3.40 0.931 Innovation 4.00 0.958 GENERIC-PRESERVATION & TRANSMISSION OF ICH

Contribution 3.80 0.974 3.59 70.605 Adoption 3.60 0.937 Usefulness 3.40 0.949 Recommendation 3.60 0.937 Transmission 4.00 0.807 Spontaneous 3.00 -0.223 Respects Tradition

3.40 0.791

Innovative 3.20 0.845

8.6 Limitations and further research

In order for our findings to be understood in context, it is important that we acknowledge the limitations of the present study. First, the data was collected using questionnaires at a single point in time. As a result, we could not rule out the possibility of spurious correlations between the dependent and independent variables and thus, the study does not allow for dynamic causal inferences [3]. Future research would benefit by employing longitudinal data. Second, in aggregating the data and performing analyses, the sample sizes used were in some cases very small. Future research should aim for larger sample sizes for this purpose. Third, all variables were self-reported, giving rise to concerns about common method bias. Although data were collected using multiple respondents, this does not necessarily completely eliminates this source of bias. To minimize possible common bias concerns, future research would benefit by gathering data from independent sources [13]. Fourth, a potential limitation of the study concerns the measurement of some latent variables with few observable variables that may restrict the validity of the construct [11]. Future research should give emphasis in the validity of constructs by including most observable variables that reflect the relevant latent variables.

8.7 Conclusions

The purpose of this study was to evaluate the i-treasures system. To the best of our knowledge similar studies in evaluating performances focus on the statistical description of variables based on respondents perceptions. However, these methods suffer from misspecification problems because they do not incorporate in the analysis the possible influences beween the variables involved [12]. Despite the limitations presented above, this study contributes to the evaluation methodologies by having proposed (and applied) the following work steps: 1. Evaluations should start by explaining the possible causality links that may exist

between the variables involved. These links should be supported with theory in

order to build the research hypotheses of the study to be tested.

2. Operational models should be developed that include all the research

hypotheses. These models will constitute the backbone of the analysis. In fact,

according to the operational models the contribution of each variable on the

ultimate dependent variable (e.g., performance) can be traced.



3. The operational model should make special reference to the mediating and/or the

moderating variables in the relationships between the variables involved. This will

minimize any misspecification problems.

4. The context within which the evaluation exercise is taking place should be

presented and explained. Accordingly, the evaluation will examine whether the

findings and conclusions will be “universalistic”, which support the view that the

system maximizes its performance irrespectively to the specific context within

which it is operating, or “contingency”, which support the view that the system

maximizes its performance according to the specific context within which it is

operating [6].

5. Following the operational model experts should develop a structured

questionnaire trying to reflect as good as possible all observable variables (items

in the questionnaire) that constitute each construct (or sub-construct).

6. The validity and reliability of all constructs, as well as the normality and

aggregation properties, should be investigated before estimation of the

operational model.

7. Structural equation modeling should be followed in estimating the operational

model. The results should be evaluated according to statistical criteria and

hypotheses.

8. The contribution of all constructs and sub-contructs on the ultimate construct

should be computed.

9. In ranking the changes / amendments of the components of the system, we

should take into consideration whether there is “room for improvement” and

whether there is “high contribution”. This can be achieved by considering the

proposed general rule, which accordingly determines a “component ranking

index”: “The lower the ratio of the mean of a component by its loading is, the

higher the priority for changes / amendments for this component”.

10. Any amendment should take into consideration the cost-benefit analysis of each

change.

Following these proposed 10 steps the intermediate overall evaluation (see D7.2 – March 2015), concluded that in a 5-level scale (i.e., ‘1’=very poor, ‘2’=poor, ‘3’=average, ‘4’=good, ‘5’=very good) the performance of the i-Treasures system was equal to 3.67. Accordingly, and following the “component ranking index” for priority improvements, it was advised which components / dimensions / items should be amended. In fact, the following guidelines for updating were proposed:

MACRO-LEVEL: Define the specific components in the Web-Platform and the use-cases that must be updated.



MESO-LEVEL: Define the specific dimensions that must be updated in each component in the Web-Platform and the use-cases.

MICRO-LEVEL: Define the specific items that must be updated in each dimension in the Web-Platform and the use-cases

Two years later, the current evaluation concluded that the overall performance of i-Treasures, is equal to ‘3.95’. This means that the current performance of i-Treasures is at the level of ‘good’, and compared to the intermediate evaluation it has been improved by 7.6 percent.

Overall, it is concluded that the current study contributed on three dimensions:

1. It has proposed an innovative and precise evaluation method, thus making a

theoretical contribution.

2. It has proved that systems like the i-Treasures belong into the so-called

contingency systems, thus contributing to the relevant knowledge.

3. The i-Treasures system is perceived by individuals to be of high quality (score

4.15 in Table 37) in terms of Web-Preservation & Transmission of ICH, thus

contributing at an empirical level.



References

[1] Bliese, P.D. (2000) Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In: Multilevel theory, research, and methods in organizations (pp. 349-382). K. J. Klein, & S. Kozlowski (Eds.). San Francisco, CA: Jossey-Bass.

[2] Bollen, K.A. (1989) Structural Equations with Latent Variables. New York: Wiley.

[3] Cavanaugh, M.A., and Noe, R.A. (1999). Antecedents and consequences of relational components of the new psychological contract. Journal of Organizational Behavior, 20, 323-340.

[4] Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.

[5] Dagnino, F.M., Hadjileontiadis, L.J, Ott, M., and Pozzi, F. (2014). An integrated platform supporting intangible cultural heritage learning and transmission: Definition of Requirements and evaluation criteria. Journal of Computing and Information Technology, 22(4): 277-292)

[6] Delery, J. and Doty, D.H. (1996) Modes of theorizing in strategic human resource management: test of universalistic, contingency and configurational performance predictions. Academy of Management Journal, 39, 802-835.

[7] Hair, F., Anderson, R., Tatham, R. and Black, W. (2008), Multivariate Data Analysis with Readings, Prentice-Hall, London.

[8] Jöreskog, K.G., & Sörbom, D. (2004) LISREL 8.7 for Windows [Computer Software]. Lincolnwood, IL: Scientific Software International, Inc.

[9] Katou, A.A., & Budhwar, P.S. (2010) Causal relationship between HRM policies and organisational performance: Evidence from the Greek manufacturing sector. European Management Journal, 28, 25-39.

[10] Kline, R.B. (1998) Principles and practice of structural equation modeling. NY: Guilford Press.

[11] Luna-Arocas, R., and Camps, J. (2008). A model of high performance work practices and turnover intentions. Personnel Review, 37, 26–46.

[12] MacKenzie, S.B., Podsakoff, P.M., & Jarvis, C.B. (2005). The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90(4), 710-730.

[13] Podsakoff, P.M., and Organ, D. (1986). Self-reports in organization research: problems and prospects. Journal of Management, 12, 531-544.

APPENDIX I .............................................................................................................. 3

9.1 The web platform and use cases evaluation protocol .......................................... 3

9.2 The technical assessment protocol...................................................................... 4

9.3 Further Details on Tasks 7.3 and 7.4 .................................................................. 6

APPENDIX II ........................................................................................................... 10

10.1 Questionnaire for the evaluation of the Web Platform (in English) ................... 10

10.2 Questionnaire for the evaluation of the Web Platform (in Greek) ..................... 15

APPENDIX III .......................................................................................................... 20

11.1 Questionnaire for the evaluation of the Web Platform and Contemporary Music Composition use case (in English) .......................................................................... 20

11.2 Questionnaire for the evaluation of the Web Platform and Contemporary Music Composition use case (in Greek) ............................................................................ 23

APPENDIX IV.......................................................................................................... 27

12.1 Questionnaire for the evaluation of the Web Platform and the Tsamiko Dance use case .................................................................................................................. 27

12.2 Questionnaire for the evaluation of the Web Platform and the Tsamiko Dance use case (in Greek) ................................................................................................. 32

APPENDIX V........................................................................................................... 37

13.1 Questionnaire for the evaluation of the Web Platform and the Pottery use case (in English) .............................................................................................................. 37

13.2 Questionnaire for the evaluation of the Web Platform and the Pottery use case (in Greek) ................................................................................................................ 42

APPENDIX VI.......................................................................................................... 47

14.1 Questionnaire for the evaluation of the Web Platform and the Salsa Dance use case (in English) ...................................................................................................... 47

14.2 Questionnaire for the evaluation of the Web Platform and the Salsa Dance use case (in Greek) ........................................................................................................ 52

APPENDIX VII ......................................................................................................... 57

15.1 Questionnaire for the evaluation of the Web Platform and the Byzantine music use case (in English) ............................................................................................... 57

15.2 Questionnaire for the evaluation of the Web Platform and the Byzantine music use case (in Greek) ................................................................................................. 60

APPENDIX VIII ........................................................................................................ 63

16.1 Questionnaire for the evaluation of the Web Platform and the Human Beat-Box use case (in French)................................................................................................ 63

APPENDIX IX.......................................................................................................... 71

17.1 Questionnaire for the evaluation of the Web Platform and Cantu in Paghjella use case 71

APPENDIX X........................................................................................................... 78

18.1 Questionnaire for the evaluation of the Web Platform and the Canto a Tenore use case .................................................................................................................. 78

APPENDIX XI.......................................................................................................... 82

19.1 Questionnaire for the evaluation of Walloon Game (in English) ....................... 82

APPENDIX XII ........................................................................................................ 86

20.1 Questionnaire for for the evaluation of the ITGD tool and the Generic Game Framework (in Greek) ............................................................................................. 86

APPENDIX XIII ........................................................................................................ 93

21. Questionnaire for the for the Overall Assessment of the System (Based on Non-Functional Requirements) ....................................................................................... 93

APPENDIX XsIV ..................................................................................................... 98

22. Questionnaire for the Assessment of the Main Technical Indices ...................... 98

(Based on Functional Requirements) ...................................................................... 98

APPENDIX XV ...................................................................................................... 102

23. Final Technical assessment reports ................................................................ 102

APPENDIX I

9.1 The web platform and use cases evaluation protocol

The evaluation protocol, which has been sent to all partners, has specific steps that are described below:

PREPARATION

Use-case leaders should establish meaningful relations with local communities. This is necessary considering that members of these communities will take part in the first evaluation (Phase A) and in final evaluations (Phase B).

Example: In evaluating (last year) the “web platform” and the “contemporary music composition - IMI” activities, UOM and AUTH used members of the following communities:

1. Students of the Applied Informatics Department of UOM 2. Students of the Music Department of AUTH 3. Students of the State Conservatory of Thessaloniki

All users should have the appropriate/necessary equipment: PCs, internet connection and sensors and cameras according to the use-case.

Accounts will be already pre-created for all the users. The accounts will have the form (for the contemporary music composition): cmc1, cmc2, cmc3,, cmce1, cmce2,… and so on. Cmc1, cmc2 etc. will be used by the students, while cmce1, cmce2 and so on will be used by the corresponding experts. The same applies for Tsamiko (ts1,ts2,…, tse1,tse2 etc.), pottery (pot1, pot2, …, pote1, pote2 etc.), canto a tenore (cat1,cat2,…, cate1,cate2 etc.), HBB (hbb1, hbb2,…, hbbe1, hbbe2 etc.) and this goes for all the sub-use cases that will be evaluated. The password of each account is the same as the username.

It is necessary to write down which user gets which account, in case we are able to re-evaluate his/her performance again in a future workshop.

Regarding account creating, please send information about the number of accounts needed to [email protected] 3 days before the demonstration date.

EVALUATION WORKSHOPS

Example: The steps used in evaluating the “web platform” and the “contemporary music composition - IMI” use-case are the following:

Introduction

1. Presentation of i-Treasures, using the official video of the project.

http://i-treasures.multimedia.uom.gr/itreasures_short_720.mp4

Web-Platform

2. Demonstration (using

http://i-treasures.multimedia.uom.gr/WebPlatformVideo.wmv), and explaining all or most of the possible tasks and activities included in the web-platform.

3. Demonstration (using a short video) of the specific use-case (e.g., contemporary music composition – IMI video is

http://www.euronews.com/2016/07/11/european-digital-database-aims-to-keep-cultural-traditions-alive).

4. Use of the web-platform by the participants (experts and simple users), according to a specific evaluation scenario that will be provided (and using their personal accounts). It must be clear that all participants must use and get involved with all the processes included in the web-platform.

5. Having followed a specific course from the “Learning Path” of the LMS, the participants must accordingly use the 3D virtual environment (e.g., music game) – with the same user account - in which there are two phases (observe and practice). In the practice phase, for example, participants use the IMI by performing expert’s gestures and at the end, the participant can see his/her score. Then, there is another activity called Final Challenge, in which previous activities are combined.

mailto:[email protected]


http://i-treasures.multimedia.uom.gr/WebPlatformVideo.wmv

http://www.euronews.com/2016/07/11/european-digital-database-aims-to-keep-cultural-traditions-alive


Use of the Web-platform and the use-case e-Questionnaires

6. Explanation of the questionnaire to be used in evaluating the web-platform and in the evaluation of the contemporary music composition – IMI use case.

7. Completion of the web-platform and the contemporary music composition use case - IMI evaluation questionnaire.

Attention: The field name user id must be filled with the username of the user that was used

by the participant to browse the web platform e.g. cmc1.

Sample sizes

Sample sizes should be as large as possible. However, adequate sample sizes are as follows:

Experts: 2-5 individuals

Simple users (learners): 40-50 individuals

Use of Interviews

The questionnaires used in evaluation are completely structured, including no open questions. This will help the statistical analysis. However, although experts are included between the individuals who completed the questionnaires (their answers will be used as controls in the evaluation process), it is advisable to move to interviews. These interviews are of the free format trying to answer questions that may not be included in the questionnaire. Two groups of interviewees will be used:

1. Experts: 2-5 individuals

2. Focus groups: 5-8 individuals. The members of the focus groups should be selected from the

simple users used to complete the questionnaires.

QUESTIONNAIRE DATA PROCESSES

All questionnaires will be prepared in English and will be translated by the leaders into the mother tongue. Questionnaires will be uploaded in LimeSurvey tool available at:

http://i-treasures.multimedia.uom.gr/survey2/index.php/admin/authentication/sa/login

List of available questionnaires for participants will be available at:


Limesurvey accounts and information about using the survey tool you can contact UOM at [email protected]

TIPS

All data will be sent to UOM (Prof. A. Katos, [email protected]) for analysis.

In case that the sample size is short, there is no need to collect data through an on line survey

method. It is advised the respondents to fill in the questionnaires on paper. But, in this case

the collected data should be sent to UOM in an Excel format.

9.2 The technical assessment protocol

The evaluation protocol, which has been sent to all partners, has specific steps that are described below:

PREPARATION

Use-case leaders should establish meaningful relations with experts, in order experts to assess the system functionalities, in terms of complying with the user requirements and expectations.

Example: In evaluating the functionalities of “web platform” and the “contemporary music composition - IMI” activities (i.e. performance, time, etc.), UOM and AUTH is using members of the following communities:

4. Students and Professors of the Applied Informatics Department of UOM 5. Students and Professors of the Music Department of AUTH (Master)

It is kindly requested to each use-case leader to propose at least 2 experts with advanced knowledge in informatics for the case of technical assessment based on Functional as well as non –Functional





Requirements. Previous working experience or studies on ICH domains will be considered as a plus. Considering that questionnaires require special knowledge in fields like Facial analysis, ontology engineering etc, it is advised that experts complete parts of questionnaires related to their expertise field, than completing the entire questionnaire.

All experts should have the appropriate/necessary equipment: PCs, internet connection and sensors and cameras according to the use-case.

Accounts will be already pre-created for all the users. The accounts will be granted with teacher permissions and will have the form (for the contemporary music composition): ta-cmce1, ta-cmce2, and so on. The same applies for Tsamiko (ta-tse1,ta-tse2,…, etc.), pottery (ta-pote1, ta-pote2 etc.), HBB (ta-hbbe1, ta-hbbe2 etc.) and this goes for all the sub-use cases that will be evaluated. The password of each account is the same as the username.

It is necessary to write down which user gets which account, in case we are able to re-evaluate his/her performance again in a future workshop.

Regarding account creating, please send information about the number of accounts needed to [email protected] 3 days before the demonstration date.

ASSESSMENT WORKSHOPS

Example: The steps which will be used in evaluating the “web platform” and the “contemporary music

composition - IMI” use-case are the following:

Introduction

8. Presentation of i-Treasures, using the official video of the project.


Web-Platform

9. Demonstration (using

http://i-treasures.multimedia.uom.gr/WebPlatformVideo.wmv), and explaining all or most of the possible tasks and activities included in the web-platform.

10. Demonstration (using a short video) of the specific use-case (e.g., contemporary music composition – IMI video is


11. It must be clear that all participants must use and get involved with all the processes included in the web-platform (Learning Management System, TTS, Repository Search) and in 3D Sensorimotor Learning suite of game-like applications. Experts will have to pay special attention to all functionalities in the game that corresponds to the specific sub-case.

Use of the Web-platform and the use-case e-Questionnaires

12. Explanation of the questionnaire to be used in evaluating Main Technical Indices and Overall Technical Assessment.

13. Completion of the questionnaires mentioned above.

Attention: The field name user id must be filled with the username of the user that was used

by the participant to browse the web platform e.g. fg-cmce1.

Sample sizes

Adequate sample sizes are as follows:

Questionnaire for Main Technical Assessment Indices: at least 2 experts

Questionnaire for Overall Technical Assessment of System: at least 2 experts

QUESTIONNAIRE DATA PROCESSES

All questionnaires will be prepared in English and will be translated by the leaders into the mother tongue. Questionnaires will be uploaded in LimeSurvey tool available at:


List of available questionnaires for participants will be available at:








Limesurvey accounts and information about using the survey tool you can contact UOM at [email protected]

TIPS





9.3 Further Details on Tasks 7.3 and 7.4

Task 7.3 – Case studies evaluation *

In Table 1, all available questionnaires for Task 7.3 evaluation are listed. Third column indicates responsible Use Case Leader.

For your convienence, there is a page in i-Treasures platform where all questionnaires, protocols and other information are available.


Please pay attention to page 6 for more instructions about evaluation results data.

Table 1

Questionnaires Download

File (*.doc)

Use Case Leader

Evaluation of the Web Platform and Contemporary Music Composition use case

download UoM

Evaluation of the Web Platform and the Byzantine music use case

download UoM

Evaluation of the Web Platform and the Tsamiko Dance use case

download CERTH

Evaluation of the Web Platform and the Pottery use case download CERTH

Evaluation of the Web Platform and the Calus Dance use case

download CERTH

Evaluation of the Web Platform and the Salsa Dance use case

download CERTH

Evaluation of the Web Platform and the Human Beat Box (HBB) use case

download CNRS/ UPMC

* PREREQUISITE : The games have to be finalized by TT

Preparation

1. Make sure during workshops, all appropriate/necessary equipment is available: PCs, internet

connection and sensors and cameras according to the use-case.

2. Make a list of participants (students , experts) and send it to [email protected], at least 3 days

before the planned workshop in order to create accounts.

3. Get the list of accounts and give them to participants.

4. Download and print questionnaires or visit online tool. Available links for downloading or online

questionnaires (Table 1)

Proposed demonstrations scenario





http://i-treasures.multimedia.uom.gr/survey2/index.php/716693/lang-en


http://i-treasures.multimedia.uom.gr/evaluation/cmc_en.doc



http://i-treasures.multimedia.uom.gr/evaluation/byz_en.doc



http://i-treasures.multimedia.uom.gr/evaluation/tsamiko.doc


http://i-treasures.multimedia.uom.gr/evaluation/pottery.doc



http://i-treasures.multimedia.uom.gr/evaluation/calus.doc



http://i-treasures.multimedia.uom.gr/evaluation/salsa.doc



http://i-treasures.multimedia.uom.gr/evaluation/hbb.doc


1. Introduction to i-Treasures project, using the official video:


2. Demonstration of web platform functionalities

3. (using http://i-treasures.multimedia.uom.gr/WebPlatformVideo.wmv), and explaining all or most of the possible tasks and activities included in the web-platform.

4. Demonstration of a video or powerpoint on 3D Sensorimotor Learning activities of the specific use-case if available (alternatively the following video of Euronews provides some general information:


5. Use case leaders log in platform and demonstrate all available functionalities from teacher/expert side. A recommended scenario is like that:

Expert (teacher)

1. Log in Platform using teacher account 2. Navigate through LMS 3. List available courses 4. Visit My Courses and Select the use case course 5. Present all available tools ( activate/deactivate tools, visit most important tools) 6. Demonstrate and explain Learning Path and the process that it is created (by LMS or

Pedagogical Planner). 7. Visit Pedagogical Planner as a teacher (it is recommended to emphasize on SSO

mechanism). 8. Create a course plan and enter it in LMS. Demonstrate most important tools (Scorm tests

etc) 9. Use TTS: Study How To Guide and then create a ‘song’ 10. Use Repository Search making some queries. Demonstrate medium level features and

high level metadata.

11. Download 3D Game Suite, Log in and demonstrate all functionalities (Tutorial, Observe Mode, Practice Mode). Practice some exercises with connected sensors.

6. Use of the web-platform by the participants (experts and simple users), according to a specific evaluation scenario that will be provided by use-case leaders. It must be clear that all participants must use and get involved with all the processes included in the web-platform under the guidance of use case leader. A recommended scenario is like that:

Learner (simple user)

1. Log in platform using given (simple user) account. 2. Use TTS: Study How To Guide and then try to create a ‘song’ 3. Use Repository Search: Study How to Guide and then make queries. 4. Navigate through LMS 5. Visit Course Catalogue, find the course and subscribe.

6. Visit Course and check available tools 7. Visit Learning path section by section (Read information, see videos, complete tests). 8. Use other available tools (Forum, Chat, Links, etc)

1. Having followed a specific course from the “Learning Path” of the LMS, the participants must accordingly use the 3D virtual environment (e.g., music game) – with the same user account - in which there are two phases (observe and practice). In the practice phase, for example, participants use the IMI by performing expert’s gestures and at the end, the participant can see his/her score. Then, there is another activity called Final Challenge, in which previous activities are combined.

2. Visit Progress tab to see score and analytics

7. Explanation step-by-step of the questionnaire to be used in evaluating the web-platform and

the specific use case before the completion.

8. Completion of the web-platform and the specific use case - evaluation questionnaire.





Task 7.4 - Technical assessment of the system

It is kindly requested to each use-case leader to propose 2-5 experts with advanced knowledge in informatics for the case of technical assessment based on Functional as well as non –Functional Requirements. Considering that these questionnaires require special knowledge in fields like Facial analysis, ontology engineering etc, it is advised that IT experts complete parts of questionnaires related to their expertise field, than completing the entire questionnaire. In the following table (Table 2), we define which

Questionnaire for the Assessment of the Main Technical Indices (Based on Functional

Requirements)

Section: Subject Use case Leader Required

Section 0 RESPONDENT DETAILS ALL partners

Experts from

Use case

leader

institute-

Advanced

knowledge in

informatics

Section 1 FACIAL EXPRESSION ANALYSIS CERTH

Section 2 HUMAN BODY MOTION AND GESTURE

RECOGNITION

CNRS(LPP),

UPMC, ARMINES,

UOM

Section 3 ELECTROENCEPHALOGRAPHY ANALYSIS AUTH

Section 4 VOCAL TRACT SENSING AND MODELLING –

ULTRASOUND ANALYSIS

CNRS(LPP), UPMC

Section 5 SOUND PROCESSING UMONS

Section 6 TEXT-TO-SONG ACAPELA

Section 7 ONTOLOGY ENGINEERING CERTH

Section 8 EDUCATIONAL PLATFORM ALL partners

Section 0 RESPONDENT DETAILS ALL partners

Questionnaire for the Overall Assessment of the System (Based on Non-Functional

Requirements)

Section 0 RESPONDENT DETAILS ALL partners Advanced

knowledge in

informatics.

More

specifically in

Web

Development

Section 1 COST OPTIMALITY / ENERGY EFFICIENCY ALL partners

Section 2 ACCESSIBILITY / USABILITY ALL partners

Section 3 DOCUMENTATION / SUPPORT ALL partners

Section 4 INTEROPERABILITY / PORTABILITY ALL partners

Section 5 EXTENSIBILITY / SCALABILITY ALL partners

Table 2



Preparation

1. Make sure during workshops, all appropriate/necessary equipment is available: PCs, internet

connection and sensors and cameras according to the use-case.

2. Make a list of participants (students , experts) and send it to [email protected], at least 3 days

before the planned workshop in order to create accounts.

3. Get the list of accounts and give them to participants.

4. Download and print questionnaires or visit online tool. Available links for downloading or online

questionnaires on Table 2.

Proposed demonstrations scenario

1. Introduction to i-Treasures project, using the official video:


2. Demonstration of web platform functionalities

3. (using http://i-treasures.multimedia.uom.gr/WebPlatformVideo.wmv), and explaining all or most of the possible tasks and activities included in the web-platform.

4. Demonstration of a video or powerpoint on 3D Sensorimotor Learning activities of the specific use-case if available (alternatively the following video of Euronews provides some general information:


5. Use case leaders log in platform and demonstrate all available functionalities from teacher/expert side. A recommended scenario has already been described in page 2 of this document )

6. It must be clear that all participants must use and get involved with all the processes included in the web-platform (Learning Management System, TTS, Repository Search) and in 3D Sensorimotor Learning suite of game-like applications. Experts will have to pay special attention to all functionalities in the game that corresponds to the specific sub-case.

7. Explanation step-by-step of the questionnaire (or parts of them) to be used in evaluating.

8. Completion of the web-platform and the specific use case - evaluation questionnaire.

Important Information for all questionnaires





For Excel files, please use columns for each question and rows for records. Please name the

column like this:

For example : Section 1 – Question 1 Q11 Section 2 –Questions 1 Sub-question1 Q211 Section 3 Question 2Sub-question 2 Q322 It is kindly requested that you customize and use attached xls file (results_example.xls). Delete or rename columns if not applicable.







APPENDIX II 10.1 Questionnaire for the evaluation of the Web Platform (in English)

(Sections 2-6, 8-9)

QUESTIONNAIRE

SECTION 2: INFORMATION

1. PURPOSE OF WEB PLATFORM

No. Please indicate your level of agreement with each of the following statements

Completely

disagree

Completely

agree

1 2 3 4 5

1 The mission of the web platform is sufficiently stated.

2 The objectives of the web platform are explicitly stated.

2. QUALITIES OF INFORMATIONAL CONTENT


Completely

disagree

Completely

agree

1 2 3 4 5

1 The information provided by the web platform is accurate.

2 The information provided by the web platform is useful in encouraging ongoing learning.

3. LEVEL OF DIFFICULTY OF INFORMATIONAL CONTENT


Completely

disagree

Completely

agree

1 2 3 4 5

1 The information provided is overly complex.

2 The information provided is overly technical.

4. INFORMATIONAL CONTENT ORGANIZATION


Completely

disagree

Completely

agree

1 2 3 4 5

1 The informational content is appropriately organized hierarchically within each thematic

section.

2 The informational content is appropriately organized in terms of multimedia presentation (sound, image, video, animation, and graphics).

SECTION 3: DESIGN

1. ACCESSIBILITY


Completely

disagree

Completely

agree

1 2 3 4 5

1 The web platform can be properly viewed by most modern browsers.

2 The web platform is adequate in providing access control mechanisms (authentication using unique username and password for each user).

3 Information is easily accessed.

2. NAVIGATION


Completely

disagree

Completely

agree

1 2 3 4 5

1 The web platform provides clear navigation mechanisms.

2 The content presentation is consistent across pages.

3 The navigation mechanism supports easily the user to be able to return to a previous page, level, or homepage.

4 It is possible for the user to know where on the web platform is at each time.

3. INTERFACE DESIGN OF THE WEB PLATFORM


Completely

disagree

Completely

agree

1 2 3 4 5

1 The web platform’s interface design is attractive.

2 The colours used at this web platform are comfortable (e.g., hue, haze, brightness, etc).

3 The web platform does not contain features that irritate, such as scrolling or blinking text.

4. INTERACTION


Completely

disagree

Completely

agree

1 2 3 4 5

1 The web platform provides adequate feedback information to the user.

2 It is easy to learn how to use the web platform.

SECTION 4: LEARNING MANAGEMENT SYSTEM (LMS) FUNCTIONALITIES

1. DESIGN


Completely

disagree

Completely

agree

1 2 3 4 5

1 Accessibility within the LMS is easy and

comfortable.

2 Navigation within the LMS is easy and intuitive.

3 Interface within the LMS is attractive.

4 Interaction (feedback) within the LMS is adequate.

2. COURSE PROPERTIES


Completely

disagree

Completely

agree

1 2 3 4 5

1 Organization: The contents provided in the course

are well organized.

2 Clarity: The contents provided in the course are

clear.

3 Completeness: The contents provided in the

course are complete.

4 Adequateness: The contents provided in the

course are unbiased and proper.

3. RESOURCES AND TOOLS


Completely

disagree

Completely

agree

1 2 3 4 5

1 I found presentation tools (e.g., SCORM, Power

Point) easy to use and useful.

2 I found resources (e.g., Audio, Video, text to Song,

Annotated Videos, Links to external resources) easy to use and useful.

4. USABILITY


Completely

disagree

Completely

agree

1 2 3 4 5

1 Ease of understanding: It was easy to understand

activities within the LMS.

2 Ease of learning: It was easy to learn processes

within the LMS.

3 Ease of handling: It was easy to handle tasks

within the LMS.

SECTION 5: SEMANTIC SEARCH FUNCTIONALITIES


Completely

disagree

Completely

agree

1 2 3 4 5

1 It is easy to understand the kind of queries is possible to make in this web platform and the kind of information that can be retrieved.

2 It is easy to understand the meaning of each search filter/criterion and how to use it.

3 It is easy to do a complex query (using several criteria).

4 The interface of the search function is satisfactory.

5 The results of the search and their presentation are satisfactory.

6 In general, is easy to use the search functionality of the web platform.

7 The search functionality offered by the web platform is satisfactory compared to other well-known search platforms.

SECTION 6: TEXT TO SONG (TTS) FUNCTIONALITIES


Completely

disagree

Completely

agree

1 2 3 4 5

1 It is easy to understand how to use the TTS engine.

2 The interface of the TTS engine is satisfactory.

3 The quality of the output of the TTS engine (generated audio file) is satisfactory.

4 The TTS engine is a useful tool for learning more about this kind of singing expression.

5 In general, it is easy to use the TTS engine.

SECTION 8: OVERALL EVALUATION OF WEB PLATFORM

`.COMPLEXITY AND USABILITY


Completely

disagree

Completely

agree

1 2 3 4 5

1 Complexity: The web platform is not complex in

helping me to understand, acquire knowledge, and get information with respect to the issues it is presenting.

2 Usability: It was easy to accomplish activities (e.g.,

basic tasks, recovering a previous situation after a

mistaken choice) of the web platform.

2. PERCEIVED PERFORMANCE


Completely

disagree

Completely

agree

1 2 3 4 5

1 The Web platform is effective (if the web platform

meets its objectives).

2 The Web platform is efficient (if the web platform is using adequate number of sources without exaggerating).

3 The Web platform provides satisfaction (if the web platform provides satisfaction to the user).

4 The Web platform is innovative (if the web platform

is incorporating innovative elements).

SECTION 9: PRESERVATION AND TRANSMISSION OF INTANGIBLE CULTURAL HERITAGE (ICH)


Completely

disagree

Completely

agree

1 2 3 4 5

1 I think that the i-Treasures technologies contribute in the preservation and transmission of ICH.

2 I think that the i-Treasures technologies can be easily used/adopted for the transmission of ICH expressions related to dancing.

3 I think that the i-Treasures platform is a useful tool to be adopted by ICH organizations/schools/ institutions to help them promote endangered ICH expressions.

4 I will recommend the use of the i-Treasures platform to colleagues, friends, and family.

5 I think that is useful to use new technologies for the capture, preservation and transmission of traditional ICH expressions.

6 I am not afraid that the use of new technologies will negatively affect or undermine the spontaneous expressions of ICH forms.

7 I think that the i-Treasures platform respects the tradition of different ICH expressions.

8 I find the i-Treasures platform innovative.

Thank you very much for your co-operation.

10.2 Questionnaire for the evaluation of the Web Platform (in Greek)

ΕΝΟΤΗΤΑ 2: ΠΛΗΡΟΦΟΡΙΑ

1. ΑΝΤΙΚΕΙΜΕΝΟ ΔΙΚΤΥΑΚΟΥ ΤΟΠΟΥ

Αρ. Παρακαλώ σημειώσατε το επίπεδο συμφωνίας σας με κάθε μία από τις

παρακάτω προτάσεις

Καθόλου Πάρα πολύ

1 2 3 4 5

1 Ο σκοπός του δικτυακού τόπου είναι επαρκώς ορισμένος.

2 Οι στόχοι του δικτυακού τόπου είναι σαφώς ορισμένοι.

2. ΠΟΙΟΤΗΤΑ ΠΕΡΙΕΧΟΜΕΝΟΥ ΠΛΗΡΟΦΟΡΙΑΣ




1 2 3 4 5

1 Η πληροφορία που παρέχει ο δικτυακός τόπος είναι ακριβής.

2 Η πληροφορία που παρέχεται είναι χρήσιμη στο να ενθαρρύνει συνεχιζόμενη μάθηση.

3. ΕΠΙΠΕΔΟ ΔΥΣΚΟΛΙΑΣ ΤΟΥ ΠΕΡΙΕΧΟΜΕΝΟΥ ΠΛΗΡΟΦΟΡΙΑΣ




1 2 3 4 5

1 Η πληροφορία που παρέχεται είναι υπερβολικά πολύπλοκη.

2 Η πληροφορία που παρέχεται είναι υπερβολικά τεχνική.

4. ΟΡΓΑΝΩΣΗ ΠΕΡΙΕΧΟΜΕΝΟΥ ΠΛΗΡΟΦΟΡΙΑΣ




1 2 3 4 5

1 Το περιεχόμενο πληροφορίας είναι σωστά και ιεραρχικά οργανωμένο σε θεματικές ενότητες.

2 Το περιεχόμενο πληροφορίας είναι κατάλληλα οργανωμένο ως προς τη πολυμεσική πληροφορία (ήχος, εικόνα,

βίντεο, γραφικά).

ΕΝΟΤΗΤΑ 3: ΣΧΕΔΙΑΣΜΟΣ

1. ΠΡΟΣΒΑΣΗ




1 2 3 4 5

1 Ο δικτυακός τόπος είναι προσβάσιμος από τους περισσότερους φυλλομετρητές.

2 Ο δικτυακός τόπος παρέχει επαρκείς μηχανισμούς ελέγχου πρόσβασης ( η ταυτοποίηση γίνεται με μοναδικό όνομα χρήστη και κωδικό για κάθε χρήστη).

3 Η πληροφορία είναι εύκολα προσβάσιμη.

2. ΠΛΟΗΓΗΣΗ




1 2 3 4 5

1 Ο δικτυακός τόπος παρέχει σαφείς μηχανισμούς πλοήγησης.

2 Η παρουσίαση του δικτυακού τόπου είναι συνεπής μεταξύ των τμημάτων του.

3 Οι μηχανισμοί πλοήγησης επιτρέπουν στο χρήστη τη δυνατότητα να επιστρέψει σε μια προηγούμενη σελίδα, επίπεδο ή στην αρχική σελίδα.

4 Ο χρήστης γνωρίζει κάθε στιγμή σε ποιο σημείο του δικτυακού τόπου βρίσκεται.

3. ΣΧΕΔΙΑΣΜΟΣ ΔΙΕΠΑΦΗΣ ΔΙΚΤΥΑΚΟΥ ΤΟΠΟΥ




1 2 3 4 5

1 Ο σχεδιασμός διεπαφής του δικτυακού τόπου είναι ελκυστικός.

2 Τα χρώματα που χρησιμοποιούνται στον δικτυακό τόπο είναι ξεκούραστα.

3 Ο δικτυακός τόπος δεν περιέχει χαρακτηριστικά που εκνευρίζουν το χρήστη, όπως κύλιση ή κείμενο που αναβοσβήνει.

4. ΑΛΛΗΛΕΠΙΔΡΑΣΗ




1 2 3 4 5

1 Ο δικτυακός τόπος παρέχει επαρκείς μηχανισμούς ανατροφοδότησης στο χρήστη.

2 Είναι εύκολο για το χρήστη να μάθει να χρησιμοποιεί τον δικτυακό τόπο.

ΕΝΟΤΗΤΑ 4: ΛΕΙΤΟΥΡΓΙΕΣ ΤΟΥ ΣΥΣΤΗΜΑΤΟΣ ΔΙΑΧΕΙΡΙΣΗΣ ΜΑΘΗΣΗΣ (ΣΔΜ)

1. ΣΧΕΔΙΑΣΜΟΣ




1 2 3 4 5

1 Η πρόσβασιμότητα στο ΣΔΜ είναι

εύκολη.

2 Η πλοήγηση στο ΣΔΜ είναι εύκολη και

διαισθητική.

3 Η διεπαφή χρήστη του ΣΔΜ είναι

ελκυστική.

4 Η αλληλεπίδραση με το ΣΔΜ είναι

επαρκής.

2. ΙΔΙΟΤΗΤΕΣ ΜΑΘΗΜΑΤΩΝ




1 2 3 4 5

1 Οργάνωση: Το περιεχόμενο των

μαθημάτων είναι κατάλληλα οργανωμένο.

2 Σαφήνεια: Το περιεχόμενο των

μαθημάτων είναι σαφές.

3 Πληρότητα: Το περιεχόμενο των

μαθημάτων είναι πλήρες.

4 Επάρκεια: Το περιεχόμενο των

μαθημάτων είναι αμερόληπτο και κατάλληλο.

3. ΠΗΓΕΣ (RESOURCES) ΚΑΙ ΕΡΓΑΛΕΙΑ




1 2 3 4 5

1 Tα εργαλεία παρουσίασης (SCORM,

Power Point) είναι εύκολα στη χρήση και χρήσιμα.

2 Οι πηγές υλικού (ήχος, βίντεο, σύνδεσμοι

σε εξωτερικές αναφορές) είναι εύκολες στη χρήση και χρήσιμες.

4. ΧΡΗΣΤΙΚΟΤΗΤΑ




1 2 3 4 5

1 Ευκολία στην κατανόηση: Ήταν εύκολο

να κατανοήσω τις δραστηριότητες στο ΣΔΜ.

2 Ευκολία μάθησης: Ηταν εύκολο να μάθω

τις διαδικασίες στο ΣΔΜ.

3 Ευκολία χειρισμού: Ήταν εύκολο να

ολοκληρώσω καθήκοντα στο ΣΔΜ.

ΕΝΟΤΗΤΑ 5: ΛΕΙΤΟΥΡΓΙΕΣ ΤΗΣ ΣΗΜΑΣΙΟΛΟΓΙΚΗΣ ΑΝΑΖΗΤΗΣΗΣ

Αρ. Παρακαλώ σημειώσατε το επίπεδο συμφωνίας σας με κάθε μία από τις παρακάτω προτάσεις


1 2 3 4 5

1 Είναι εύκολο να κατανοήσεις το είδος των ερωτημάτων που μπορείς να κάνεις στο δικτυακό τόπο και το είδος της πληροφορίας που μπορείς να ανακτήσεις.

2 Είναι εύκολο να καταλάβεις τη σημασία κάθε φίλτρου/ κριτήριου αναζήτησης και πώς να το χρησιμοποιήσεις.

3 Είναι εύκολο να κάνεις ένα σύνθετο ερώτημα (χρησιμοποιώντας πολλά κριτήρια).

4 Η διεπαφή χρήστη της αναζήτησης είναι ικανοποιητική.

5 Τα αποτελέσματα της αναζήτησης και η παρουσίαση τους είναι ικανοποιητικά.

6 Γενικά είναι εύκολο να χρησιμοποιήσεις τη λειτουργία αναζήτησης στο δικτυακό τόπο.

7 Η λειτουργία αναζήτησης που παρέχεται από τον δικτυακό τόπο είναι ικανοποιητική σε σχέση με άλλους γνωστούς δικτυακούς τόπους αναζήτησης.

ΕΝΟΤΗΤΑ 6: ΛΕΙΤΟΥΡΓΙΕΣ ΤΗΣ ΕΦΑΡΜΟΓΗΣ ΣΥΝΘΕΣΗΣ ΚΕΙΜΕΝΟΥ ΣΕ ΤΡΑΓΟΥΔΙ



1 2 3 4 5

1 Είναι εύκολα κατανοητό πώς να χρησιμοποιήσεις την εφαρμογή Σύνθεση Κειμένου σε Τραγούδι.

2 Η διεπαφή χρήστη της εφαρμογής Σύνθεση Κειμένου σε Τραγούδι είναι ικανοποιητική

3 Η ποιότητα του παραγόμενου αρχείου ήχου είναι ικανοποιητική.

4 Η εφαρμογή Σύνθεση Κειμένου σε Τραγούδι είναι ένα χρήσιμο εργαλείο για την περαιτέρω μάθηση σχετικά με το είδος των εκφράσεων τραγουδιού..

5 Γενικά, η εφαρμογή Σύνθεση Κειμένου σε Τραγούδι είναι εύκολη στη χρήση.

ΕΝΟΤΗΤΑ 8: ΣΥΝΟΛΙΚΗ ΑΞΙΟΛΟΓΗΣΗ ΤΟΥ ΔΙΚΤΥΑΚΟΥ ΤΟΠΟΥ

1. ΠΟΛΥΠΛΟΚΟΤΗΤΑ ΚΑΙ ΧΡΗΣΤΙΚΟΤΗΤΑ



Διαφωνώ

απόλυτα

Συμφωνώ απόλυτα

1 2 3 4 5

1 Πολυπλοκότητα: Η διαδικτυακή

πλατφόρμα δεν υιοθετεί πολύπλοκους τρόπους για να με βοηθήσει να κατανοήσω, να αποκτήσω νέα γνώση και να λάβω πληροφορίες σε σχέση με τα θέματα που παρουσιάζει.

2 Ευχρηστία: Ήταν εύκολο να ολοκληρώσω

δραστηριότητες (π.χ. βασικές ενέργειες, ανάκτηση μιας προηγούμενης κατάστασης μετά από μια λανθασμένη) στην διαδικτυακή πλατφόρμα.

2. ΑΝΤΙΛΗΠΤΗ ΕΠΙΔΟΣΗ

Αρ. Παρακαλώ σημειώσατε το επίπεδο συμφωνίας σας Διαφωνώ Συμφωνώ

με κάθε μία από τις παρακάτω προτάσεις απόλυτα απόλυτα

1 2 3 4 5

1 Αποτελεσματικότητα (αν η διαδικτυακή

πλατφόρμα πληροί τους σκοπούς της)

2 Αποδοτικότητα (αν η διαδικτυακή πλατφόρμα

κάνει χρήση ικανοποιητικού αριθμού πηγών χωρίς να υπερβάλλει)

3 Ικανοποίηση (αν η διαδικτυακή πλατφόρμα

παρέχει ικανοποίηση στο χρήστη)

4 Καινοτομία (αν η διαδικτυακή πλατφόρμα

ενσωματώνει καινοτόμα στοιχεία).

ΕΝΟΤΗΤΑ 9: ΔΙΑΤΗΡΗΣΗ ΚΑΙ ΔΙΑΔΟΣΗ ΤΗΣ ΑΥΛΗΣ ΠΟΛΙΤΙΣΤΙΚΗΣ ΠΑΡΑΔΟΣΗΣ


Διαφωνώ

απόλυτα


1 2 3 4 5

1 Πιστεύω ότι οι τεχνολογίες του i-Treasures συμβάλλουν στη διατήρηση και διάδοση της πολιτιστικής παράδοσης (ICH ).

2 Πιστεύω ότι οι τεχνολογίες του i-Treasures μπορούν να υιοθετηθούν και εφαρμοστούν εύκολα για την διάδοση των εκφράσεων ICH που σχετίζονται με το χορό

3 Πιστεύω ότι η πλατφόρμα του i-Treasures είναι ένα χρήσιμο εργαλείο προς αξιοποίηση από οργανισμούς/σχολεία/ινστιτούτα ICH για την ενίσχυση της προώθησης εκφράσεων ICH που βρίσκονται υπό εξαφάνιση.

4 Θα προτείνω σε συναδέλφους, φίλους και οικογένεια τη χρήση της πλατφόρμας i-Treasures.

5 Πιστεύω ότι είναι ωφέλιμη η χρήση νέων τεχνολογιών, για την συλλογή διατήρηση και μετάδοση παραδοσιακών ICH.

6 Δε φοβάμαι ότι η χρήση νέων τεχνολογιών θα επηρεάσει αρνητικά ή θα υπονομεύσει τις αυθόρμητες εκφράσεις των μορφών ICH.

7 Πιστεύω ότι η πλατφόρμα του i-Treasures σέβεται τις παραδόσεις των διαφόρων εκφράσεων ICH.

8 Θεωρώ ότι η πλατφόρμα i-Treasures είναι πρωτοποριακή.

APPENDIX III 11.1 Questionnaire for the evaluation of the Web Platform and Contemporary Music

Composition use case (in English)

QUESTIONNAIRE

SECTION 1: RESPONDENT DETAILS

1. Gender: Male 1 Female 2

2. Age (in years): ............................................

.3. Educational level (tick the highest):

Primary Secondary College University Post Grad

1 2 3 4 5

4. Familiarity in using computers:

Very poor Not much Adequate Good Very good

1 2 3 4 5

5. Music Literacy (specifically piano):

Not at all Not much Adequate Good Very good

1 2 3 4 5

5.1 Familiarity with classical music (e.g., Beethoven’s sonatas, Bach’s fugues):


1 2 3 4 5

5.2 Familiarity with contemporary art music (e.g., music of the 20th

and 21st century, such as the

works of Stravinsky, Messiaen, Cage, Bouzel, Xenakis, etc.):


1 2 3 4 5

6. Respondent’s ID: Expert 1 Simple user 2

7. User ID: ........................................................

8. Date: ........................................................

9. Place: ........................................................

SECTION 2-6: (Appendix II)

SECTION 7: 3D ENVIRONMENT SML FUNCTIONALITIES

(USE CASE: CONTEMPORARY MUSIC COMPOSITION)

1. QUALITY CHARACTERISTICS OF CMC GAME


Completely

disagree

Completely

agree

1 2 3 4 5

1 I have clearly understood the objectives of the

game.

2 I found the game scenario clear and satisfactory.

3 I have easily understood what I have to do in each activity of the game.

4 The “Getting started” section was informative.

5 I found helpful the instructions offered by the virtual tutor.

6 I found pleasant the 3D game environment in

terms of design and aesthetics.

7 I found satisfactory the appearance of the expert and learner avatars.

8 I found accurate the visualization of the musician’s movements by the avatars.

9 I found helpful to watch the virtual avatar of the expert in order to learn the basic musical gestures.

10 I found helpful to watch the video of the expert in

order to learn the basic musical gestures.

11 The virtual avatar helped me (visual feedback) to

improve/correct my own performance.

12 I found comfortable to practice the musical

gestures and watch the virtual avatar at the same time.

13 It was easy to navigate through the 3D virtual

environments and change the position of the virtual camera (zoom in/out, rotate virtual avatar).

14 I found the existence of several windows in the screen not to be confusing and annoying.

15 I found the general feedback provided by the

platform satisfactory.

2. QUALITY CHARACTERISTICS OF INTANGIBLE MUSICAL INSTRUMENT


Completely

disagree

Completely

agree

1 2 3 4 5

1 It is useful because it helps me to learn how to

perform musical gestures and feel the correct emotion.

2 It is easy to place my hands correctly & start

performing.

3 It is simple to use in terms of performing

/reproducing musical gestures.

4 It is user friendly.

5 It is flexible, as it gives me the ability to add my

own audio files and to train the system with my own musical gestures and emotional status.

6 I easily remember how to perform/reproduce

musical gestures.

7 It is easy to learn to perform/reproduce musical

gestures.

3. LEARNING EXPERIENCE


Completely

disagree

Completely

agree

1 2 3 4 5

1 The visualization of the recognition results

(evaluation score) in real time helps me to improve/correct my gesture quickly and easily.

2 The resynthesized sound that I produced in real

time helps me to improve/correct my gesture quickly and easily (audio feedback).

3 The resynthesized sound is fluid.

4 The resynthesized sound is immediate.

5 The images that have been shown to me excite me

emotionally.

6 The time was enough for me to follow the

emotional state at every level of the game.

7 The colors of emotions were representative.

8 I did not have to perform some activity many times in order understand it.

9 I did not find the game stressful in any way.

10 I had fun playing the game.

11 I found the “Final Challenge” activity satisfactory.

12 The final challenge activity motivated me to focus

more on learning how to perform musical gestures.

13 I think that this game makes the musical learning experience easier compared to traditional techniques (e.g., observing a teacher live).

14 I think that this game makes the musical learning experience more enjoyable compared to traditional techniques (e.g., observing a teacher live).

4. PERCEIVED PERFORMANCE OF CMC AND INTANGIBLE MUSICAL INSTRUMENT


Completely

disagree

Completely

agree

1 2 3 4 5

1 IMI is effective (if IMI meets its objectives).

2 IMI is efficient (if IMI responses satisfactorily and in a short time).

3 IMI provides satisfaction (if IMI provides satisfaction to the user).

4 IMI is innovative (if IMI offers novel tools/

techniques in ICH transmission).

SECTION 8 - 9: APPENDIX I

11.2 Questionnaire for the evaluation of the Web Platform and Contemporary Music Composition use case (in Greek)

ΕΡΩΤΗΜΑΤΟΛΟΓΙΟ

ΕΝΟΤΗΤΑ 1: ΣΤΟΙΧΕΙΑ ΕΡΩΤΩΜΕΝΟΥ

1. Γένος: Άνδρας 1 Γυναίκα 2

2. Ηλικία (σε χρόνια): .............................................

3. Επίπεδο εκπαίδευσης (σημειώστε το υψηλότερο):

Βασική Γυμνάσιο / Λύκειο

ΙΕΚ Ανώτατη

(ΑΕΙ/ΤΕΙ)

Μετα-πτυχιακή

1 2 3 4 5

4. Αντιληπτή παιδεία Η/Υ:

Πολύ φτωχή

Χαμηλή Μέτρια Αρκετή Πάρα πολύ

1 2 3 4 5

5. Αντιληπτή παιδεία μουσικής (στην προκειμένη περίπτωση πιάνο):

Καθόλου Χαμηλή Μέτρια Αρκετή Πάρα πολύ

1 2 3 4 5

5.1 Εξοικείωση με την κλασική μουσική (π.χ., σονάτες Beethoven, φούγκες Bach):


1 2 3 4 5

6. Ταυτότητα ερωτώμενου: Εμπειρογνώμονας 1 Απλός χρήστης 2

7. User ID:

8. Ημερομηνία: ........................................................

9. Τόπος: ........................................................

ΕΝΟΤΗΤΑ 2-6 : (Appendix II)

ΕΝΟΤΗΤΑ 7: ΛΕΙΤΟΥΡΓΙΕΣ 3D ΑΙΣΘΗΤΗΡΙΟΚΙΝΗΤΙΚΗΣ ΜΑΘΗΣΗΣ

( ΠΕΡΙΠΤΩΣΗ : ΣΥΝΘΕΣΗ ΣΥΓΧΡΟΝΗΣ ΜΟΥΣΙΚΗΣ ΣΣΜ)

1. ΠΟΙΟΤΙΚΑ ΧΑΡΑΚΤΗΡΙΣΤΙΚΑ ΤΟΥ ΠΑΙΧΝΙΔΙΟΥ ΣΣΜ



1 2 3 4 5

1 Κατανόησα πλήρως τους στόχους του παιχνιδιού.

2 Το σενάριο χρήσης του παιχνιδιού σαφές και ικανοποιητικό.

3 Κατανόησα εύκολα τι έπρεπε να κάνω σε κάθε άσκηση του παιχνιδιού.

4 Η ενότητα Getting Started παρείχε πληροφορίες.

5 Βρήκα χρήσιμες της οδηγίες που παρείχε το εικονικό avatar ειδικού (virtual tutor).

6 Το τρισδιάστατο περιβάλλον του παιχνιδιού είναι ευχάριστο όσον αφορά το σχεδιασμό και την αισθητική

7 Τα εικονικό avatar του ειδικού και του εκπαιδευόμενου έχουν ικανοποιητική εμφάνιση. Η οπτικοποίηση των κινήσεων του μουσικού από τα εικονικά avatar είναι ακριβής.

8 Η οπτικοποίηση των κινήσεων του μουσικού από τα εικονικά avatar είναι ακριβής.

9 Η παρακολούθηση του εικονικού avatar του ειδικού με βοήθησε να μάθω τις βασικές μουσικές χειρονομίες.

10 Το video του ειδικού ήταν χρήσιμο στην εκμάθηση των βασικών μουσικών χειρονομιών.

11 Το εικονικό avatar με βοήθησε (οπτική ανάδραση) να βελτιώσω/διορθώσω την εκτέλεση των χειρονομιών.

12 Μου φάνηκε βολικό να εκτελώ τις μουσικές χειρονομίες παρακολουθώντας ταυτόχρονα το εικονικό avatar.

13 Είναι εύκολη η πλοήγηση στο εικονικό τρισδιάστατο περιβάλλον και η αλλαγή της θέσης της εικονικής

camera.

14 Η ύπαρξη πολλών παραθύρων στην οθόνη δεν με μπέρδεψε / ενόχλησε.

2. ΠΟΙΟΤΙΚΑ ΧΑΡΑΚΤΗΡΙΣΤΙΚΑ ΤΟΥ ΑΥΛΟΥ ΜΟΥΣΙΚΟΥ ΟΡΓΑΝΟΥ (ΑΜΟ)



1 2 3 4 5

1 Είναι χρήσιμο γιατί με βοήθησε να μάθω να εκτελώ μουσικές χειρονομίες και να νιώσω τη συγκίνηση.

2 Είναι εύκολο να τοποθετήσω τα χέρια μου σωστά και να αρχίσω την εκτέλεση.

3 Είναι εύκολο στη χρήση όσον αφορά την εκτέλεση/αναπαραγωγή μουσικών χειρονομιών.

4 Είναι φιλικό προς το χρήστη.

5 Είναι ευέλικτο και μου δίνει τη δυνατότητα να προσθέσω δικά μου αρχεία ήχου και να εκπαιδεύσω το σύστημα με δικές μου μουσικές χειρονομίες και συναισθηματικές καταστάσεις

6 Θυμάμαι εύκολα πώς να τα εκτελώ/αναπαράγω μουσικές χειρονομίες.

7 Είναι εύκολο να μάθω να εκτελώ/αναπαράγω μουσικές χειρονομίες.

8 Διασκέδασα πολύ με το παιχνίδι

9 Νομίζω ότι το αυτό του είδους το παιχνίδι είναι πολύ πιο ενδιαφέρον στο να μάθει κανείς βυζαντινή μουσική από ότι ο παραδοσιακός τρόπος (παρατηρώντας τον ειδικό)

3. ΕΜΠΕΙΡΙΑ ΜΑΘΗΣΗΣ


Πολύ κακή Πολύ καλή

1 2 3 4 5

1 Η οπτικοποιήση των αποτελεσμάτων αναγνώρισης (score) σε πραγματικό χρόνο με βοήθησε να βελτιώσω/διορθώσω τις χειρονομίες μου εύκολα και γρήγορα.

2 Ο ήχος που παρήγαγα σε πραγματικό χρόνο με βοήθησε να βελτιώσω/διορθώσω τις χειρονομίες μου εύκολα και γρήγορα (ηχητική ανάδραση).

3 Ο ανασυντιθέμενος ήχος είναι ρευστός.

4 Ο ανασυντιθέμενος ήχος είναι άμμεσος.

5 Οι εικόνες που μου έδειξαν με διεγείραν συναισθηματικά.

6 Ο χρόνος ήταν επαρκής για να ακολουθήσω τη συναισθηματική κατάσταση σε κάθε επίπεδο του παιχνιδιού.

7 Τα χρώματα των συναισθημάτων είναι

αντιπροσωπευτικά

8 Δεν χρειάστηκε να επαναλάβω πολλές φορές μια άσκηση για να την καταλάβω.

9 Σε καμία περίπτωση το παιχνίδι δεν προκαλεί στρες.

10 Διασκέδασα παίζοντας το παιχνίδι.

11 Η άσκηση “Final Challenge” είναι ικανοποιητική.

12 Η άσκηση “Final Challenge” με βοήθησε να επικεντρωθώ περισσότερο στην εκμάθηση της εκτέλεσης μουσικών χειρονομιών.

13 Θεωρώ ότι το συγκεκριμένο παιχνίδι καθιστά την εμπειρία της μουσικής εκμάθησης ευκολότερη, σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ., παρατήρηση του δασκάλου ζωντανά).

14 Θεωρώ ότι το συγκεκριμένο παιχνίδι καθιστά την εμπειρία της μουσικής εκμάθησης πιο απολαυστική σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ., παρατήρηση του δασκάλου ζωντανά).

4. ΑΝΤΙΛΗΠΤΗ ΕΠΙΔΟΣΗ ΤΟΥ ΠΑΙΧΝΙΔΙΟΥ ΣΣΜ ΚΑΙ ΤΟΥ ΑΥΛΟΥ ΜΟΥΣΙΚΟΥ ΟΡΓΑΝΟΥ (ΑΜΟ)


Διαφωνώ

απόλυτα


1 2 3 4 5

1 Αποτελεσματικότητα (αν το παιχνίδι πληροί τους

στόχους του)

2 Απόδοση (αν το παιχνίδι αποκρίνεται

ικανοποιητικά και σε σύντομο χρονικό διάστημα)

3 Ικανοποίηση (αν το παιχνίδι παρέχει

ικανοποίηση στο χρήστη)

4 Καινοτομία (αν το παιχνίδι προσφέρει νέα

εργαλεία/τεχνικές στην μετάδοση της Άυλης Πολιτιστικής Κληρονομιάς)

ΕΝΟΤΗΤΑ 8 - 9: APPENDIX II

APPENDIX IV

12.1 Questionnaire for the evaluation of the Web Platform and the Tsamiko Dance use case

QUESTIONNAIRE



2. Age (in years): .............................................

3. Educational level (tick the highest):


1 2 3 4 5



1 2 3 4 5

5. Experience in dancing:

Not at all Not much Adequate Good Very Good

1 2 3 4 5

5.1 Experience in dancing Greek traditional dances:


1 2 3 4 5

5.2 Experience in dancing Tsamiko:


1 2 3 4 5


7. User ID: ........................................................

8. Date: ........................................................

9. Place: ........................................................

SECTION 2-6: APPENDIX II

SECTION 7: 3D ENVIRONMENT SML FUNCTIONALITIES (USE CASE: TSAMIKO DANCE)

1. INFORMATIONAL CONTENT FOR TSAMIKO DANCE


Completely

disagree

Completely

agree

1 2 3 4 5

1 The information provided by the web platform for Tsamiko dance is useful.

2 The information provided by the web platform covers different aspects of Tsamiko dance (historical and cultural, geographical, etc).

3 The web-platform provides an adequate number of external links to websites promoting Tsamiko dance.

2. COURSES WITH RESPECT TO TSAMIKO DANCE IN LMS


Completely

disagree

Completely

agree

1 2 3 4 5

1 I have learned more about Tsamiko (cultural background and the dance itself) after concluding this course.

2 The Tsamiko course was appropriate for my age and experience.

3 This course has succeeded to teach me the basics of Tsamiko.

3. INTERACTIONAL CHARACTERISTICS OF THE TSAMIKO DANCE GAME-LIKE APPLICATION

3.1 Game Scenario


Completely

disagree

Completely

agree

1 2 3 4 5


game.



4 I think that the sequence of offered activities (i.e., parts/entities taught independently) helped me to learn how to dance Tsamiko in a progressive and smooth way.



3.2 Visualization


Completely

disagree

Completely

agree

1 2 3 4 5



2 I found satisfactory the appearance of the expert and learner dancer avatars.

3 I found accurate the visualization of the dance movements by the avatars.

4 I found helpful to watch the virtual avatar of the expert in order to learn the basic movements of

Tsamiko.


order to learn the basic movements of Tsamiko.

6 I found helpful to watch the video focusing on the virtual avatar’s legs in order to learn the basic

movements of Tsamiko.

7 The overlay of the virtual avatar and the user ghost image helped me (visual feedback) to

improve/correct my own dancing performance.

8 I found comfortable to practice the dance

movements and watch the virtual avatar at the same time.

3.3 User Performance Evaluation


Completely

disagree

Completely

agree

1 2 3 4 5

1 The evaluation score displayed in the screen

helped me to understand when my movements were not correct and, thus, I improved/corrected my dancing performance.

2 I found the evaluation algorithm used by the game

accurate (i.e., it actually provided a low score when my movements were not correct and a high score when my movements were correct).

3 I had to perform some activity many times because I could not understand what to do.

4 I had to perform some activity many times because the evaluation of my performance was not accurate (i.e., my movements were correct but the evaluation score was low).

3.4 Sensor Set-up


Completely

disagree

Completely

agree

1 2 3 4 5

1 I easily understood where I should stand (in which

area) when practicing the dance movements.

2 It was comfortable to dance within a specific area

so as to be captured by the sensors.

3 The placement of sensors did not cause any

disturbance to me and did not affect my performance.

4 It will be easy for me to handle the sensor(s) by my own (i.e., connect them to the PC and place them in the correct place so as to cover the dance area).

3.5 Usability


Completely

disagree

Completely

agree

1 2 3 4 5

1 It was easy to learn how to play the game.

2 I would be able to learn how to play the game alone (without any external guidance).

3 It was easy to perform the different activities.

4 It was easy to follow the avatars’ movements and practice them myself.




7 I found the general feedback provided by the game

satisfactory.

3.6 Learning Experience


Completely

disagree

Completely

agree

1 2 3 4 5



more on learning how to dance Tsamiko.

3 I think that this game makes the dance learning experience easier compared to traditional techniques (e.g., observing a teacher live).

4 I think that this game makes the dance learning experience more enjoyable compared to traditional techniques (e.g., observing a teacher live).

5 I would use a similar game to learn how to perform some other dance by myself.

6 The game helped me to learn how to dance Tsamiko.



9 I think that the game respects the tradition of Tsamiko dance.

10 I would like to see such a game or similar technologies included in the educational process of my school/organization.

11 I would find it interesting to be able to contest against other individuals for the best dance performance using this game (i.e., support of multi-player scenario).

4. PERCEIVED PERFORMANCE OF THE TSAMIKO GAME-LIKE APPLICATION


Completely

disagree

Completely

agree

1 2 3 4 5

1 The Tsamiko game is effective (if the game meets

its objectives).

2 The Tsamiko game is efficient (if the game responses satisfactorily and in a short time).

3 The Tsamiko game provides satisfaction (if the game provides satisfaction to the user).

4 The Tsamiko game is innovative (if the game

offers novel tools/ techniques in ICH transmission).


12.2 Questionnaire for the evaluation of the Web Platform and the Tsamiko Dance use case (in Greek)



1. Φύλο: Άνδρας 1 Γυναίκα 2

2. Ηλικία (σε χρόνια): .............................................


Βασική Γυμνάσιο/ Λύκειο

ΙΕΚ Ανώτατη (ΑΕΙ/ΤΕΙ)

Μεταπτυχιακή

1 2 3 4 5

4. Εξοικείωση στη χρήση Η/Υ:

Πολύ φτωχή

Όχι αρκετή Μέτρια Καλή Άριστη

1 2 3 4 5

5. Εξοικείωση στον χορό:

Καθόλου Όχι αρκετή Μέτρια Καλή Πολύ καλή

1 2 3 4 5

5.1 Εξοικείωση στον Ελληνικό Πaραδοσιακό Χορό:


1 2 3 4 5

5.2 Εξοικείωση στον Τσάμικο ορό:


1 2 3 4 5


7. Ταυτότητα χρήστη: ........................................................

8. Ημερομηνία: . .......................................................

9. Τόπος: ........................................................

Ενότητα 2-6: APPENDIX II

ΕΝΟΤΗΤΑ 7: ΛΕΙΤΟΥΡΓΙΕΣ ΤΟΥ ΤΡΙΣΔΙΑΣΤΑΤΟΥ ΠΕΡΙΒΑΛΛΟΝΤΟΣ (ΣΕΝΑΡΙΟ ΧΡΗΣΗΣ ΤΣΑΜΙΚΟΥ ΧΟΡΟΥ)

1. ΠΟΙΟΤΗΤΑ ΠΛΗΡΟΦΟΡΙΑΚΟΥ ΠΕΡΙΕΧΟΜΕΝΟΥ ΤΣΑΜΙΚΟΥ

Αρ. Παρακαλείστε να αναφέρετε τον βαθμό που συμφωνείτε με καθεμία από τις ακόλουθες φράσεις


1 2 3 4 5

1 Οι πληροφορίες που παρέχονται στην διαδικτυακή πλατφόρμα για τον Τσάμικο χορό είναι χρήσιμες;

2 Οι πληροφορίες που παρέχονται στην διαδικτυακή πλατφόρμα καλύπτουν διάφορες πτυχές του Τσάμικου χορού (ιστορικές, πολιτιστικές, γεωγραφικές, κτλ.);

3 Παρέχει η διαδικτυακή πλατφόρμα επαρκή αριθμό εξωτερικών συνδέσμων σε ιστοσελίδες που προωθούν το Τσάμικο;

2. ΣΥΣΤΗΜΑ ΔΙΑΧΕΙΡΗΣΗΣ ΜΑΘΗΣΗΣ: ΠΟΙΟΤΗΤΑ ΤΩΝ ΜΑΘΗΜΑΤΩΝ ΤΣΑΜΙΚΟΥ



1 2 3 4 5

1 Μάθατε περισσότερα πράγματα σχετικά με το Τσάμικο (π.χ. πολιτιστικό υπόβαθρο) μετά την ολοκλήρωση αυτού του μαθήματος;

2 Το μάθημα του Τσάμικου που προσφέρθηκε ήταν κατάλληλο για την ηλικία και την εμπειρία των μαθητών;

3 Πιστεύετε πως το μάθημα αυτό κατάφερε να σας διδάξει τα βασικά του Τσάμικου;

3. ΠΟΙΟΤΙΚΑ ΧΑΡΑΚΤΗΡΙΣΤΙΚΑ ΤΟΥ ΕΚΠΑΙΔΕΥΤΙΚΟΥ ΠΑΙΧΝΙΔΙΟΥ ΓΙΑ ΤΟ ΤΣΑΜΙΚΟ

1. Σενάριο παιχνιδιού



1 2 3 4 5

1 Έχετε κατανοήσει σαφώς τους στόχους του παιχνιδιού;

2 Το σενάριο του παιχνιδιού ήταν σαφές και ικανοποιητικό;

3 Ήταν εύκολα κατανοητό τι πρέπει να κάνετε σε κάθε δραστηριότητα του παιχνιδιού.

4 Πιστεύετε ότι η ακολουθία των προσφερόμενων δραστηριοτήτων βοηθά τον χρήστη να μάθει να χορεύει Τσάμικο με σταδιακό και ομαλό τρόπο;

5 Πόσο κατατοπιστική ήταν η ενότητα "Εισαγωγή";

6 Οι βοήθεια/οδηγίες που προσφέρονται από τον εικονικό εκπαιδευτή ήταν χρήσιμες;

2. Απεικόνιση



1 2 3 4 5

1 Πόσο ευχάριστο βρήκατε το 3D περιβάλλον του παιχνιδιού όσον αφορά το σχεδιασμό και την αισθητική;

2 Πόσο ικανοποιητική βρίσκετε την εμφάνιση των εικονικών χαρακτήρων (άβαταρ) του ειδικού και του μαθητή;

3 Πόσο ικανοποιητική/ακριβής είναι η οπτικοποίηση των χορευτικών κινήσεων από τους εικονικούς χαρακτήρες;

4 Πόσο χρήσιμη ήταν η παρακολούθηση του εικονικού χαρακτήρα προκειμένου να μάθετε τα βασικά βήματα του Τσάμικου;

5 Πόσο χρήσιμη ήταν η παρακολούθηση του βίντεο του ειδικού, προκειμένου να μάθετε τα βασικά βήματα του Τσάμικου;

6 Πόσο χρήσιμη ήταν η παρακολούθηση του βίντεο που εστιάζει στα πόδια του εικονικού χαρακτήρα προκειμένου να μάθετε τα βασικά βήματα του Τσάμικου;

7 Ο εικονικός χαρακτήρας (ανάδραση) σας βοήθησε να βελτιώσετε/διορθώσετε τη δική σας χορευτική απόδοση;

8 Πόσο άνετη βρήκατε την εξάσκηση των χορευτικών κινήσεων και την παρακολούθηση του εικονικού χαρακτήρα ταυτόχρονα;

3. Αξιολόγηση Απόδοσης Χρήστη



1 2 3 4 5

1 Σας βοήθησε η βαθμολογία αξιολόγησης που

εμφανίζεται στην οθόνη να καταλάβετε πότε οι κινήσεις σας δεν ήταν σωστές και, ως εκ τούτου, να βελτιώσετε/διορθώσετε την χορευτική σας επίδοση;

2 Ο αλγόριθμος αξιολόγησης που χρησιμοποιείται

από το παιχνίδι ήταν ακριβής; (Παρείχε χαμηλή βαθμολογία όταν οι κινήσεις σας δεν ήταν σωστές και υψηλή βαθμολογία όταν οι κινήσεις σας ήταν σωστές;).

3 Χρειάστηκε να εκτελέσετε κάποια άσκηση πολλές φορές επειδή δεν είχατε καταλάβει τι έπρεπε να κάνετε;

4 Χρειάστηκε να εκτελέσετε κάποια άσκηση πολλές φορές επειδή η αξιολόγηση της απόδοσής σας δεν ήταν ακριβής (οι κινήσεις σας ήταν σωστές, αλλά η βαθμολογία ήταν χαμηλή);

4. Εγκατάσταση και Ρύθμιση Αισθητήρων



1 2 3 4 5

1 Πόσο εύκολα καταλάβατε πού θα πρέπει να στέκεστε (σε ποιο χώρο/περιοχή) όταν κάνετε

εξάσκηση στις χορευτικές κινήσεις;

2 Πόσο άνετο ήταν να χορεύετε σε μια συγκεκριμένη περιοχή, έτσι ώστε να γίνεται

σωστή καταγραφή από τους αισθητήρες;

3 Η τοποθέτηση των αισθητήρων επηρέασε την

απόδοσή σας; Προκλήθηκε κάποια ενόχληση στον χορευτή;

4 Θα είναι εύκολο για σας να χειριστείτε τον αισθητήρα (ες) μόνοι σας; (σύνδεση με τον υπολογιστή και να τοποθέτηση στη σωστή θέση ώστε να καλύπτουν την περιοχή του χορού).

5. Ευχρηστία



1 2 3 4 5

1 Βρήκατε εύκολο το να μάθετε πώς να παίζετε το παιχνίδι;

2 Θα είστε σε θέση να μάθετε πώς να παίζετε το παιχνίδι μόνοι σας (χωρίς οποιαδήποτε εξωτερική καθοδήγηση);

3 Βρήκατε εύκολο να εκτελέσετε τις διάφορες ασκήσεις;

4 Βρήκατε εύκολο να ακολουθήσετε τις κινήσεις του εικονικού χαρακτήρα και να εξασκηθείτε μόνοι σας;

5 Πόσο εύκολο ήταν να περιηγηθείτε μέσα στα 3D

εικονικά περιβάλλοντα και να αλλάξετε τη θέση της εικονικής κάμερας (μεγέθυνση/σμίκρυνση, περιστροφή εικονικού χαρακτήρα);

6 Βρήκατε την ύπαρξη πολλών παραθύρων στην οθόνη ελαφρώς ενοχλητική/κουραστική;

7 Βρίσκετε ικανοποιητική την γενική ανάδραση που

παρέχεται από την πλατφόρμα;

6. Εμπειρία Εκμάθησης



1 2 3 4 5

1 Βρίσκετε ικανοποιητική την δραστηριότητα της "Τελικής άσκησης";

2 Σας παρότρυνε η δραστηριότητα της τελικής

άσκησης να επικεντρωθείτε περισσότερο στην εκμάθηση του Τσάμικου χορού;

3 Καθιστά αυτό το παιχνίδι την εμπειρία εκμάθησης Τσάμικου χορού ευκολότερη σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ. παρατηρώντας έναν δάσκαλο ζωντανά);

4 Καθιστά αυτό το παιχνίδι την εμπειρία εκμάθηση Τσάμικου χορού πιο ευχάριστη σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ. παρατηρώντας έναν

δάσκαλο ζωντανά);

5 Θα χρησιμοποιούσατε ένα παρόμοιο παιχνίδι για να μάθετε μόνος σας κάποιον άλλο χορό;

6 Πόσο σας βοήθησε το παιχνίδι να μάθετε να χορεύετε Τσάμικο;

7 Βρήκατε το παιχνίδι αγχωτικό με οποιονδήποτε τρόπο;

8 Ήταν το παιχνίδι διασκεδαστικό;

9 Θεωρείτε ότι το παιχνίδι σέβεται την παράδοση του Τσάμικου;

10 Θα θέλατε να δείτε ένα τέτοιο παιχνίδι ή παρόμοιες τεχνολογίες να συμπεριλαμβάνονται στην εκπαιδευτική διαδικασία του σχολείου/οργανισμού σας;

11 Θα το βρίσκατε ενδιαφέρον να είστε σε θέση να αγωνίζεστε ενάντια σε άλλους μαθητές για την καλύτερη χορευτική επίδοση χρησιμοποιώντας αυτό το παιχνίδι; (υποστήριξη του σεναρίου πολλαπλών παικτών)

4. ΑΠΟΔΟΣΗ ΤΗΣ ΕΚΠΑΙΔΕΥΤΙΚΗΣ ΕΦΑΡΜΟΓΗΣ ΓΙΑ ΤΟ ΤΣΑΜΙΚΟ



1 2 3 4 5





3 Ικανοποίηση (αν το παιχνίδι παρέχει ικανοποίηση

στο χρήστη)



ΕΝΟΤΗΤΑ 8-9: APPENDIX II

Σας ευχαριστούμε πολύ για τη συνεργασία.

APPENDIX V

13.1 Questionnaire for the evaluation of the Web Platform and the Pottery use case (in English)

QUESTIONNAIRE



2. Age (in years): .............................................



1 2 3 4 5



1 2 3 4 5

5. Experience in pottery making:


1 2 3 4 5


7. User ID: ........................................................

8. Date: ........................................................

9. Place: ........................................................


SECTION 7: 3D ENVIRONMENT SML FUNCTIONALITIES (USE CASE: POTTERY MAKING)

1. INFORMATIONAL CONTENT OF POTTERY


Completely

disagree

Completely

agree

1 2 3 4 5

1 The information provided by the web platform for Pottery making is useful.

2 The information provided by the web platform covers different aspects of Pottery making (historical and cultural, geographical, etc).

3 The web-platform provides an adequate number of external links to websites promoting Pottery making.

2. COURSES WITH RESPECT TO POTTERY MAKING IN LMS


Completely

disagree

Completely

agree

1 2 3 4 5

1 I have learned more about Pottery (cultural background and the Pottery itself) after concluding this course.

2 The Pottery course was appropriate for my age and experience.

3 This course has succeeded to teach me the basics of Pottery.

3. INTERACTIONAL CHARACTERISTICS OF THE POTTERY GAME-LIKE APPLICATION

1. Game Scenario


Completely

disagree

Completely

agree

1 2 3 4 5


game.



4 I think that the sequence of offered activities (i.e., parts/entities taught independently) helped me to learn how to learn Pottery making in a progressive and smooth way.



2. Visualization


Completely

disagree

Completely

agree

1 2 3 4 5



2 I found satisfactory the appearance of the expert and learner pottery avatars.

3 I found accurate the visualization of the potter’s movements by the avatars.

4 I found helpful to watch the virtual avatar of the expert in order to learn the basic stages of Pottery

making.


order to learn the basic stages of Pottery.

6 I found helpful to watch the video focusing on the virtual avatars legs in order to learn the basic

stages of Pottery making.

7 The virtual avatar helped me (visual feedback) to

improve/correct my own pottery making performance.

8 I found comfortable to practice the pottery

gestures and watch the virtual avatar at the same time.

3. User Performance Evaluation


Completely

disagree

Completely

agree

1 2 3 4 5


helped me to understand when my movements were not correct and, thus, I improved/corrected my pottery making performance.





4. Sensor Set-up


Completely

disagree

Completely

agree

1 2 3 4 5


area) when practicing the pottery making.

2 It was comfortable to move my hands at a specific area so as to be captured by the sensors.



4 It will be easy for me to handle the sensor(s) by my own (i.e., connect them to the PC and place them in the correct place so as to cover the pottery making area).

5. Usability


Completely

disagree

Completely

agree

1 2 3 4 5








7 I found the general feedback provided by the

platform satisfactory.

6. Learning Experience


Completely

disagree

Completely

agree

1 2 3 4 5



more on learning how to make pottery.

3 I think that this game makes the pottery learning experience easier compared to traditional techniques (e.g., observing a teacher live).

4 I think that this game makes the pottery learning experience more enjoyable compared to traditional techniques (e.g., observing a teacher live).

5 The game helped me to substantially learn how to make pottery.



8 I think that the game respects the tradition of

Pottery making.


10 I would find it interesting to be able to contest against other individuals for the best pottery making performance using this game (i.e., support of multi-player scenario).

11 The current version of the game affected positively my learning experience.

12 I do believe that the visualization of clay will enhance the learning experience.

4. PERCEIVED PERFORMANCE OF THE POTTERY GAME-LIKE APPLICATION


Completely

disagree

Completely

agree

1 2 3 4 5

1 The Pottery application is effective (if the game meets its objectives).

2 The Pottery application is efficient (if the game responses satisfactorily and in a short time).

3 The Pottery application provides satisfaction (if the

game provides satisfaction to the user).

4 The Pottery application is innovative (if the game offers novel tools/ techniques in ICH transmission).


Thank you very much for your co-operation.

13.2 Questionnaire for the evaluation of the Web Platform and the Pottery use case (in Greek)




2. Ηλικία (σε χρόνια): .............................................





1 2 3 4 5


Πολύ φτωχή


1 2 3 4 5

5. Εξοικείωση με την αγγειοπλαστική:


1 2 3 4 5



8. Ημερομηνία: . .......................................................

9. Τόπος: ........................................................

EΝΟΤΗΤΑ 2-6: APPENDIX II

ΕΝΟΤΗΤΑ 7: 3D ENVIRONMENT SML FUNCTIONALITIES(USE CASE: POTTERY MAKING)

1. ΠΟΙΟΤΗΤΑ ΠΛΗΡΟΦΟΡΙΑΚΟΥ ΠΕΡΙΕΧΟΜΕΝΟΥ ΑΓΓΕΙΟΠΛΑΣΤΙΚΗΣ

Αρ.

Παρακαλείστε να αναφέρετε τον βαθμό που συμφωνείτε με καθεμία από τις ακόλουθες φράσεις


1 2 3 4 5

1 Οι πληροφορίες που παρέχονται στην διαδικτυακή πλατφόρμα για την αγγειοπλαστική είναι χρήσιμες;

2 Οι πληροφορίες που παρέχονται στην διαδικτυακή πλατφόρμα καλύπτουν διάφορες πτυχές της αγγειοπλαστικής (ιστορικές, πολιτιστικές, γεωγραφικές, κτλ.);

3 Παρέχει η διαδικτυακή πλατφόρμα επαρκή αριθμό εξωτερικών συνδέσμων σε ιστοσελίδες που προωθούν την αγγειοπλαστική;

2. ΣΥΣΤΗΜΑ ΔΙΑΧΕΙΡΗΣΗΣ ΜΑΘΗΣΗΣ: ΠΟΙΟΤΗΤΑ ΤΩΝ ΜΑΘΗΜΑΤΩΝ ΑΓΓΕΙΟΠΛΑΣΤΙΚΗΣ



1 2 3 4 5

1 Μάθατε περισσότερα πράγματα σχετικά με την αγγειοπλαστική (π.χ. πολιτιστικό υπόβαθρο) μετά την ολοκλήρωση αυτού του μαθήματος;

2 Το μάθημα της αγγειοπλαστικής που προσφέρθηκε ήταν κατάλληλο για την ηλικία και την εμπειρία των μαθητών;

3 Πιστεύετε πως το μάθημα αυτό κατάφερε να σας διδάξει τα βασικά της αγγειοπλαστικής;

3. ΠΟΙΟΤΙΚΑ ΧΑΡΑΚΤΗΡΙΣΤΙΚΑ ΤΟΥ ΕΚΠΑΙΔΕΥΤΙΚΟΥ ΠΑΙΧΝΙΔΙΟΥ ΓΙΑ ΤΗΝ ΑΓΓΕΙΟΠΛΑΣΤΙΚΗ




1 2 3 4 5

1 Έχετε κατανοήσει σαφώς τους στόχους του

παιχνιδιού;

2 Το σενάριο του παιχνιδιού ήταν σαφές και

ικανοποιητικό;

3 Ήταν εύκολα κατανοητό τι πρέπει να κάνετε σε κάθε δραστηριότητα του παιχνιδιού;

4 Πιστεύετε ότι η ακολουθία των προσφερόμενων δραστηριοτήτων βοηθά τον χρήστη να μάθει αγγειοπλαστική με σταδιακό και ομαλό τρόπο;





1 2 3 4 5

1 Πόσο ευχάριστο βρήκατε το 3D περιβάλλον του

παιχνιδιού όσον αφορά το σχεδιασμό και την αισθητική;

2 H ανάδραση (σκορ) σας βοήθησε να

βελτιώσετε/διορθώσετε τη δική σας απόδοση;

3 Πόσο άνετη βρήκατε την εξάσκηση των χειρονομιών αγγειοπλαστικής;

3. Αξιολόγηση απόδοσης χρήστη



1 2 3 4 5


εμφανίζεται στην οθόνη να καταλάβετε πότε οι κινήσεις σας δεν ήταν σωστές και, ως εκ τούτου, να βελτιώσετε/διορθώσετε την επίδοσή σας;


από το παιχνίδι ήταν ακριβής;

3 Χρειάστηκε να εκτελέσετε κάποια δραστηριότητα πολλές φορές επειδή δεν είχατε καταλάβει τι έπρεπε να κάνετε;

4 Χρειάστηκε να εκτελέσετε κάποια δραστηριότητα πολλές φορές επειδή η αξιολόγηση της απόδοσής σας δεν ήταν ακριβής (οι κινήσεις σας ήταν σωστές, αλλά η βαθμολογία ήταν χαμηλή);




1 2 3 4 5

1 Πόσο εύκολα καταλάβατε που θα πρέπει να τοποθετήσετε τα χέρια σας (σε ποια περιοχή) κατά την εφαρμογή της αγγειοπλαστικής;

2 Πόσο άνετο ήταν να μετακινήσετε τα χέρια σας σε μια συγκεκριμένη περιοχή, έτσι ώστε να γίνεται σωστή καταγραφή από τους αισθητήρες;

3 Η τοποθέτηση των αισθητήρων επηρέασε την απόδοσή σας; Προκλήθηκε κάποια ενόχληση στον αγγειοπλάστη;

4 Θα είναι εύκολο για σας να χειριστείτε τον αισθητήρα(ες) μόνοι σας; (σύνδεση με τον υπολογιστή και τοποθέτηση στη σωστή θέση).




1 2 3 4 5



3 Βρήκατε εύκολο να εκτελέσετε τις διάφορες δραστηριότητες;

4 Βρίσκετε ικανοποιητική την γενική ανάδραση που παρέχεται από την πλατφόρμα;

6. Εμπειρία εκμάθησης



1 2 3 4 5

1 Καθιστά αυτό το παιχνίδι την εμπειρία εκμάθησης αγγειοπλαστικής ευκολότερη σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ. παρατηρώντας έναν δάσκαλο ζωντανά);

2 Καθιστά αυτό το παιχνίδι την εμπειρία εκμάθησης αγγειοπλαστικής πιο ευχάριστη σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ. παρατηρώντας έναν δάσκαλο ζωντανά);

3 Πόσο σας βοήθησε το παιχνίδι να μάθετε αγγειοπλαστική;



6 Θεωρείτε ότι το παιχνίδι σέβεται την παράδοση της αγγειοπλαστικής;


8 Θα το βρίσκατε ενδιαφέρον να είστε σε θέση να αγωνίζεστε ενάντια σε άλλους μαθητές για την καλύτερη επίδοση στην αγγειοπλαστική χρησιμοποιώντας αυτό το παιχνίδι; (υποστήριξη του σεναρίου πολλαπλών παικτών)

9 Η υπάρχουσα έκδοση του παιχνιδιού υποστηρίζει την εικονική αγγειοπλαστική. Επηρέασε αυτό την εμπειρία μάθησής σας;

10 Η οπτικοποίηση του πηλού ενισχύει την εμπειρία μάθησης;

11 Θεωρείτε χρήσιμη τη δυνατότητα να σώσετε τα αρχεία των τρισδιάστατων αγγείων που δημιουργήσατε, ώστε να μπορείτε στο μέλλον να τα επεξεργαστείτε ή να τα εκτυπώσετε;

4. ΑΠΟΔΟΣΗ ΤΗΣ ΕΚΠΑΙΔΕΥΤΙΚΗΣ ΕΦΑΡΜΟΓΗΣ ΓΙΑ ΤΗΝ ΑΓΓΕΙΟΠΛΑΣΤΙΚΗ



1 2 3 4 5










APPENDIX VI

14.1 Questionnaire for the evaluation of the Web Platform and the Salsa Dance use case (in English)

QUESTIONNAIRE



2. Age (in years): .............................................



1 2 3 4 5



1 2 3 4 5



1 2 3 4 5

5.1 Experience in dancing latin dances:


1 2 3 4 5

5.2 Experience in dancing Salsa:


1 2 3 4 5


7. User ID: ........................................................

8. Date: ........................................................

9. Place: ........................................................


SECTION 7: 3D ENVIRONMENT SML FUNCTIONALITIES (USE CASE: SALSA DANCE)

1. INFORMATIONAL CONTENT FOR SALSA DANCE


Completely

disagree

Completely

agree

1 2 3 4 5

1 The information provided by the web platform for Salsa dance is useful.

2 The information provided by the web platform covers different aspects of Salsa dance (historical and cultural, geographical, etc).

3 The web-platform provides an adequate number of external links to websites promoting Salsa dance.

2. COURSES WITH RESPECT TO SALSA DANCE IN LMS


Completely

disagree

Completely

agree

1 2 3 4 5

1 I have learned more about Salsa (cultural background and the dance itself) after concluding this course.

2 The Salsa course was appropriate for my age and experience.

3 This course has succeeded to teach me the basics of Salsa.

3. INTERACTIONAL CHARACTERISTICS OF THE SALSA DANCE GAME-LIKE APPLICATION

1. Game Scenario


Completely

disagree

Completely

agree

1 2 3 4 5


game.



4 I think that the sequence of offered activities (i.e., parts/entities taught independently) helped me to learn how to dance Salsa in a progressive and smooth way.



2. Visualization


Completely

disagree

Completely

agree

1 2 3 4 5






Salsa.


order to learn the basic movements of Salsa.





Completely

disagree

Completely

agree

1 2 3 4 5

1 The evaluation score displayed in the screen at

the end of each exercise helped me to understand when my movements were not correct and, thus, I improved/corrected my dancing performance.





4. Sensor Set-up


Completely

disagree

Completely

agree

1 2 3 4 5








5. Usability


Completely

disagree

Completely

agree

1 2 3 4 5







satisfactory.



Completely

disagree

Completely

agree

1 2 3 4 5

1 I found the activities satisfactory.




5 The game helped me to learn how to dance Salsa.



8 I think that the game respects the tradition of Salsa dance.



11 It would be helpful to be able to watch a video focusing on the virtual avatar’s legs in order to

learn the basic movements of Salsa.

12 The addition of the overlay of the virtual avatar and the user ghost image helped me (visual feedback)

would help me to improve/correct my own dancing

performance.

13 It would be useful to be able to navigate through

the 3D virtual environments and change the position of the virtual camera (zoom in/out, rotate virtual avatar).

14 It would be helpful to be able to watch the evaluation score displayed in the screen during the activities (i.e., to actually provide a low score when my movements are not correct and a high score when my movements are correct).

4. PERCEIVED PERFORMANCE OF THE SALSA GAME-LIKE APPLICATION


Completely

disagree

Completely

agree

1 2 3 4 5

1 The Salsa game is effective (if the game meets its objectives).

2 The Salsa game is efficient (if the game responses

satisfactorily and in a short time).

3 The Salsa game provides satisfaction (if the game provides satisfaction to the user).

4 The Salsa game is innovative (if the game offers novel tools/ techniques in ICH transmission).

14.2 Questionnaire for the evaluation of the Web Platform and the Salsa Dance use case (in Greek)




2. Ηλικία (σε χρόνια): .............................................


Βασική Γυμνάσιο/Λύκειο



1 2 3 4 5

4. Εξοικείωση στη χρήση Η/Υ :

Πολύ φτωχή


1 2 3 4 5

5. Εξοικείωση στον χορό:

Καθόλου ‘Όχι αρκετή Μέτρια Καλή Πολύ καλή

1 2 3 4 5

5.1 Εξοικείωση στους Latin χορούς:


1 2 3 4 5

5.2 Εξοικείωση στο χορό Salsa:


1 2 3 4 5



8. Ημερομηνία: ........................................................

9. Τόπος: ........................................................

ΕΝΟΤΗΤΑ 2-6: APPENDIX VII

ΕΝΟΤΗΤΑ 7: ΛΕΙΤΟΥΡΓΙΕΣ ΤΟΥ ΤΡΙΣΔΙΑΣΤΑΤΟΥ ΠΕΡΙΒΑΛΛΟΝΤΟΣ (ΣΕΝΑΡΙΟ ΧΡΗΣΗΣ ΧΟΡΟΥ SALSA)

1. ΠΟΙΟΤΗΤΑ ΠΛΗΡΟΦΟΡΙΑΚΟΥ ΠΕΡΙΕΧΟΜΕΝΟΥ ΧΟΡΟΥ SALSA



1 2 3 4 5

1 Οι πληροφορίες που παρέχονται στην διαδικτυακή πλατφόρμα για τον χορό Salsa είναι χρήσιμες;

2 Οι πληροφορίες που παρέχονται στην διαδικτυακή πλατφόρμα καλύπτουν διάφορες πτυχές του χορού Salsa (ιστορικές, πολιτιστικές, γεωγραφικές, κτλ.);

3 Παρέχει η διαδικτυακή πλατφόρμα επαρκή αριθμό εξωτερικών συνδέσμων σε ιστοσελίδες που προωθούν το χορό Salsa;

2. ΣΥΣΤΗΜΑ ΔΙΑΧΕΙΡΙΣΗΣ ΜΑΘΗΣΗΣ: ΠΟΙΟΤΗΤΑ ΤΩΝ ΜΑΘΗΜΑΤΩΝ SALSA



1 2 3 4 5

1 Μάθατε περισσότερα πράγματα σχετικά με το χορό Salsa (π.χ. πολιτιστικό υπόβαθρο) μετά την ολοκλήρωση αυτού του μαθήματος;

2 Το μάθημα του χορού Salsa που προσφέρθηκε ήταν κατάλληλο για την ηλικία και την εμπειρία των μαθητών;

3 Πιστεύετε πως το μάθημα αυτό κατάφερε να σας διδάξει τα βασικά του χορού Salsa;

3. ΠΟΙΟΤΙΚΑ ΧΑΡΑΚΤΗΡΙΣΤΙΚΑ ΤΟΥ ΕΚΠΑΙΔΕΥΤΙΚΟΥ ΠΑΙΧΝΙΔΙΟΥ ΓΙΑ ΤΟ ΧΟΡΟ SALSA




1 2 3 4 5




4 Πιστεύετε ότι η ακολουθία των προσφερόμενων δραστηριοτήτων βοηθά τον χρήστη να μάθει να χορεύει Salsa με σταδιακό και ομαλό τρόπο;


6 Οι βοήθεια/οδηγίες που προσφέρονται από τον

εικονικό εκπαιδευτή ήταν χρήσιμες;




1 2 3 4 5

1 Πόσο ευχάριστο βρήκατε το 3D περιβάλλον του παιχνιδιού όσον αφορά το σχεδιασμό και την αισθητική;

2 Πόσο ικανοποιητική βρίσκετε την εμφάνιση των εικονικών χαρακτήρων (άβαταρ) του ειδικού και του μαθητή;

3 Πόσο ικανοποιητική/ακριβής είναι η οπτικοποίηση των χορευτικών κινήσεων από τους εικονικούς χαρακτήρες;

4 Πόσο χρήσιμη ήταν η παρακολούθηση του εικονικού χαρακτήρα προκειμένου να μάθετε τα βασικά βήματα του χορού Salsa;

5 Πόσο χρήσιμη ήταν η παρακολούθηση του βίντεο του ειδικού, προκειμένου να μάθετε τα βασικά βήματα του χορού Salsa;

6 Πόσο άνετη βρήκατε την εξάσκηση των χορευτικών κινήσεων και την παρακολούθηση του εικονικού χαρακτήρα ταυτόχρονα;

3. Αξιολόγηση Απόδοσης Χρήστη



1 2 3 4 5


εμφανίζεται στην οθόνη να καταλάβετε πότε οι κινήσεις σας δεν ήταν σωστές και, ως εκ τούτου, να βελτιώσετε/διορθώσετε την χορευτική σας επίδοση;



3 Χρειάστηκε να εκτελέσετε κάποια άσκηση πολλές φορές επειδή δεν είχατε καταλάβει τι έπρεπε να κάνετε;

4 Χρειάστηκε να εκτελέσετε κάποια άσκηση πολλές φορές επειδή η αξιολόγηση της απόδοσής σας δεν ήταν ακριβής (οι κινήσεις σας ήταν σωστές, αλλά η βαθμολογία ήταν χαμηλή);




1 2 3 4 5

1 Πόσο εύκολα καταλάβατε πού θα πρέπει να στέκεστε (σε ποιο χώρο/περιοχή) όταν κάνετε

εξάσκηση στις χορευτικές κινήσεις;

2 Πόσο άνετο ήταν να χορεύετε σε μια συγκεκριμένη περιοχή, έτσι ώστε να γίνεται

σωστή καταγραφή από τους αισθητήρες;

3 Η τοποθέτηση των αισθητήρων επηρέασε την

απόδοσή σας; Προκλήθηκε κάποια ενόχληση στον χορευτή;

4 Θα είναι εύκολο για σας να χειριστείτε τον αισθητήρα (ες) μόνοι σας; (σύνδεση με τον υπολογιστή και να τοποθέτηση στη σωστή θέση ώστε να καλύπτουν την περιοχή του χορού).




1 2 3 4 5



3 Βρήκατε εύκολο να εκτελέσετε τις διάφορες ασκήσεις;

4 Βρήκατε εύκολο να ακολουθήσετε τις κινήσεις του εικονικού χαρακτήρα και να εξασκηθείτε μόνοι σας;


6 Βρίσκετε ικανοποιητική την γενική ανάδραση που

παρέχεται από την πλατφόρμα;




1 2 3 4 5

1 Βρήκατε τις δραστηριότητες ικανοποιητικές;

2 Καθιστά αυτό το παιχνίδι την εμπειρία εκμάθησης χορού Salsa ευκολότερη σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ. παρατηρώντας έναν δάσκαλο ζωντανά);

3 Καθιστά αυτό το παιχνίδι την εμπειρία εκμάθηση χορού Salsa πιο ευχάριστη σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ. παρατηρώντας έναν δάσκαλο ζωντανά);

4 Θα χρησιμοποιούσατε ένα παρόμοιο παιχνίδι για να μάθετε μόνος σας κάποιον άλλο χορό;

5 Πόσο σας βοήθησε το παιχνίδι να μάθετε να χορεύετε Salsa;



8 Θεωρείτε ότι το παιχνίδι σέβεται την παράδοση του

χορού Salsa;


10 Θα το βρίσκατε ενδιαφέρον να είστε σε θέση να αγωνίζεστε ενάντια σε άλλους μαθητές για την καλύτερη χορευτική επίδοση χρησιμοποιώντας αυτό το παιχνίδι; (υποστήριξη του σεναρίου πολλαπλών παικτών)

11 Πόσο χρήσιμη θα ήταν η παρακολούθηση ενός βίντεο που εστιάζει στα πόδια του εικονικού χαρακτήρα προκειμένου να μάθετε τα βασικά βήματα του χορού Salsa;

12 Θα σας βοηθούσε η προσθήκη εικονικού χαρακτήρα (ανάδραση) να βελτιώσετε/διορθώσετε τη δική σας χορευτική απόδοση;

13 Πόσο χρήσιμο θα ήταν να περιηγηθείτε μέσα στα

3D εικονικά περιβάλλοντα και να αλλάζατε τη θέση της εικονικής κάμερας (μεγέθυνση/σμίκρυνση, περιστροφή εικονικού χαρακτήρα);

14 Πόσο χρήσιμο θα ήταν να απεικονίζεται ένας βαθμός αξιολόγησης στην οθόνη κατά την διάρκεια των δραστηριοτήτων (να παρέχεται ένας χαμηλός βαθμός όταν οι κινήσεις μου δεν είναι σωστές και ένας υψηλός βαθμός όταν οι κινήσεις μου είναι σωστές);

4. ΑΠΟΔΟΣΗ ΤΗΣ ΕΚΠΑΙΔΕΥΤΙΚΗΣ ΕΦΑΡΜΟΓΗΣ ΓΙΑ ΤΟ ΧΟΡΟ SALSA



1 2 3 4 5










APPENDIX VII 15.1 Questionnaire for the evaluation of the Web Platform and the Byzantine music

use case (in English)

QUESTIONNAIRE



2. Age (in years): .............................................



1 2 3 4 5



1 2 3 4 5

5. Role in Byzantine music:

Cantor Chanter Chorister Priest Other

1 2 3 4 5

6. Experience in Byzantine music:


1 2 3 4 5

6.1 Experience in choral music:


1 2 3 4 5

6.2 Experience in European music:


1 2 3 4 5

6.3 Experience in stringed musical instruments (e.g., qanun, tambouras):


1 2 3 4 5


8. User ID: ........................................................

9. Date: ........................................................

10. Place: ........................................................

SECTION 2-6: APPENDIX I

SECTION 7: 3D ENVIRONMENT SML FUNCTIONALITIES (USE CASE: Byzantine music)

1. QUALITY CHARACTERISTICS OF BYZANTINE MUSIC (BM) LEARNING PLATFORM


Completely

disagree

Completely

agree

1 2 3 4 5


game.







7 I found satisfactory the appearance of the expert and learner avatars.

8 I found accurate the musician’s movements in 3d environment.

9 I found helpful to watch the byzantine music score

in order to follow the hymn.


order to learn the basic rhythm gestures.

11 The expert’s video helped me (visual feedback) to

improve/correct my own performance.

12 I found comfortable to practice the hymn with

musical rhythm gestures and watch the expert’s video at the same time.


14 I found the feedback game satisfactory.

2. LEARNING EXPERIENCE


Completely

disagree

Completely

agree

1 2 3 4 5

1 It is useful because it helps me to improve/correct my chanting technique.

2 I think that comparing my performance with expert’s one is a way to evaluate my chanting techniques.

3 It is user friendly.

4 It is easy to learn to perform/reproduce watching

the expert’s musical gestures.

5 The evaluation helped me to improve my chant

6 The singing of based tone before you start is very useful for singing at the correct pitch.

7 The use of metronome was very useful


9 I think that this game makes the learning experience of Byzantine music more enjoyable compared to traditional techniques (e.g., observing a teacher live).

3. PERCEIVED PERFORMANCE OF THE BYZANITNE MUSIC (BM) LEARNING PLATFORM


Completely

disagree

Completely

agree

1 2 3 4 5

1 The BM game is effective (if the game meets its

objectives).

2 The BM game is efficient (if the game responses satisfactorily and in a short time).

3 The BM game provides satisfaction (if the game provides satisfaction to the user).

4 The BM game is innovative (if the game offers novel tools/ techniques in ICH transmission).

SECTION 8-9: APPENDIX I

15.2 Questionnaire for the evaluation of the Web Platform and the Byzantine music use case (in Greek)



1. Γένος: Άνδρας 1 Γυναίκα 2

2. Ηλικία (σε χρόνια): .............................................


Βασική Γυμνάσιο / Λύκειο

ΙΕΚ Ανώτατη

(ΑΕΙ/ΤΕΙ)

Μετα-πτυχιακή

1 2 3 4 5

4. Αντιληπτή παιδεία Η/Υ:

Πολύ φτωχή

Χαμηλή Μέτρια Αρκετή Πάρα πολύ

1 2 3 4 5

5. Επίπεδο Βυζαντινής μουσικής

Τραγουδιστής

Ψάλτης Χορωδός Κληρικός Άλλο

1 2 3 4 5

6. Εμπειρία με τη βυζαντινή μουσική:


1 2 3 4 5

6.1 Εμπειρία με τη χορωδιακή μουσική:


1 2 3 4 5

6.2 Εμπειρία με την ευρωπαϊκή μουσική


1 2 3 4 5

6.3 Εμπειρία στα άταστα έγχορδα όργανα (όπως ο ταμπουράς, λύρα κ.α.)


1 2 3 4 5


8. User ID: ........................................................

9. Ημερομηνία: ........................................................

10. Τόπος: ........................................................

ΕΝΟΤΗΤΑ 2-6: APPENDIX I

ΕΝΟΤΗΤΑ 7: ΛΕΙΤΟΥΡΓΙΕΣ 3D ΑΙΣΘΗΤΗΡΙΟΚΙΝΗΤΙΚΗΣ ΜΑΘΗΣΗΣ

( ΠΕΡΙΠΤΩΣΗ: ΒΥΖΑΝΤΙΝΗΣ ΜΟΥΣΙΚΗΣ (ΒΜ))

1. ΠΟΙΟΤΙΚΑ ΧΑΡΑΚΤΗΡΙΣΤΙΚΑ ΤΟΥ ΠΑΙΧΝΙΔΙΟΥ ΤΗΣ ΒΥΖΑΝΤΙΝΗΣ ΜΟΥΣΙΚΗΣ



1 2 3 4 5

1 Κατάλαβα αμέσως τα αντικείμενα του παιχνιδιού

2 Βρήκα το σενάριο του παιχνιδιού ξεκάθαρο και ικανοποιητικό.

3 Κατάλαβα αμέσως τι έπρεπε να κάνω σε κάθε δραστηριότητα

4 To πεδίο “Getting started” με ενημέρωσε

πλήρως.

5 Βρήκα πολύ ικανοποιητικές τις οδηγίες του οδηγού πλοήγησης εικονικής πραγματικότητας (virtual tutor).

6 Το περιβάλλον του 3d παιχνιδιού ήταν ωραίο αισθητικά και καλοσχεδιασμένο .

7 Βρήκα ικανοποιητική την εμφάνιση του avatar

ειδικού και του εκπαιδευόμενου.

8 Βρήκα ακριβή την απεικόνιση των κινήσεων του μουσικού στο 3d περιβάλλον.

9 Βρήκα χρήσιμη την παρακολούθηση της παρτιτούρας για τη μουσική απόδοση του ύμνου.

10 Βρήκα πολύ χρήσιμο το βίντεο του ειδικού για να μάθω τις βασικές χειρονομίες του μετρήματος του ρυθμού.

11 Το βίντεο του ειδικού (οπτική ανάδραση) για τη βελτίωση / διόρθωση των δική μου απόδοση.

12 Ήταν πιο εύκολο να κάνω πρακτική εξάσκηση με τις ρυθμικές κινήσεις του ύμνου βλέποντας το βίντεο του ειδικού την ίδια στιγμή

13 Δεν ήταν ενοχλητικό και δεν προκαλούσε σύγχυση η ύπαρξη πολλών παραθύρων στην οθόνη

14 Βρήκα ικανοποιητική την ανάδραση του παιχνιδιού

2. ΕΜΠΕΙΡΙΑ ΜΑΘΗΣΗΣ



1 2 3 4 5

1 Είναι χρήσιμο γιατί με βοηθάει να βελτιωθεί / διορθωθεί η τεχνική μου στην ψαλτική.

2 Νομίζω ότι συγκρίνοντας την εκτέλεση του ύμνου με εκείνη του ειδικού με βοηθάει να βελτιώσω στο ψάλσιμό μου.

3 Είναι φιλικό στο χρήστη.

4 Είναι ευκολότερο να μετρήσεις τον χρόνο παρακολουθώντας τις κινήσεις του ειδικού

5 Η αξιολόγηση με βοήθησε να βελτιωθώ στο ψάλσιμό μου

6 Η αρχική μελωδική γραμμή που δίνεται πριν ξεκινήσω με βοηθάει στο καθορισμό του τονικού ύψους.

7 Βρήκα το μετρονόμο πολύ χρήσιμο

8 Διασκέδασα πολύ με το παιχνίδι

9 Νομίζω ότι το αυτό του είδους το παιχνίδι είναι πολύ πιο ενδιαφέρον στο να μάθει κανείς βυζαντινή μουσική από ότι ο παραδοσιακός τρόπος (παρατηρώντας τον ειδικό)

3. ΑΝΤΙΛΗΠΤΗ ΕΠΙΔΟΣΗ ΤΟΥ ΠΑΙΧΝΙΔΙΟΥ ΤΗΣ ΒΥΖΑΝΤΙΝΗΣ ΜΟΥΣΙΚΗΣ


Πολύ κακή Πολύ καλή

1 2 3 4 5

1 Το παιχνίδι BM είναι αποτελεσματικό (αν το παιχνίδι πληροί τους στόχους του).

2 Το παιχνίδι BM είναι αποτελεσματικό (αν ανταποκριθεί το παιχνίδι σε σύντομο χρονικό διάστημα).

3 Το παιχνίδι BM παρέχει ικανοποίηση (από το παιχνίδι παρέχει ικανοποίηση στον χρήστη).

4 Το παιχνίδι BM είναι καινοτόμο (αν το παιχνίδι προσφέρει νέα εργαλεία / τεχνικές στη μετάδοση ICH).

ΕΝΟΤΗΤΑ 8 - 9: APPENDIX II

APPENDIX VIII 16.1 Questionnaire for the evaluation of the Web Platform and the Human Beat-Box

use case (in French)

QUESTIONNAIRE

SECTION 1: DETAILS CONCERNANT LA PERSONNE REPONDANTE

1. Sexe: Masculin 1 Féminin 2

2. Age (en années): .............................................

3. Niveau d’éducation (entourer le plus élevé):

Primaire Secondaire Collège Université De 3ème

cycle

1 2 3 4 5

4. Familiarité dans l’utilisation des ordinateurs:

Presque inexistante

Peu importante

Adéquate Bonne Très bonne

1 2 3 4 5

5. Expérience en chant (Rock, jazz, pop …):

Aucune Peu importante


1 2 3 4 5

6. Expérience en Beatbox:



1 2 3 4 5

6.1 Experience dans l’imitation des percussions (Cymbales, caisses …):



1 2 3 4 5

6.2 Experience dans l’imitation des instruments à cordes (Guitares …):



1 2 3 4 5

6.3 Experience dans l’imitation des instruments à vent (Trommpettes, flûtes …):



1 2 3 4 5

6.4 Experience dans l’imitation des sons électroniques (Shuffling, vocoder …):



1 2 3 4 5

6.5 Experience dans l’imitation des animaux (Crickets, mouches …):



1 2 3 4 5

7. Statut de la personne répondante Experte 1 Utilisateur 2

8. Identité de l’utilisateur : ........................................................

9. Date: ........................................................

10. Lieu: ........................................................


1. OBJECTIF DE LA PLATEFORME WEB

No. Merci d’indiquer votre niveau d’accord avec chacune des affirmations suivantes

En total

Désaccord

En complet accord

1 2 3 4 5

1 La mission de la plateforme web est suffisamment établie (claire, visible).

2 Les objectifs de la plateforme web sont suffisamment établis (claire, visible).

2. QUALITE DU CONTENU INFORMATIF


En total

Désaccord

En complet accord

1 2 3 4 5

1 L’information fournie par la plateforme web est précise

2 L’information fournie par la plateforme web est utile en encourageant la formation continue

3. NIVEAU DE DIFFICULTE DU CONTENU INFORMATIF


En total

désaccord

En complet accord

1 2 3 4 5

1 L’information fournie est trop complexe.

2 L’information fournie est trop technique.

4. ORGANISATION DU CONTENU INFORMATIF


En total

désaccord

En complet accord

1 2 3 4 5

1 Le contenu informatif est organisé de manière appropriée, hiérarchiquement à l’intérieur de chaque section thématique

2 Le contenu informatif est organisé de manière appropriée en terme de présentation multimédia

(son, image, vidéo, animation, et graphiques).

SECTION 3: CONCEPTION

1. ACCESSIBILITE


En total

désaccord

En complet accord

1 2 3 4 5

1 La plateforme web peut être correctement visualisée par les navigateurs modernes

2 La plateforme web est suffisante pour fournir des mécanismes de contrôle d'accès (authentification en utilisant le nom d'utilisateur et mot de passe pour chaque utilisateur).

3 L’information est facilement accessible.

2. NAVIGATION


En total

désaccord

En complet accord

1 2 3 4 5

1 La plateforme web fournit des mécanismes de navigation clairs.

2 La présentation du contenu est cohérente sur toutes les pages.

3 Le mécanisme de navigation permet facilement à l'utilisateur d'être en mesure de revenir à une page ou un niveau précédents, ou la page d'accueil.

4 Il est possible pour l’utilisateur de savoir où il est sur la plateforme à chaque instant.

3. CONCEPTION DE L’INTERFACE DE LA PLATFORM WEB


En total

désaccord

En complet accord

1 2 3 4 5

1 La conception de l'interface de la plateforme web est attrayante.

2 Les couleurs utilisées sur cette plateforme web sont confortables (par exemple la teinte, la netteté, la luminosité, etc.)

3 La plateforme Web ne contient pas de caractéristiques qui irritent, telles que le défilement ou le texte clignotant.

4. INTERACTION


En total

désaccord

En complet accord

1 2 3 4 5

1 La plateforme web fournit des informations de feedback adéquat pour l’utilisateur

2 Il est facile d’apprendre comment utiliser cette plateforme web.

SECTION 4: FONCTIONNALITES DU SYSTEME DE GESTION DE L’APPRENTISSAGE (SGA) (Learning Management System LMS)

1. CONCEPTION


En total

désaccord

En complet accord

1 2 3 4 5

1 L’accessibilité au sein du SGA est facile et

confortable

2 La navigation au sein du SGA est facile et

intuitive.

3 L’interface au sein du SGA est attrayante

4 L’interaction (le feedback) au sein du SGA est

adéquate.

2. PROPRIETES DES COURS


En total

désaccord

En complet accord

1 2 3 4 5

1 Organisation: les contenus fournis dans le cours

sont bien organisés.

2 Clarté: les contenus fournis dans le cours sont

clairs.

3 Exhaustivité: les contenus fournis dans le cours

sont complets.

4 Adéquation: les contenus fournis dans le cours

sont fiables et appropriés.

3. RESSOURCES ET OUTILS


En total

désaccord

En complet accord

1 2 3 4 5

1 J’ai trouvé la présentation des outils (par exemple

SCORM, Power Point) facile à utiliser et utile.

2 J’ai trouvé que les ressources (par exemple audio,

vidéo, texte à chant, vidéos annotées, liens vers des ressources externes) faciles à utiliser et utiles.

4. CONVIVIALITE


En total

désaccord

En complet accord

1 2 3 4 5

1 Facilité de compréhension: il était facile de

comprendre les activités dans le SGA.

2 Facilité d’apprentissage: il était facile d’apprendre

les processus dans le SGA.

3 Maniabilité: Il était facile de gérer des tâches au

sein du SGA.

SECTION 5: FONCTIONNALITES DE RECHERCHES SEMANTIQUES


En total

désaccord

En complet accord

1 2 3 4 5

1 Il est facile de comprendre quel type de requêtes est possible à faire dans cette plateforme web et le type d'informations qui peuvent être récupérées.

2 Il est facile de comprendre le sens de chaque critère / filtre de recherche, et comment l'utiliser.

3 Il est facile d’effectuer une requête complexe (en utilisant plusieurs critères).

4 L'interface de la fonction de recherche est satisfaisante.

5 Les résultats de la recherche et leur présentation sont satisfaisants.

6 En général, il est facile d'utiliser la fonctionnalité de recherche de la plateforme Web.

7 La fonctionnalité de recherche offerte par la plateforme web est satisfaisante par rapport à d'autres platesformes de recherche bien connues.

SECTION 7: ENVIRONMENT 3D: SML FUNCTIONALITIES (CAS du HUMAN BEAT-BOX)

1. CHARACTERISTIQUES de QUALITE de la PLATEFORME D’APPRENTISSAGE

du HUMAN BEAT-BOX (HBB )


En total

désaccord

En complet accord

1 2 3 4 5

1 Il était facile de stabiliser le casque sur ma tête.

2 En général, j’ai trouvé facile de manipuler le casque pendant un certain temps.

3 Il était facile de trouver l’endroit exact pour appliquer la sonde ultrason (US) pour obtenir une image claire du contour de la langue.

4 L’ultrason ne m’interdisait pas de faire des gestes pendant que je produisais certains sons.

5 En général, j’ai trouvé facile de manipuler l’US pendant un certain temps.

6 Il était facile de régler la caméra des lèvres pour capturer leur image.

7 La caméra des lèvres ne m’interdisait pas de faire des gestes pendant que je produisais certains sons.

8 Il était facile de trouver l’endroit exact pour appliquer les capteurs EGG en utilisant l’interface matérielle professionnelle de Glottal Enterprise

9 Les capteurs EGG ne m’interdisaient pas de faire des gestes pendant que je produisais certains sons.

10 En général, j’ai trouvé facile de manipuler les capteurs EGG pendant un certain temps.

11 Il était facile de trouver l’endroit exact pour appliquer les accéléromètres nasals pour obtenir un signal correct.

12 Les accéléromètres nasals ne m’interdisaient pas de faire des gestes pendant que je produisais certains sons.

13 En général, j’ai trouvé facile de manipuler les accéléromètres nasals pendant un certain temps.

14 En général, j’ai trouvé facile de manipuler le casque et ses capteurs pendant un certain temps.

2. EXPERIENCE D APPRENTISSAGE


En total

désaccord

En complet accord

1 2 3 4 5

1 Le casque m’a aidé à améliorer/corriger ma technique vocale.

2 L’évaluation (comparaison à la performance de l’expert) basée sur le son enregistré à partir du micro contribue à l’amélioration/correction de ma technique vocale.

3 La modélisation de la langue et son évaluation ont contribué à l’amélioration/correction de ma technique vocale.

4 La visualisation des lèves et son évaluation ont contribué à l’amélioration/correction de ma technique vocale.

5 L’évaluation basée sur les signaux EGG nt contribué à l’amélioration/correction de ma technique vocale.

6 L’évaluation basée sur les accéléromètres nasals a contribué à l’amélioration/correction de ma technique vocale.

7 Le casque est un outil efficace pour apprendre à produire les sons HBB.

8 Je pense que comparer la performance d’un apprenant à celle d’un chanteur professionnel en utilisant des capteurs matériels est un moyen d’évaluer sa technique vocale.

3. PERFORMANCE PERCUE DE LA PLATEFORME D’APPRENTISSAGE DU

HUMAN BEAT-BOX (HBB)


En total

désaccord

En complet accord

1 2 3 4 5

1 Le jeu HBB est valide (s’il répond à ses objectifs)

2 Le jeu HBB est rentable (si les réponses du jeu sont satisfaisantes et en peu de temps)

3 Le jeu HBB fournit de la satisfaction (s’il fournit de la satisfaction à l’utilisateur).).

4 Le jeu HBB est innovant (si le jeu offre de nouveaux outils/techniques dans la transmission des PCI)

SECTION 8: EVALUATION GENERALE DE LA PLATEFORME WEB

1. COMPLEXITE ET CONVIVIALITE


En total

désaccord

En complet accord

1 2 3 4 5

1 Complexité: La plate-forme web n’est pas

complexe en m’aidant à comprendre, acquérir des connaissances, et obtenir des informations en ce qui concerne les questions qu'elle présente.

2 Convivialité: Il était facile d'accomplir des activités

(par exemple, les tâches de base, la récupération d'une situation antérieure après un choix d'erreur) de la plateforme web.

2. PERFORMANCE PERCUE


En total

désaccord

En complet accord

1 2 3 4 5

1 La plateforme Web est valide (si la plate-forme Web répond à ses objectifs)

2 La plateforme Web est rentable (Si la plate-forme web utilise un nombre suffisant de sources sans exagérer).

3 La plateforme Web fournit de la satisfaction (si elle fournit de la satisfaction à l’utilisateur).

4 La plateforme Web est innovante (si la plateforme Web intègre des éléments novateurs).

SECTION 9: PRESERVATION ET TRANSMISSION DU PATRIMOINE CULTUREL IMMATERIEL (PCI)


En total

désaccord

En complet accord

1 2 3 4 5

1 Je pense que les technologies i-Treasures contribuent à la préservation et à la transmission du PCI

2 Je pense que les technologies i-Treasures peuvent être facilement utilisées / adoptées pour la transmission des PCI liés au chant

3 Je pense que la plateforme i-Treasures est un outil utile à adopter par les organisations / écoles / institutions du PCI, pour les aider à promouvoir les expressions de du PCI en voie de disparition.

4 Je vais recommander l'utilisation de la plateforme i-Treasures à des collègues, des amis et la famille.

5 Je pense qu'il est utile d'utiliser les nouvelles technologies pour la capture, la préservation et la transmission des expressions du PCI traditionnel.

6 Je ne crains pas que l'utilisation des nouvelles technologies aie un effet négatif ou fragilise les expressions spontanées de formes de PCI.

7 Je pense que la plate-orme i-Treasures respecte la tradition des différentes expressions du PCI.

8 Je trouve la plateforme i-Treasures innovante.

Merci beaucoup de votre participation !

APPENDIX IX 17.1 Questionnaire for the evaluation of the Web Platform and Cantu in Paghjella use

case

QUESTIONNAIRE

SECTION 1: DETAILS CONCERNANT LA PERSONNE REPONDANTE

1. Sexe: Masculin 1 Féminin 2

2. Age (en années): ..........................................

3. Niveau d’éducation

(entourer le plus élevé):

4. Familiarité dans l’utilisation des ordinateurs:

Presque inexistante

Peu importante


1 2 3 4 5

5. Expérience en chant (Rock, jazz, pop …):



1 2 3 4 5

6. Expérience dans le Cantu in Paghjella :



1 2 3 4 5

6.1 Dans quelle voix ?

Primaire Secondaire Collège Université De 3ème

cycle

1 2 3 4 5

Secunda 1 Bassu 2 Terza 3

7. Statut de la personne répondante Experte 1 Utilisateur 2

8. Identité de l’utilisateur : .....................................................

9. Date: ....................................................

10. Lieu: .................................................


1. OBJECTIF DE LA PLATEFORME WEB


En total désaccord

En complet accord

1 2 3 4 5

1 La mission de la plateforme web est suffisamment établie.

2 Les objectifs de la plateforme web sont suffisamment établis.

2. QUALITE DU CONTENU INFORMATIF


En total désaccord

En complet accord

1 2 3 4 5

1 L’information fournie par la plateforme web est précise

2 L’information fournie par la plateforme web est utile en encourageant la formation continue

3. NIVEAU DE DIFFICULTE DU CONTENU INFORMATIF


En total désaccord

En complet accord

1 2 3 4 5

1 L’information fournie est trop complexe.

2 L’information fournie est trop technique.

4. ORGANISATION DU CONTENU INFORMATIF


En total désaccord

En complet accord

1 2 3 4 5

1 Le contenu informatif est organisé de manière appropriée, hiérarchiquement à l’intérieur de chaque section thématique

2 Le contenu informatif est organisé de manière appropriée en terme de présentation multimédia (son, image, vidéo, animation, et graphiques).

SECTION 3: CONCEPTION

1. ACCESSIBILITE


En total désaccord

En complet accord

1 2 3 4 5

1 La plateforme web peut être correctement visualisée par les navigateurs modernes

2 La plateforme web est suffisante pour fournir des mécanismes de contrôle d'accès (authentification en utilisant le nom d'utilisateur et mot de passe pour chaque utilisateur).

3 L’information est facilement accessible.

2. NAVIGATION


En total désaccord

En complet accord

1 2 3 4 5

1 La plateforme web fournit des mécanismes de navigation clairs.

2 La présentation du contenu est cohérente sur toutes les pages.

3 Le mécanisme de navigation permet facilement à l'utilisateur d'être en mesure de revenir à une page ou un niveau précédents, ou à la page d'accueil.

4 Il est possible pour l’utilisateur de savoir où il est sur la plateforme à chaque instant.

3. CONCEPTION DE L’INTERFACE DE LA PLATFORM WEB


En total désaccord

En complet accord

1 2 3 4 5

1 La conception de l'interface de la plateforme web est attrayante.

2 Les couleurs utilisées sur cette plateforme web sont confortables (par exemple la teinte, la netteté, la luminosité, etc.)

3 La plateforme Web ne contient pas de caractéristiques qui irritent, telles que le défilement ou le texte clignotant.

4. INTERACTION


En total désaccord

En complet accord

1 2 3 4 5

1 La plateforme web fournit des informations de feedback adéquat pour l’utilisateur

2 Il est facile d’apprendre comment utiliser cette plateforme web.

SECTION 4: FONCTIONNALITES DU SYSTEME DE GESTION DE L’APPRENTISSAGE (SGA/LMS)

1. CONCEPTION


En total désaccord

En complet accord

1 2 3 4 5

1 L’accessibilité au sein du SGA est facile et confortable

2 La navigation au sein du SGA est facile et intuitive.

3 L’interface au sein du SGA est attrayante

4 L’interaction (le feedback) au sein du SGA est adéquate.

2. PROPRIETES DES COURS


En total désaccord

En complet accord

1 2 3 4 5

1 Organisation: les contenus fournis dans le cours sont bien organisés.

2 Clarté: les contenus fournis dans le cours sont clairs.

3 Exhaustivité: les contenus fournis dans le cours sont complets.

4 Adéquation: les contenus fournis dans le cours sont fiables et appropriés.

3. RESSOURCES ET OUTILS


En total désaccord

En complet accord

1 2 3 4 5

1 J’ai trouvé les outils de présentation (par exemple SCORM, Power Point) faciles à utiliser et utiles.

2 J’ai trouvé que les ressources (par exemple audio, vidéo, vidéos annotées, liens vers des ressources externes) faciles à utiliser et utiles.

4. CONVIVIALITE


En total désaccord

En complet accord

1 2 3 4 5

1 Facilité de compréhension: il était facile de comprendre les activités dans le SGA.

2 Facilité d’apprentissage: il était facile d’apprendre les processus dans le SGA.

3 Maniabilité: Il était facile de gérer des tâches au sein du SGA.

SECTION 5: FONCTIONNALITES DE RECHERCHES SEMANTIQUES


En total désaccord

En complet accord

1 2 3 4 5

1 Il est facile de comprendre quel type de requêtes est possible à faire dans cette plateforme web et le type d'informations qui peuvent être récupérées.

2 Il est facile de comprendre le sens de chaque critère / filtre de recherche, et comment l'utiliser.

3 Il est facile d’effectuer une requête complexe (en utilisant plusieurs critères).

4 L'interface de la fonction de recherche est satisfaisante.

5 Les résultats de la recherche et leur présentation sont satisfaisants.

6 En général, il est facile d'utiliser la fonctionnalité de recherche de la plateforme Web.

7 La fonctionnalité de recherche offerte par la plateforme web est satisfaisante par rapport à d'autres platesformes de recherche bien connues.

SECTION 6: FUNCTIONALITES du TEXTE A CHANT : non disponible pour le Cantu in Paghjella

SECTION 7: FUNCTIONALITES DE L »ENVIRONNEMENT 3D SML (non disponible pour le Cantu in Paghjella)

SECTION 8: EVALUATION GENERALE DE LA PLATEFORME WEB

1. COMPLEXITE ET CONVIVIALITE


En total désaccord

En complet accord

1 2 3 4 5

1 Complexité: La plate-forme web n’est pas complexe en m’aidant à comprendre, acquérir des connaissances, et obtenir des informations en ce qui concerne les questions qu'elle présente.

2 Convivialité: Il était facile d'accomplir des activités (par exemple, les tâches de base, la récupération d'une situation antérieure après un choix d'erreur) de la plateforme web.

2. PERFORMANCE PERCUE


En total désaccord

En complet accord

1 2 3 4 5

1 La plateforme Web est valide (si la plate-forme Web répond à ses objectifs)

2 La plateforme Web est rentable (Si la plate-forme web utilise un nombre suffisant de sources sans exagérer).

3 La plateforme Web fournit de la satisfaction (si elle fournit de la satisfaction à l’utilisateur).

4 La plateforme Web est innovante (si la plateforme Web intègre des éléments novateurs).

SECTION 9: PRESERVATION ET TRANSMISSION DU PATRIMOINE CULTUREL IMMATERIEL (PCI)


En total désaccord

En complet accord

1 2 3 4 5

1 Je pense que les technologies i-Treasures contribuent à la préservation et à la transmission du PCI

2 Je pense que les technologies i-Treasures peuvent être facilement utilisées / adoptées pour la transmission des PCI liés au chant

3 Je pense que la plateforme i-Treasures est un outil utile à adopter par les organisations / écoles / institutions du PCI, pour les aider à promouvoir les expressions de du PCI en voie de disparition.

4 Je vais recommander l'utilisation de la plateforme i-Treasures à des collègues, des amis et la famille.

5 Je pense qu'il est utile d'utiliser les nouvelles technologies pour la capture, la préservation et la transmission des expressions du PCI traditionnel.

6 Je ne crains pas que l'utilisation des nouvelles technologies aie un effet négatif ou fragilise les expressions spontanées de formes de PCI.

7 Je pense que la plate-forme i-Treasures respecte la tradition des différentes expressions du PCI.

8 Je trouve la plateforme i-Treasures innovante.

Merci beaucoup de votre participation !

APPENDIX X 18.1 Questionnaire for the evaluation of the Web Platform and the Canto a Tenore use

case

QUESTIONNAIRE



2. Age (in years): .............................................



1 2 3 4 5

4. Familiarity in using

computers:

5. Experience in singing (Rock, jazz, pop …):


1 2 3 4 5

6. Experience in Canto a Tenore singing:


1 2 3 4 5

6.1 In which role?

Boghe 1 Mesu boghe 2 Bassu 3 Contra 4

7. Respondent’s ID: Expert 1 User 2

8. User ID: ........................................................

9. Date: ........................................................

10. Place: ........................................................


1 2 3 4 5

SECTION 2,3: APPENDIX II

SECTION 4: LEARNING MANAGEMENT SYSTEM (LMS) FUNCTIONALITIES (CANTO A TENORE COURSE)

A. DESIGN

1. ACCESSIBILITY OF THE LMS


Completely

disagree

Completely

agree

1 2 3 4 5

1 Accessing the Canto a Tenore course on the LMS was easy.

2 The colours used in the Canto a Tenore course on the LMS are comfortable (e.g., hue, haze, brightness, etc.)

2. NAVIGATION OF THE LMS


Completely

disagree

Completely

agree

1 2 3 4 5

1 It is easy to understand where you are at each stage of your navigation within the Canto a Tenore course.

2 It is intuitive to navigate within the Canto a Tenore course (obvious or obscured signs; clear or obscure page titles/headings; clear or obscure navigation buttons, etc.)

3. INTERFACE OF THE LMS


Completely

disagree

Completely

agree

1 2 3 4 5

1 The interface of the LMS where the Canto a Tenore course was delivered, is attractive?

2 The LMS where the Canto a Tenore course was delivered, doesn’t contain features that irritate, such as horizontal of vertical scrolling or blinking text.

4. INTERACTION OF THE LMS


Completely

disagree

Completely

agree

1 2 3 4 5

1 The LMS where the Canto a Tenore course was delivered, provides adequately feedback information to the user.

2 It is easy to learn how to use the LMS where the Canto a Tenore course was delivered.

B. CONTENTS OF THE CANTO A TENORE COURSE

1. ADEQUATENESS OF THE COURSE CONTENTS


Completely

disagree

Completely

agree

1 2 3 4 5

1 The contents provided in the course are unbiased/proper.

2. COMPLETENESS OF THE COURSE CONTENTS


Completely

disagree

Completely

agree

1 2 3 4 5

1 The contents provided in the course are complete.

3. CLARITY OF THE COURSE CONTENTS


Completely

disagree

Completely

agree

1 2 3 4 5

1 The contents provided in the course are clear.

4. ORGANIZATION OF THE COURSE CONTENTS


Completely

disagree

Completely

agree

1 2 3 4 5

1 The contents provided in the course are well organized.

C. TECHNOLOGY (RESOURCES & TOOLS) USED DURING THE COURSE

1. EASE OF USE & 2. USEFULNESS

No. Please indicate your level of agreement with each of the following statements regarding the ease of use and the usefulness of the following resources

….are easy to use

…are useful

Completely Completely

disagree agree

Completely Completely

disagree agree

1 2 3 4 5 1 2 3 4 5

1 The SCORM presentations

2 The Power Point presentations

3 The audio resources…

4 The video resources …

5 The Text to Song engine…

6 The annotated videos…

7 The links to external resources…

D. ACTIVITIES PROPOSED DURING THE COURSE

1. ACTIVITY USEFULNESS


Completely

disagree

Completely

agree

1 2 3 4 5

1 The individual study activities (based on presentations of texts, PowerPoints, videos, etc.) are useful

2 The individual practical activities (based on exercises) are useful.

3 The discussion and sharing activities are useful.

4 The assessment activities (based on tests and assignments) are useful.

SECTION 7: 3D ENVIRONMENT SML FUNCTIONALITIES (not applicable for Canto a Tenore)

SECTION 5,6,8,9: APPENDIX II

APPENDIX XI 19.1 Questionnaire for the evaluation of Walloon Game (in English)

QUESTIONNAIRE



2. Age (in years): .............................................



1 2 3 4 5



1 2 3 4 5



1 2 3 4 5

5.1 Experience in dancing Walloon traditional dances:


1 2 3 4 5


7. User ID: ........................................................

8. Date: ........................................................

9. Place: ........................................................

1. INTERACTIONAL CHARACTERISTICS OF THE WALLOON DANCE GAME-LIKE APPLICATION

1. Game Scenario


Completely

disagree

Completely

agree

1 2 3 4 5


game.



4 I think that the sequence of offered activities (i.e., parts/entities taught independently) helped me to learn how to perform traditional Walloon dance in a progressive and smooth way.



2. Visualization


Completely

disagree

Completely

agree

1 2 3 4 5






Tsamiko.





Completely

disagree

Completely

agree

1 2 3 4 5


helped me to understand when my movements were not correct and, thus, I improved/corrected my dancing performance.


accurate (i.e., it actually provided a low score when

my movements were not correct and a high score when my movements were correct).



4. Sensor Set-up


Completely

disagree

Completely

agree

1 2 3 4 5








5. Usability


Completely

disagree

Completely

agree

1 2 3 4 5









satisfactory.



Completely

disagree

Completely

agree

1 2 3 4 5



more on learning how to dance Walloon dance.




6 The game helped me to learn how to dance Walloon dance.



9 I think that the game respects the tradition of the Walloon dance.



2. PERCEIVED PERFORMANCE OF THE TSAMIKO GAME-LIKE APPLICATION


Completely

disagree

Completely

agree

1 2 3 4 5

1 The Walloon dance game is effective (if the game meets its objectives).

2 The Walloon dance game is efficient (if the game

responses satisfactorily and in a short time).

3 The Walloon dance game provides satisfaction (if the game provides satisfaction to the user).

4 The Walloon dance game is innovative (if the game offers novel tools/ techniques in ICH transmission).

APPENDIX XII 20.1 Questionnaire for for the evaluation of the ITGD tool and the Generic Game

Framework (in Greek)




2. Ηλικία (σε χρόνια): .............................................





1 2 3 4 5


Πολύ φτωχή


1 2 3 4 5

5. Εξοικείωση στο χορό:


1 2 3 4 5

5.1 Εξοικείωση στον Ελληνικό Παραδοσιακό Χορό:


1 2 3 4 5



8. Ημερομηνία: ........................................................

9. Τόπος: ........................................................

10. Χορός που καταγράφηκε: ........................................................

ΕΝΟΤΗΤΑ 2: ΚΑΤΑΓΡΑΦΗ ΜΑΘΗΜAΤΩΝ ΧΟΡΩΝ ΜΕ ΧΡΗΣΗ ΤΟΥ ΕΡΓΑΛΕΙΟΥ ΣΧΕΔΙΑΣΗΣ ΚΑΙ ΔΗΜΙΟΥΡΓΙΑΣ ΕΚΠΑΙΔΕΥΤΙΚΩΝ ΠΑΙΧΝΙΔΙΩΝ ITGD

1. Εγκατάσταση και ρύθμιση αισθητήρων



1 2 3 4 5

1 Πόσο εύκολα καταλάβατε πού θα πρέπει να στέκεστε (σε ποιο χώρο/περιοχή) όταν γίνεται καταγραφή των χορευτικών σας κινήσεων;

2 Πόσο άνετα ήταν να χορεύετε σε μια συγκεκριμένη περιοχή, έτσι ώστε να γίνεται σωστή καταγραφή από τους αισθητήρες;

3 Η τοποθέτηση των αισθητήρων επηρέασε την απόδοσή σας; Προκλήθηκε κάποια ενόχληση στον χορευτή;

4 Θα είναι εύκολο για σας να χειριστείτε τον αισθητήρα (ες) μόνοι σας; (σύνδεση με τον υπολογιστή και τοποθέτηση στη σωστή θέση ώστε να καλύπτει την περιοχή του χορού).

2. Διαδικασία καταγραφής μαθημάτων χορών με χρήση του εργαλείου ITGD



1 2 3 4 5

1 Η διαδικασία καταγραφής των κινήσεων του χορού ήταν εύκολη και κατανοητή.

2 Η διαδικασία καταγραφής των κινήσεων του χορού ήταν ευχάριστη.

3 Η χρήση του εργαλείου ITGD για την καταγραφή των κινήσεων ήταν εύκολη.

4 Στο μέλλον θα είναι εύκολο για εσάς να χειριστείτε τον εργαλείο ITGD μόνοι σας;

5 Στο μέλλον θα επιθυμούσατε να χρησιμοποιήσετε το εργαλείο ITGD για να δημιουργήσετε και άλλα εκπαιδευτικά παιχνίδια για εκμάθηση χορού.

3. Συμπληρώστε στο παρακάτω πλαίσιο, τυχόν παρατηρήσεις / βελτιώσεις / σχόλια για τη διαδικασία καταγραφής χορού με το εργαλείο ITGD.

ΕΝΟΤΗΤΑ 3: ΛΕΙΤΟΥΡΓΙΕΣ ΠΑΙΔΑΓΩΓΙΚΟΥ ΣΧΕΔΙΑΣΤΗ (PEDAGOGICAL PLANNER)

Αρ. Παρακαλείστε να αναφέρετε τον βαθμό που Καθόλου Πάρα πολύ

συμφωνείτε με καθεμία από τις ακόλουθες φράσεις 1 2 3 4 5

1 Είναι εύκολα κατανοητός ο τρόπος χρήσης του εργαλείου Παιδαγωγικού Σχεδιαστή (Pedagogical Planner).

2 Η διεπαφή του εργαλείου Παιδαγωγικού Σχεδιαστή είναι ικανοποιητική.

3 Η ποιότητα του αποτελέσματος (παιδαγωγικό πλάνο) του Παιδαγωγικού Σχεδιαστή είναι ικανοποιητική.

4 Το εργαλείο Παιδαγωγικού Σχεδιαστή είναι ένα χρήσιμο εκπαιδευτικό εργαλείο για τους δασκάλους χορού.

5 Σε γενικές γραμμές η χρήση του εργαλείου Παιδαγωγικού Σχεδιαστή είναι εύκολη.

ΕΝΟΤΗΤΑ 4: ΛΕΙΤΟΥΡΓΙΕΣ ΤΟΥ ΠΑΡΑΓΟΜΕΝΟΥ ΑΠΟ ΤΟ ΕΡΓΑΛΕΙΟ ITGD ΕΚΠΑΙΔΕΥΤΙΚΟΥ ΠΑΙΧΝΙΔΙΟΥ ΚΑΙ ΤΡΙΣΔΙΑΣΤΑΤΟΥ ΠΕΡΙΒΑΛΛΟΝΤΟΣ




1 2 3 4 5




4 Πιστεύετε ότι η ακολουθία των προσφερόμενων δραστηριοτήτων βοηθά τον χρήστη να μάθει να χορεύει τον διδασκόμενο χορό με σταδιακό και ομαλό τρόπο;

5 Οι βοήθεια/οδηγίες που προσφέρονται από τον εικονικό εκπαιδευτή ήταν χρήσιμες;



2. Απεικόνιση παιχνιδιού



1 2 3 4 5

1 Πόσο ευχάριστο βρήκατε το 3D περιβάλλον του παιχνιδιού όσον αφορά το σχεδιασμό και την

αισθητική;

2 Πόσο ικανοποιητική βρίσκετε την εμφάνιση των εικονικών χαρακτήρων (άβαταρ) του ειδικού και

του μαθητή;

3 Πόσο ικανοποιητική/ακριβής είναι η οπτικοποίηση των χορευτικών κινήσεων από τους εικονικούς

χαρακτήρες;

4 Πόσο χρήσιμη πιστεύετε ότι είναι η παρακολούθηση του εικονικού χαρακτήρα για την διδασκαλία των

βασικών βημάτων του χορού που διαλέξατε να καταγράψετε;

5 Πόσο χρήσιμη πιστεύετε ότι είναι η παρακολούθηση του βίντεο της καταγραφής, για τη διδασκαλία του

χορού;

6 Πόσο άνετη πιστεύετε ότι είναι η εξάσκηση των

χορευτικών κινήσεων και η παρακολούθηση του εικονικού χαρακτήρα ταυτόχρονα από τους μαθητές;




1 2 3 4 5


2 Πιστεύετε πως οι χρήστες θα είναι σε θέση να μάθουν πώς να παίζεται το παιχνίδι μόνοι τους (χωρίς οποιαδήποτε εξωτερική καθοδήγηση);

3 Είναι εύκολο για το χρήστη να εκτελέσει τις διάφορες ασκήσεις;

4 Είναι εύκολο για τους χρήστες να ακολουθήσουν τις κινήσεις του εικονικού χαρακτήρα και να εξασκηθούν μόνοι τους;

5 Πόσο εύκολη είναι η περιήγηση μέσα στο 3D

εικονικό περιβάλλον και η αλλαγή της θέση της εικονικής κάμερας (μεγέθυνση/σμίκρυνση, περιστροφή εικονικού χαρακτήρα);


7 Βρίσκετε ικανοποιητική τη γενική ανάδραση που

παρέχεται από το παιχνίδι;




1 2 3 4 5

1 Καθιστά αυτό το παιχνίδι την εμπειρία εκμάθησης χορού ευκολότερη σε σύγκριση με τις παραδοσιακές τεχνικές (π.χ. παρατηρώντας έναν δάσκαλο ζωντανά);

2 Πόσο πιστεύετε ότι θα βοηθούσε το παιχνίδι για να διδάξετε σε έναν αρχάριο να χορεύει το συγκεκριμένο χορό;



5 Θεωρείτε ότι το παιχνίδι σέβεται την παράδοση του χορού που επιλέξατε να διδάξετε;


7 Θα θέλατε στο μέλλον να χρησιμοποιήσετε ένα τέτοιο παιχνίδι ή παρόμοιες τεχνολογίες για να διδάξετε χορούς;

5. Καταλληλότητα χρήσης για διαφορετικούς τύπους χορού

Αρ. Παρακαλείστε να αναφέρετε το βαθμό που συμφωνείτε με καθεμία από τις ακόλουθες φράσεις


1 2 3 4 5

Για ποια είδη χορού πιστεύετε ότι είναι κατάλληλο ένα τέτοιο εκπαιδευτικό παιχνίδι;

1 Παραδοσιακούς χορούς

3 Latin χορούς (Tango, Salsa, Rumba κτλ.)

4 Μοντέρνο / σύγχρονο χορό

5 RnΒ χορό

6 Κάποιο άλλο είδος χορού

7 Ατομικός χορός

8 Χορό σε ζευγάρι

9 Χορό σε ομάδα

6. Καταλληλότητα χρήσης για διαφορετικούς τύπους ασκήσεων

Αρ. Παρακαλείστε να αναφέρετε το βαθμό που συμφωνείτε με καθεμία από τις ακόλουθες φράσεις


1 2 3 4 5

Εκτός από την εκμάθηση χορού ποιες άλλες εφαρμογές πιστεύετε ότι θα μπορούσε να έχει ένα τέτοιο παιχνίδι;

1 Δημιουργία μαθημάτων ασκήσεων εκγύμνασης

3 Δημιουργία μαθημάτων για την διδασκαλία αθλητικών δεξιοτήτων σε σπορ (π.χ. τένις, μπάσκετ, γκολφ κτλ.)

4 Δημιουργία μαθημάτων για τη διδασκαλία πολεμικών τεχνών

5 Δημιουργία μαθημάτων ασκήσεων για αποκατάσταση τραυματισμών

6 Δημιουργία μαθημάτων για μουσικο-χορευτική αγωγή παιδιών

7 Κάποιο άλλο είδος κινήσεων:

7. Συμπληρώστε στο παρακάτω πλαίσιο, τυχόν παρατηρήσεις / βελτιώσεις / σχόλια για τo παραγόμενο ηλεκτρονικό εκπαιδευτικό παιχνίδι.

ENOTHTA 5: ΑΠΟΔΟΣΗ ΤΗΣ ΕΚΠΑΙΔΕΥΤΙΚΗΣ ΤΡΙΣΔΙΑΣΤΑΤΗΣ ΕΦΑΡΜΟΓΗΣ ΓΙΑ ΤΟ ΧΟΡΟ ΠΟΥ ΚΑΤΑΓΡΑΨΑΤΕ



1 2 3 4 5








εργαλεία/τεχνικές για τη μετάδοση/εκμάθηση της Άυλης Πολιτιστικής Κληρονομιάς)

ΕΝΟΤΗΤΑ 6: ΔΙΑΦΥΛΑΞΗ ΚΑΙ ΜΕΤΑΔΟΣΗ ΤΗΣ ΑΥΛΗΣ ΠΟΛΙΤΙΣΤΙΚΗΣ ΚΛΗΡΟΝΟΜΙΑΣ (ΑΠΚ)



1 2 3 4 5

1 Πόσο νομίζετε ότι τα εκπαιδευτικά εργαλεία του i-Treasures (Παιδαγωγικός Σχεδιαστής και Εργαλείο Σχεδίασης και Δημιουργίας Εκπαιδευτικών Παιχνιδιών) και το εκπαιδευτικό υλικό που μπορεί να δημιουργηθεί με αυτά (εκπαιδευτικά μαθήματα και εκπαιδευτικά 3Δ παιχνίδια) συμβάλλουν στη διατήρηση και τη μετάδοση της ΑΠΚ;

2 Πιστεύετε ότι τα εκπαιδευτικά εργαλεία του i-Treasures και το παραγόμενο εκπαιδευτικό υλικό μπορούν εύκολα να χρησιμοποιηθούν/ υιοθετηθούν για τη μετάδοση εκφράσεων της ΑΠΚ σχετιζόμενων με το χορό;

3 Πιστεύετε ότι τα εκπαιδευτικά εργαλεία του i-Treasures και το παραγόμενο εκπαιδευτικό υλικό μπορούν να υιοθετηθούν από οργανισμούς/ σχολεία / ιδρύματα ΑΠΚ ώστε να τα βοηθήσουν στην προώθηση απειλούμενων με εξαφάνιση εκφράσεων ΑΠΚ;

4 Θα προτείνατε στους συναδέλφους, τους φίλους και την οικογένειά σας τη χρήση των εργαλείων του i-

Treasures;

5 Πώς αισθάνεστε αναφορικά με τη χρήση νέων τεχνολογιών για τη σύλληψη, τη διατήρηση και τη μετάδοση των παραδοσιακών εκφράσεων ΑΠΚ; Πιστεύετε ότι είναι χρήσιμες;

6 Φοβάστε ότι η χρήση των νέων τεχνολογιών θα επηρεάσει αρνητικά ή θα υπονομεύσει τις αυθόρμητες εκφράσεις μορφών της ΑΠΚ;

7 Πιστεύετε ότι το εκπαιδευτικό υλικό που παράγεται από τη χρήση των εργαλείων του i-Treasures σέβεται την παράδοση των διαφορετικών εκφράσεων ΑΠΚ;

8 Πόσο καινοτόμα βρίσκετε ότι είναι τα εκπαιδευτικά εργαλεία του i-Treasures;

APPENDIX XIII 21 Questionnaire for the for the Overall Assessment of the System (Based on Non-

Functional Requirements)

QUESTIONNAIRE

(Source: Based on Dagnino, F.M., Hadjileontiadis, L.J, Ott, M., and Pozzi, F. (2014) An integrated platform supporting intangible cultural heritage learning and transmission:

Definition of Requirements and evaluation criteria. Journal of Computing and Information Technology, 22(4): 277-292)



2. Age (in years): .............................................



1 2 3 4 5

4. Perceived familiarity in using computers:


1 2 3 4 5

5. Respondent’s ID: Expert 1 Member of Focus Group 2

6. Date: ........................................................

7. Place: ........................................................

SECTION 1: COST OPTIMALITY / ENERGY EFFICIENCY


Completely

disagree

Completely

agree

1 2 3 4 5

1.1 The infrastructure (e.g. servers, ups units, broadband services etc.) of the i-Treasures platform is cost- and energy-efficient.

1.2 The maintenance of the infrastructure is cost-efficient.

1.3 It is affordable for the single user – institution to acquire the necessary hardware equipment for the i-Treasures application (e.g. sensors, recording devices).

1.4 The user needs to install on-fee third party applications for the system to function properly.

1.5 The proper functioning of the i-Treasures platform requires updated hardware purchase (e.g. personal computer with new processor, amount of RAM etc.).

SECTION 2: ACCESSIBILITY / USABILITY


Completely

disagree

Completely

agree

1 2 3 4 5

2.1 The content of the i-Treasures is presented in different forms (auditory, text, visual elements).

2.2 The web- platform content is rendered properly across different browsers.

2.3 The i-Treasures web platform is responsive.

2.4 The web-pages are accessible even when newer web technologies are not available (e.g. JavaScript, CSS3 style-sheets).

2.5 The i-Treasures platform supports different languages.

2.6 The platform provides clear navigation mechanisms.

2.7 The platform supports simple and advanced search.

2.8 The content presentation is consistent across pages.

2.9 The platform provides feedback on connected hardware configuration.

2.10 Response times for task accomplishment (e.g. application loading, refresh time, search results etc.) are proper and adequate.

2.11 The i-Treasures web platform complies with the universal access standards.

2.12 The i-Treasures web platform allows for personalization.

SECTION 3: DOCUMENTATION / SUPPORT


Completely

disagree

Completely

agree

1 2 3 4 5

3.1.1 There is a complete, accurate and clear documentation accompanying the i-Treasures platform.

3.1.2 The assimilation of the documentation requires background or expertise from the users.


NO YES

0 1

3.2.1 The documentation is partitioned into sections for users, developers and administrators.

3.2.2 There are links to supporting information resources.

3.2.3 There is a “getting started” or a “how to” guide for different use cases.

3.2.4 The documentation is available in the i-Treasures website.

3.2.5 There is a web page (e.g. FAQs), a forum or e-mail lists for additional support.

SECTION 4: INTEROPERABILITY / PORTABILITY


Completely

disagree

Completely

agree

1 2 3 4 5

4.1 The platform components/modules are compatible with third party services.

4.2 The platform functions on different and commonly available operating systems (OSs).

4.3 The downloadable content is compatible with the different OSs.

4.4 Besides personal computers, the platform is compatible with tablets or other mobile devices.

SECTION 5: EXTENSIBILITY / SCALABILITY


Completely

disagree

Completely

agree

1 2 3 4 5

5.1 The core/custom components of the i-Treasures platform are modular.

5.2 Backward compatibility is taken into account during upgrades.

5.3 The system can be remotely managed.

5.4 The server deployment allows for horizontal scalability.

SECTION 6: AUDITING


Completely

disagree

Completely

agree

1 2 3 4 5

6.1.1 The auditing system is centralized and secured.


NO YES

0 1

6.2.1 The i-Treasures platform provides reports on user activity (per application) and general user behavior

6.2.2 The platform supports auditing of failed login attempts in order to detect brute force attacks.

SECTION 7: SECURITY / PRIVACY


Completely

disagree

Completely

agree

1 2 3 4 5

7.1.1 User data are transferred and stored securely (e.g. use of encryption algorithms, HTTPS etc.)?

7.1.2 User data are properly available.

7.1.3 The system user authentication takes measures in cases of misuse (lost or stolen passwords, account locks).


NO YES

0 1

7.2.1 The i-Treasures platform has an authentication system

7.2.2 The i-Treasures database supports role-based access control based on user privileges.

SECTION 8: FAULT TOLERANCE / RECOVERABILITY


NO YES

0 1

8.1.1 There is a software failure monitoring procedure (e.g. error logs)

8.1.2 The architecture of the platform allows for autonomous functioning of services when a module fails and is being disabled

8.1.3 The platform is functional in cases when external web dependencies (e.g. Europeana platform) go offline


Completely

disagree

Completely

agree

1 2 3 4 5

8.2.1 Hardware maintenance and software upgrades are planned in a systematic way.

8.2.2 The system includes regular backups of system components and images.

8.2.3 System backup images can be restored on different hardware.

SECTION 9: LICENSING / COPYRIGHT


NO YES

0 1

9.1 An appropriate license has been adopted.

9.2 The type of license is clearly stated in all platform aspects (e.g. website, source codes).

9.3 Platform components do include a copyright statement

9.4 It is clearly stated who funded the project, developed the platform and owns the copyright

9.5 The platform has a trademark, that doesn’t violate other trademarks.

APPENDIX XIV 22 Questionnaire for the Assessment of the Main Technical Indices

(Based on Functional Requirements)

QUESTIONNAIRE

(Source: Based on Dagnino, F.M., Hadjileontiadis, L.J, Ott, M., and Pozzi, F. (2014) An integrated platform supporting intangible cultural heritage learning and transmission:

Definition of Requirements and evaluation criteria. Journal of Computing and Information Technology, 22(4): 277-292)



2. Age (in years): .............................................



1 2 3 4 5

4. Perceived familiarity in using computers:


1 2 3 4 5

5. Respondent’s ID: Expert 1 Member of Focus Group 2

6. Date: ........................................................

7. Place: ........................................................

SECTION 1: FACIAL EXPRESSION ANALYSIS


Completely

disagree

Completely

agree

1 2 3 4 5

1.1 Facial feature tracking accuracy was acceptable.

1.2 Facial Action Unit (AU) recognition accuracy was acceptable.

1.3 Basic emotion recognition accuracy was acceptable.

SECTION 2: HUMAN BODY MOTION AND GESTURE RECOGNITION


Completely

disagree

Completely

agree

1 2 3 4 5

2.1 Quality of Human Body Motion and Gesture Recognition structure was acceptable.

2.2 Human Body Motion and Gesture recognition Accuracy was acceptable.

2.3 Sensitiveness of the Human Body Motion and Gesture recognition was acceptable.

2.4 Performance identification accuracy was acceptable.

SECTION 3: ELECTROENCEPHALOGRAPHY ANALYSIS


Completely

disagree

Completely

agree

1 2 3 4 5

3.1 Quality of EEG recording was acceptable.

3.2 Affective Sate Detection Efficiency was acceptable.

3.3 Computational Time was acceptable.

SECTION 4: VOCAL TRACT SENSING AND MODELLING – ULTRASOUND ANALYSIS


Completely

disagree

Completely

agree

1 2 3 4 5

4.1 Quality of US pictures was acceptable.

4.2 Tongue contour extraction (Validation error; Mean Square of Distance between manual contour used as ground truth and automatically extracted contour) was acceptable.

SECTION 5: SOUND PROCESSING


Completely

disagree

Completely

agree

1 2 3 4 5

5.1 Individual drums sound recognition Accuracy was acceptable.

5.2 Fundamental Frequency (F0) Analysis was acceptable.

5.3 Respiration (Inhalation detection F-score; Recognition accuracy for egressive and ingressive sounds) was acceptable.

5.4 Voice Tone (Modulation index to suggest emotions) was acceptable.

5.5 Special Vocal Effect (Spectral distortion level; Nonlinearity Indices; Fractal Dimension) was acceptable.

5.6 Doubling of Period (Detection of vocal and ventricular folds simultaneous vibration) was acceptable.

5.7 Onset Detection Analysis (Onset detection F-score: it is computed when considering as false the data falling outside a given threshold (typically of 25 ms)) was acceptable.

SECTION 6: TEXT-TO-SONG


Completely

disagree

Completely

agree

1 2 3 4 5

6.1 Supported Inputs (Ability to read a musical score with lyrics; Ability to read a byzantine musical score; Ability to read a MIDI file with lyrics; Number of supported languages) were acceptable.

6.2 Musical Indicators (Number of supported vocal techniques; Number of available voices; Number of supported singing styles) were acceptable.

6.3 Technical Indicators (System latency (time lapsed from reading the score until the first sample is output)) were acceptable.

SECTION 7: ONTOLOGY ENGINEERING

No.

Ontology assessment in terms of:

Completely

disagree

Completely

agree

1 2 3 4 5

7.1 Class tree depth, was acceptable.

7.2 Class tree breadth, was acceptable.

7.3 Tree branching factor was acceptable.

7.4 Attribute richness: number attributes in all classes

/ number classes, was acceptable.

7.5 Relationships richness: number relations / (number subclasses+ number relations), was acceptable.

7.6 Class richness: number class used/ number class defined, was acceptable.

7.7 Average population: number instances/ number classes, was acceptable.

7.8 Cohesion: number separate relationship graph components, was acceptable.

7.9 Importance: number instances of class and subclasses / number total instances, was acceptable.

7.10 Inheritance richness of a class C: average of subclasses per class that are descendants of C.

7.11 Relationship richness of a class C: number instances of relations (properties) of C with another class / number relations including C, was acceptable.

7.12 Connectivity of a class C: number instances of other classes connected to instances of that class via any relationship, was acceptable.

SECTION 8: EDUCATIONAL PLATFORM


Completely

disagree

Completely

agree

1 2 3 4 5

8.1 Multi-method learning (Availability of learning materials in various formats (audio, video, text, images, etc.); Availability of testing tools (e.g., quizzes)); Availability of Text-to-Song tool; Availability of video annotation tool; Availability of annotated musical score), was acceptable.

8.2 Learning process adjustment (Possibility to set/choose topics; Possibility to set /choose difficulty levels; Possibility to set/choose contexts; Possibility to offer recovery activities according to responses to quizzes; Availability of tracking functionalities), was acceptable.

8.3 Users’ interactions (Possibility to interact with others; Possibility to work/learn in group), were acceptable.

8.4 Performance decomposition (Availability of musical instruments separation; Availability of voices separation), was acceptable.

8.5 Text to song functionality (Availability of entering marks and lyrics; Availability of entering modern musical annotation), was acceptable.

APPENDIX XV 23. Final Technical assessment reports

This appendix presents the final technical assessment reports for the modules that were assessed during Task 7.2 “Laboratory Testing of Module”, i.e.:

Body & Gesture Recognition (regarding Tsamiko sub use-case)


Body & Gesture Recognition (regarding Walloon sub use-case)


EEG Analysis (regarding Contemporary Music Composition use-case)


Text-to-Song (regarding Singing use-case)

Vocal Tract Sensing and Modeling (regarding Singing use-case)

Facial Expression Analysis (regarding Singing use-case)

Multimodal Data Fusion & Semantic Analysis (regarding Tsamiko and Salsa sub use-cases).

Body & Gesture Recognition (regarding Tsamiko sub use-case) Test Leader: CERTH

Summary

This report presents the technical assessment results of BGDCAM (Body and Gesture Data Capture and Recognition Module). The features that were tested are related to the analysis of the body motion for the Tsamiko dance sub-use case. They include the error rates in percentages of step segmentation and error rates of the features extracted from each step. Those error percentages were deduced by measuring the total number of divergences from the correct Tsamiko choreography produced by the analysis module. The module was tested on three dance sequences performed by three different expert performers (two male and one female). Both single step and double step dance variations were performed in order to cover all styles: single step, double step and female.

Results

Assessment Indices results

Assessment Category

Assessment Indices

ID Description

Desirable Value or Fail/Pass criteria

Value

Step features error rates

Foot_err Active foot detection error rate (numerical value [0,1])

<0.025 0.018

Direction_err Direction of movement detection error rate (numerical value [0,1])

<0.025 0.001

Raised_err Raised foot detection error rate (numerical value [0,1])

<0.05 0.038

Double_err Double step detection error rate (numerical value [0,1])

<0.05 0.014

Step segmentation error rate

Segmentation_err Percentage of erroneous dance steps segmentations (numerical value [0,1])

<0.15 0.053

Comments:

- The above results correspond to the offline analyser implemented in MATLAB.

Evaluation of Results

The performance of the BGDCAM module regarding the offline analysis of the Tsamiko dance is satisfactory. The overall Global Precision of the module as calculated by the FIS structure is 0.89 while the GP for a corresponding baseline system is 0.55. These results apply to the offline test scenario as described above.

The above results show that the precision of the detection is very close to the baseline values. The step segmentation error rate is satisfactory (5%) and improved compared to the previous TAR (see D7.2). In addition, all feature detection error rates are below 5%, which is a very robust result.

In closing, the behavior of the module has been improved compared to the previous TAR (see D7.2) due to the use of data generated by the Kinect 2 sensor instead by the Kinect 1. The Kinect 2 sensor’s overall accuracy of joint positions in Kinect 2 is higher than in Kinect 1.


Test Leader: CERTH

Summary

This report presents the technical assessment results of SALSASRM (Salsa Step Recognition Module). The module was evaluated with respect with the ability to recognize (classify) correctly Salsa steps. Thus, the features that were tested are the step classes. The indices are two: accuracy and F-measure, which provide a measure for the efficiency of the classification prowess. The module was tested on about 600 Salsa steps of 20 classes.

Results


Assessment Category

Assessment Indices

ID Description


Value

Step classification indices

Accuracy Percentage of steps classified correctly in the testing set

>0.50 0.71

Fmeasure Index that measures the classification efficiency in terms of precision and recall metrics. These two metrics, precision and recall, are proportional to the percentage of true positive classification results in testing.

>0.30 0.47


The performance of the SALSASRM module regarding the Salsa step classification was proved to be satisfactory and improved with respect to the results provided in the previous report, (especially for the accuracy metric). The overall accuracy of the module as calculated by the FIS structure is 0.71 while the GP for a corresponding baseline system is 0.50 (given that there are 20 possible classes of steps to be recognized). Note that, even the maximum precision score is 1, this result is clearly sufficient to provide useful results to the semantic analysis module (MDFSA), since the latter can treat the inaccuracies efficiently. The above results show that the classification of the steps is a procedure of sufficient accuracy, and also, in such a degree that makes efficient the usage of the Salsa step classification

results as features to the MDFSA module (WP4 multimodal semantic analysis and data fusion).

Body Recognition (regarding Walloon sub use-case)

Test Leader: UMONS

Summary

This report presents the technical assessment result for the current version of the sensor-dependent gesture recognition module. This module aims at recognizing gestures captured by a Microsoft Kinect (low-cost sensor) where the training dataset was captured using a high precision motion capture system. Hidden Markov Models (HMMs) and an adaptation procedure based on the Maximum Likelihood Linear Regression (MLLR) algorithm were used to achieve this objective. The module was tested on a Walloon dance sequences of one person performing four different steps. These steps were repeated 30 to 40 times with an average duration of 1.3s for each step. 70% of the data was used for training where the remaining 30% was used for test.

Results


Assessment Category

Assessment Indices

ID Description


Value

Gesture recognition accuracy

Acc Overall classification accuracy (% value [0, 100])

>75 84.48

Comments:

To measure the performance of this system, we compare the recognition accuracy against a system that works under the same situation but including no adaptation procedure.


The performance of this module is very satisfactory. The system is able to recognize Walloon dance steps captured by a Kinect V2 with an accuracy of 84.48%, where the models are created with a different motion capture system (Qualisys optical motion capture system). The overall Performance Index (PI) of the module is then equal to 84.48%, given by the FIS structure, while the PI for a baseline system including no adaptation is 75%.

This result highlights the satisfactory recognition of dance gestures using our module. The focus in the future work will involve improving the classification performance, increasing the number of gestures and testing the algorithm with a larger number of subjects.


Test Leader: UOM

Summary

This report presents results with respect to the testing of the Body and Finger Recognition performance for Contemporary Music Composition use case. The module aims at recognizing the musical gestures of the expert (pianist) based on:

finger positions extracted from Leap Motion Controller and

joint positions for upper body extracted from Kinect X-Box 360

(additionally) Euler angles for upper body extracted from Animazoo motion capture suit.

Estimated assessment indices include the Precision, Recall and Discriminability for Finger and Body Recognition Performance respectively. The module was tested for 3 musical gestures which were included in the musical vocabulary:

1) ascending scale performed in legato style, 2) descending arpeggio performed in staccato style, and 3) a small excerpt from a famous Greek song.

In total 6 pianists participated performing 3 musical gestures, 5 iterations for each gesture. Therefore, a database of 90 gestures examples was created.

The evaluation method that was used is jackknife and concluded that the model satisfactory recognizes performance, verifying that it can be used in real demonstrations. Three metrics have been used to evaluate the system:

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =# 𝑜𝑓 𝑇𝑟𝑢𝑒_𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛𝑠

# 𝑜𝑓 𝑇𝑟𝑢𝑒_𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛𝑠 +# 𝑜𝑓 𝐹𝑎𝑙𝑠𝑒_𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛𝑠

𝑅𝑒𝑐𝑎𝑙𝑙 =# 𝑜𝑓 𝑇𝑟𝑢𝑒_𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛𝑠

# 𝑜𝑓 𝑇𝑟𝑢𝑒_𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛𝑠 +# 𝑜𝑓 𝑀𝑖𝑠𝑠𝑒𝑑_𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛𝑠

𝐷𝑖𝑠𝑐𝑟𝑖𝑚𝑖𝑛𝑎𝑏𝑖𝑙𝑖𝑡𝑦 = 𝐿(𝐴𝑐𝑡𝑢𝑎𝑙𝑙𝑦𝑃𝑒𝑟𝑓𝑜𝑟𝑚𝑒𝑑_𝐺𝑒𝑠𝑡𝑢𝑟𝑒) − 𝐿(𝑃𝑎𝑖𝑟_𝐺𝑒𝑠𝑡𝑢𝑟𝑒)

Variances

The data acquired from all the sensors were normalized to take values from [-1, 1], for the algorithm to process them.

Results


Assessment Category

Assessment Indices

ID Description


Value

Finger Recognition Performance

PrecisionFin

Precision=#true-positive/(#true-positive+#false-positive)

Precision is high when less false positives appear in the classification and low in the other case (numerical value [0,1])

≥ 0.70 0.96

RecallFin

Recall=#true-positive/(#true-positive+#false-negative)

Recall is high when less negative positives appear in the classification, while a lower precision corresponds to more negative positives (numerical value [0,1])

≥ 0.70 0.96

Discrimina-bilityFin

Discriminability: L(ActuallyPerformed_Gesture) – L(Pair_Gesture). Discriminability is high when the likelihood of the actually performed gesture (the one we intend to recognize) is much bigger than the likelihood of the pair gestures of the dataset that we use for the Jackknife. (numerical value [0,1])

≥0.30 0.65

Body Recognition Performance

PrecisionBod

Precision=#true-positive/(#true-positive+#false-positive)


≥ 0.70 0.97

RecallBod

Recall=#true-positive/(#true-positive+#false-negative)


≥ 0.70 0.95

Discrimina-bilityBod

Discriminability: L(ActuallyPerformed_Gesture) – L(Pair_Gesture). Discriminability is high when the likelihood of the actually performed gesture (the one we intend to recognize) is much bigger than the likelihood of the pair gestures of the dataset that we use for the Jackknife. (numerical value [0,1])

≥0.30 0.5


The performance of the updated BGRec module is better than the previous version, as it can predict body movements, fingers and therefore musical gestures with 91% overall accuracy, while the previous version of the module had 73% overall accuracy. Thus, the overall Performance Index (PI) of the current module as calculated by the FIS structure is 0. 91.

The main strength of the current version of the module is the implementation of the confidence bounds. By implementing them, the algorithm is now able to recognize expressive variations that might take place between the learner’s and expert’s performance in contrast to constant values that are used for tolerance in the previous version of the module. The better results of the recognition are proving the above. More specifically:

for the finger recognition the precision is 96% and the recall is 96% (in contrast to 95% of the precision and recall by evaluating the previous version)

for the body recognition the precision is 97% and the recall is 95% (in contrast to 96% of the precision and 93% of the recall by evaluating the previous version)

Moreover, the above high recognition results resulted to a more fluid and immediate gesture sonification, because the new synthesized signal was much closer to the template sound. Additionally, because of the fact that the recognition was more stable, less oscillations occurred between the different classes in recognition.

The major weakness of the BGRec module is that the computation of the confidence bounds is off-line resulting to a necessary offline pre-processing step before the real-time and online recognition take place. This is our focus of future work, to implement the whole process of computing the confidence bounds online.

Summarizing, the above results show that the precision of the recall for Finger and Body Recognition Performance are much higher than the baseline values (above 90%). This shows that the BFRec module is very satisfactory as it can predict body movements and fingers with high accuracy. The discriminability value is also above the baseline value validating the model.

EEG Analysis

Test Leader: AUTH

Summary

This report presents the technical assessment results for the final version of the EmoRecon module. The latter module aims at recognising the affective state of the user based on EEG signal analysis and classification. Estimated assessment indices include the number of affective states detected, the average classification accuracy of each state, the average overall classification accuracy, the required computational time, the average EEG signal quality, and the time window required for affective state detection. The module was tested for two (positive/negative) and four (positive-low/positive-high/negative-low/negative-high) emotional states detection. In both cases, the EEG datasets used for classification were acquired during the overall Data Collection phase of the project based on affective images (20 subjects × 15 images × 4 affective states × 5 s stimulus duration-EEG dataset), affective sounds (12 subjects × 6 sounds × 4 affective states × 6 s stimulus duration-EEG dataset) and affective videos (7 subjects × 6 videos × 4 affective states × 8-12 s stimulus duration-EEG dataset) emotion-elicitation procedures. The Emotiv EPOC (Emotiv Inc.) 14-channel EEG headset was used for the recordings and the average signal quality was computed as the mean signal quality outputted by the headset for the 14 channels and for all subjects, during the two emotion-elicitation procedures.

Variances

After the EEG data collection procedure, certain subjects reported that they were not concentrated and did not evaluate the self-experienced emotional state honestly or correctly. Consequently, results from these subjects were not taken into account.

Results


Assessment Category

Assessment Indices

ID Description


Value

Affective State Detection Efficiency

Acc Overall classification accuracy (numerical value [0,1])

>0.5 0.7551

LLAcc Classification accuracy of the negative valence-low arousal state (numerical value [0,1])

>0.5 0.7249

LHAcc Classification accuracy of the negative valence-high arousal state (numerical value [0,1])

>0.5 0.7448

HLAcc Classification accuracy of the positive valence-low arousal state (numerical value [0,1])

>0.5 0.7751

HHAcc Classification accuracy of the positive valence-high arousal state (numerical value [0,1])

>0.5 0.7895

NumOfSt Number or affective states detected (discrete value {2,4,6,8})

≥4 4

Quality of EEG recording

EEGQual Quality of EEG data as recorded by Emotiv EPOC device (numerical value [1,4])

≥3.2 3.6

Computational Time

DetTime The computational time required by the module to predict the emotional state (seconds, [0,1])

≤0.25 0.072

Emotional Time Window

WinLen The time window during which the EEG data are collected. When the time window is finished, the data are fed to the module for analysis (seconds, [0,15])

≤7 4

Comments:

- The above results correspond to the four-class problem.

- Classification accuracies are the average accuracies derived from the image and sound elicited EEG datasets.

- During the two affective states detection test the assessment indices corresponding to negative-low (LLAcc) and positive-low (HLAcc) emotional states were not taken into account.


The performance of EmoRecon module is quite satisfactory as it can predict four emotional states with 75% average accuracy while the correct prediction rate for each individual emotional state varies from

72% to 79%. These results may not seem exceptional but they are significantly higher than the 25% chance level of the four class classification problem, given the difficulty of the problem. The computational time is 0.072 s, significantly less than the acceptance threshold, while the required time window is set to 4 s which is passable for a real-time application. Compared to the previous version of EmoRecon, the final version exhibits higher overall classification accuracy (approximately 10%), as well as higher EEG data quality and shorter computational time (approximately 15%). The overall Performance Index (PI) of the module as calculated by the corresponding FIS structure is 0.76 while the PI for the baseline system is 0.31. Compared to the PI of the previous version of EmoRecon, the improvement of the overall PI is only 1%. This is attributed to the fact that the number of recognized affective states (i.e., four), and the time window during which the EEG data are collected (i.e., 4 sec.) have remained the same and these two variables play significant roles to the rule base of this fuzzy system. However, the performance index of the detection efficiency evaluation fuzzy sub-system has increased approximately from 67.5% to 74%.


Test Leader: ARMINES

Summary

This report presents results with respect to the testing of segmenting depth maps of scenes where humans interact with objects by using gestures in the Pottery use case. The algorithm developed, based on Random Decision Forests, aims at recognizing the scene segments. We extend the typical application of RDFs, by introducing the Hierarchical Random Decision Forests (HRDFs). More precisely, we use three RDFs in a tree structure architecture. The parent RDF is used to create a rough initial segmentation of the scene, while the two children RDFs are used to further classify the regions of the left and right arm, hand and fingers respectively. More precisely we use in total 11 clusters or region of interest that were manually annotated. These clusters are:

Body

Wheel

Clay

Right Arm

Right Palm

Right fingers

Right thumb

Left arm

Left palm

Left fingers

Left thumb

For the use case of pottery, we calculated the accuracy of the classification by applying the algorithm to 20 images, which were used as testing set. The accuracy was computed for the whole HRDF architecture for all the regions of interest of the wheel-throw art of pottery.

Accuracy=(sum(true_positives)+sum(true_negatives))/sum(total_population).

Results


Assessment Category

Assessment Indices

Description ID


Value

Accuracy Recall=#true-positive/(#true-positive+#false-negative)


Accuracy=(sum(true_positives)+sum(true_negatives))/su

m(total_population).

Accuracy is high when a high number of positive detection appears while alower accuracy corresponds to more negatives

Body

≥ 0.60 0.9361

Wheel ≥ 0.60 0.9699

Clay

≥ 0.60 0.7153

Right Arm

≥ 0.60 0.8508

Right Palm

≥ 0.60 0.6460

Right fingers

≥ 0.50 0.5445

Right thumb

≥ 0.50 0.4634

Left arm

≥ 0.60 0.8530

Left palm

≥ 0.60 0.6946

Left fingers

≥ 0.50 0.5010

Left thumb

≥ 0.50 0.6201


Both the body of the potter and the wheel have the highest accuracy scores which is above 90%. The least accuracy can be found for the right thumb, which equals to 46%. In he classification of both hands, both arms have the highest accuracy and the second highest is the one of both palms. The fingers and the thumb are most of the time occluded by the clay, or are self occluded. This seems to be the reason that thumbs and fingers in both hands, have low detection accuracy than the other regions of interest. The main strength of the module is the fact the HRDF architecture provides accurate enough results for all the segmented regions of interest in a complicated scene, as the one of pottery.

The main weakness is linked to occlusions. Since the method used is based on computer vision techniques, occlusion issues are still important. When the potter puts his hands into the clay, the system can not easily detect them.

Text-to-Song

Test Leader: ACAPELLA

Summary

This report presents the technical assessment results of Singing Voice Synthesis module. The module is intended to synthesize songs in the studied traditional singing sub-use cases. The nature of the system output makes the evaluation highly subjective, and we will include human subjective scoring in following evaluations. The system was tested for the implemented capabilities and for speed. The capabilities that were tested were the system’s ability to read required inputs, its ability to synthesize different vocal techniques, and its timely response. The detailed list of the evaluated features is shown below. The time measurements were performed on 100 runs of the system, using different lyrics.

Results


Assessment Category

Assessment Indices

ID Description


Value

Supported inputs

MS Ability to read a musical score with lyrics (binary value {0,1})

1 1

BMS Ability to read a byzantine musical score (binary value {0,1})

1 1

MIDI Ability to read a MIDI file with lyrics (binary value {0,1})

1 1

LNG Number of supported languages (discrete numeric values {0,..,4})

>3 3

Musical VTH Number of supported vocal techniques (numerical discrete [0,Inf) )

>4 5

V Number of available voices (numerical discrete {0,…,10})

>8 6

STY Number of supported singing styles (numerical discrete {0,…,3})

>3 3

Computational Time

LT Computing time (ratio between time elapsed to synthesize a song, and the song's duration; numerical [0, 10])

≤0.25 0.16

Comments:

- The computational time presented is the mean of the values obtained by running the program on 100 different verses


After implementing the Byzantine sub-use case, and finalizing the Canto a Tenore system, the system’s capabilities have increased significantly, pushing the Performance Index to a good value. Eventhough we are now sporting a more complex vocoder, the Computational Time remained in reasonable limits.

The overall Performance Index (PI) as calculated by the FIS structure is 0.97.

The results obtained so far show a good behavior of the system, predicting a high performance of the system once all the sub-use cases will be implemented.

Vocal Tract Sensing and Modeling

Test Leader: UPMC

Summary

This report presents the technical assessment results of the TongueContour Extraction module. This module aims at extracting tongue contour from ultrasound images. The features that are tested include the quality of ultrasound recording (dataQuality), computing time (elapsedTime) and a contour quality evaluation (contourQuality) based on the Mean Sum of Distances (MSD) between the extracted contour and the contour extracted manually. As stated in the previous deliverable D7.2, we have tested a new contour extraction algorithm that does not require a heavy learning phase as the one based on deep learning. Moreover, it intends to be mostly speaker-independent. It has been tested on data recorded from 5 speakers with the same equipment as previously. Details on the algorithm can be found in (Robust contour tracking in ultrasound tongue image sequences, Clinical Linguistics and Phonetics, 2016, vol.30, Issue 3-5).

Variances

In the first version of the software, the contour was extracted by a machine learning method where each image in the training set was associated to a contour. During the extraction procedure, the input image was also reconstructed along with the image of the contour. In the new version, the contour is extracted in a more standard way thanks to a snake and does not rely on an autoencoder. So there is no image at the output, but only the coordinates of the points of the contour in the image. The MSError was an indicator of the quality of the autoencoder, but it is pointless now.

Results


Assessment Category

Assessment Indices

ID Description


Value

Quality of ultrasound recording

dataQuality Ultrasound data quality (numerical value [0,1])

>0.5 0.6

Tongue Contour Quality

MSD Mean Sum of Distances between the extracted contour and a manually labelled contour

<0.8 0.7

Computational Time

elapsedTime The computational time required by the module to extract tongue shape (seconds)

<0.5 0.9

Comments:

- DataQuality scores correspond to a qualitative appreciation of the quality of ultrasound recording

- Common MSD values are between 2 and 5 pixels. 2.5 pixels is an average value in literature. We choose the equivalence: 2.5 px = 0.5 – 5 px = 1. 0.5 corresponds to a target score and 1 corresponds to a poor score.

- elapsedTime was fixed according to the computation time of an algorithm that was used as an upper boundary. Its performance of 3 fps was given the score 1 and elapsedTime value is actually given by the ratio between computing time and this upper boundary.


The performance of the Tongue Contour Extraction module is reasonably satisfactory. The main improvement of the new algorithm lies in the fact that it does not require a training phase and that it can be used on any speaker. The computation time still exceeds the required desirable value, but it can be significantly reduced (by a factor 2/3) if one accepts to lose some robustness. Contour quality scores are acceptable. The extracted contours are generally very close to the manually extracted

contour, but the score is lowered by outliers. It should be noted that the ultimate target of this module is to drive the 3D tongue model of the Human Beat Box (HBB) game. Extensive testing has shown good performances and robustness of the HBB game with respect to the data acquisition, audio, video and ultrasound recording as well as communications between the local computers and the remote scoring module in UMONS. However, the animation of the 3D tongue model was highly sensitive i) to the position of the tissue points extracted from the contour of the ultrasound image and ii) to the values of the parameters used to translate their positions on the ultrasound image to displacements of the nodes of the 3D model. Furthermore, the real-time constraint was too demanding in order to implement more complex algorithms. We thus have investigated several speckle tracking methods to find out if one of these methods could be an alternative to our contour extraction-based method. Instead of extracting the contour of each image and find out on these contours corresponding tissue points from one image to the subsequent one, these methods aim to track small regions of the ultrasound images. This task is made difficult due to the speckle noise, but this speckle noise can also be used as a marker of a specific region from one image to the subsequent one if the displacement is not to large. At a high frame rate as the one we use, this can be true. The methods we tested are i) deformation registration, ii) optical flow and iii) local invariant features such as the Scale Invariant Feature Transform (SIFT) flow. On short time scales, these methods can provide good results, but they all exhibit a cumulative error that prevent their ability to track tissue points in the context of HBB game. In this context, they do not provide a robust and fast enough method to drive the 3D model.

Facial Expression Analysis

Test Leader: CERTH

Summary

This report presents the technical assessment results of the Facial Expression Analysis module. This module aims at recognizing the basic facial muscle movements of the user based on the analysis of 2D and 3D image streams recorded by a Kinect sensor. The module first detects the position of 81 facial landmarks (positioned at the eyes, eyebrows, nose, mouth, and face boundaries) and, then, extracts a set of facial measurements (e.g. distances like eye opening and eyebrow displacement or surface deformations like nose wrinkling), which are subsequently used to detect/recognize basic facial muscle movements known as Facial Action Units (e.g. eyebrows raised, lip corners pulled up, jaw drop, etc).

The features tested include the localization error of facial landmarks and the true positive and false positive recognition rates of facial action units (AUs). More specifically, to measure the performance of the facial feature tracking algorithm, we compare the estimated facial feature positions against their real (ground truth) positions in a set of 250 pairs (color+depth) of images depicting subjects with different facial expressions (from the database described below). The ground truth positions of facial features were manually annotated. The mean localization error is the 3D distance between the actual and the estimated feature positions in mm.

To measure the performance of the facial action unit detection algorithm, we use the true positive and false positive ratios. True positive is the correct classification rate of facial action units (facial action unit A is correctly classified as A); while, false positive is the wrong classification rate (for facial action unit A, the times that another action unit is erroneously classified as A).

The evaluation database consists of 64 pairs of color and depth image sequences of 6 subjects. The resolution of recorded images is 640x480. The two data streams (depth and color) are synchronized automatically and are also registered (1 to 1 pixel correspondence is established). In each sequence, the human subject displays a single action unit 2-3 times. Facial action periods last approximately 5-10 sec and are proceeded and followed by short neutral state periods. Facial action and neutral face periods were manually identified in each of these sequences and an appropriate tag was assigned to each frame (actually a set of AU labels).

Results


Assessment Category

Assessment Indices

ID Description


Value

Facial feature tracking accuracy

LocalError Localization error for the detection of facial landmarks in mm (numerical value [0-Inf])

<6 2.6

Facial Action Unit (AU) recognition accuracy

TruePositiveAU Classification accuracy of detected facial action units (numerical value [0,1])

>0.75 0.85

FalsePositiveAU False positive rate (numerical value [0,1])

<0.20 0.12

Comments:

- To measure the performance of the face tracking algorithm, we compare the estimated facial feature positions against their real (ground-truth) positions (LocalError).

- To measure the performance of the facial action unit detection algorithm, we compare for each frame the detected facial action units against the ground truth labels. We then calculate the true positive and false positive recognition ratios (TruePositiveAU, FalsePositiveAU).


The performance of the facial feature tracking algorithm is quite good. The mean localization error is 2.6 mm. The highest localization errors are observed for landmarks lying on lip boundaries when the mouth is stretched and the lower jaw is pulled downwards (AU27). In this case, the Point Distribution Model (PDM) of the face may fail to stretch the lower lip line and the jaw line enough so as to coincide with their actual positions.

The performance of the facial expression recognition algorithm is satisfactory as it can detect the correct facial action units at a rate of 85% (true positive). The highest correct recognition rates are observed for AU1 (inner eyebrow raised), AU2 (outer eyebrow raised) and AU26 (jaw drop).

The false positive rate is 12.5%. The higher false positive rates are observed for AU15 and AU25 (lips part). AU15 is mistaken with AU0 (neutral face) when the mouth corners are pressed downwards only slightly. Similarly, in the case of AU25 sometimes when the lips are just slightly apart, the open/closed mouth classifier fails and erroneously classifies the mouth as closed (AU0).

The above results highlight the satisfactory recognition of facial action units and the potential of the proposed algorithms.

Multimodal Data Fusion & Semantic Analysis

Test Leader: CERTH

Summary

This report presents the technical assessment results of the Final Version of the Multimodal Data Fusion and Semantic Analysis (MDFSA) module. This module analyzes the output of the WP3 analysis modules, i.e., the medium level features, and provides the high level features. In this report the assessment of the module is performed for the Tsamiko and Salsa sub-use cases. The module is evaluated, first, in terms of the ability to classify correctly and, second, in terms of the redundancy of the information taken as input in order to perform semantic analysis. Thus, the metrics used to assess the performance of the semantic analysis WP4 module are grouped in two categories. Four metrics that are commonly used to evaluate the classification performance of a classifier belong in the first category, while the mutual information based criteria that are used to assess the redundancy of the information

being processed fall in the second category.

The module was evaluated for the Tsamiko and Salsa sub-use cases. Specifically, the indices that assess the efficiency of MDSFA are based on the classification results, i.e., the ability of the module to correctly decide which Tsamiko style or Salsa choreography a dance performance belong. The results for the Tsamiko sub-use case presented herein show that the classification performance is quite high and also shows that there is no redundancy of the information being processed.

Variances

1. “Average precision” index has been replaced with “accuracy” (ACC), since the former was not very indicative of the classification performance and without providing an explanation of the results.

2. MF3 is a new index that has replaced the previous one, which was based on te mutual information between the high level and all the medium level features. In this case, MF3 is an index that is based on the mutual information between two set of features each one belonging on one of the two modalities. MF3, in essence, provides the degree of information conveyed from the knowledge about the features of one modality to the values of the feature of the other modality. Of course, the information conveyed is not expected to be strong (i.e., one cannot deduce the values of the features having known the values with a high degree of certainty), thus, the lower bound for MF3 is only 0.33.

Results


Assessment Category

Assessment Indices

ID Description


Value

Affective State Detection Efficiency

PREC Precision=#true-positive/(#true-positive+#false-positive)


>0.75 0.99

REC Recall=#true-positive/(#true-positive+#false-negative)


>0.75 0.94

F F-measure is the harmonic mean of precision and recall. (numerical value [0,1])

>0.75 0.965

ACC Accuracy= (#true_positive+#true_negative)/

(#all_data)

>0.75 0.97

Multimodal Data Fusion

MF1 Mutual information (normalized variation) between medium level features and high level features: index that assesses the redundancy of each medium level feature (numerical value [0,1])

≥0.75 0.99

MF2 Mutual information (normalized variation) between medium level features and high level features of the two modalities. Index that assess the redundancy of the group of medium level features of the

≥0.75 0.99

modality (numerical value [0,1])

MF3 Mutual information (normalized variation) between two groups of medium level features belonging in the audio modality and the visual modality. In essence, MF3 assess the redundancy of a modality given the other one (numerical value [0,1])

≥0.33 0.64

Comments:

The above results are based on the application of the MDFSA module to the Tsamiko sub-use case.


The classification efficiency of the model being assessed, as the classification metrics indicate, is quite high. Indeed, all metrics have values above 95%, much higher than the lower threshold. This shows that the MDFSA module exhibits a high efficiency in terms of the semantic analysis task that it has been designed to undertake.

Moreover, the values of the MF1 and MF2 indices, which are close to 1 (the upper limit), show that there is no redundant medium level features that could be excluded from the model. In contrast, excluding a feature or more (either as single features or group of features) belonging in the same modality, one could not maintain the classification accuracy of the module and, thus, its semantic analysis efficiency. Moreover, the relatively high value of MF3 shows that features between two modalities are not independent which validates the multimodal treatment of the problem.

Lastly, the overall metric assessment of the module, PI, is quite high: PI=0.86. This demonstrates the high efficiency of the classification and semantic analysis of the module being assessed, under the present circumstances, i.e., for the Tsamiko and Salsa sub-use case benchmark experiments.

The major strengths of this module are: 1) Handle any number of detected events (e.g., dance steps and beats), 2) highly efficient at recognizing dance styles and particular choreographies, 3) efficient assessment of the synchronization between the dancer steps and beats, 4) efficient discrimination between experts and amateurs (as a result of the above), 5) the multimodal problem formulation is straightforward, due to the nature of the model, and 6) being generative models, they are sufficient to learn from a limited amount of training data, in contrast with discriminative models (e.g., SVMs). On the contrary, the major weaknesses of this module are: 1) Depend heavily on the accuracy of the medium lever features given as input, 2) depend on the timing precision of the detected events that given as input with their attributes to the MEBN mode, and 3) cannot be applied in real-time for very large scale problems.

Deliverable No: D7.3 Final Evaluation Report - CORDIS...Deliverable No: D7.3 Final Evaluation Report i-Treasures ICT-600676 Filename: D7.3_deliverable_v8.docx Page 6 of 91 7.10.4 Evaluation

Documents