Homonymy Traceability Nomadism of evaluation methodologies !! and Synonymy Need : One evaluation per evaluator per situation Difficult communication between actors No standard for information sharing, cooperation and collaboration ! Local Contraints: Budget, data collection, etc. Different norms depending on the Communities of Practice Need to refine indicators + collected data Call-flow: entry, path, exit Caller history Call motive Reference values Analysis Framing Development Experiments Pilot Generalisation Exploitation Specification of project scope, specifications and project plan. Feasibility evaluation. The service is maintained by both the hosting providers and the initial developers (for corrections). Access is granted to the overall flow of traffic. . Service tested under real conditions, with a fragment of the real flow of traffic. The evaluation constitute a comprehensive validation of the service before its access is extended to all users. It includes the relevance of the technical architecture.so as to detect and fix the last beta-solution issues. Test the service so as to identify design issues and refine the development to the users practices. Alignment of observation with a priori projections. Tests organised on a panel of recruited users. Development of a prototype integrating the initial design, linguistic, usability and technical considerations. Test within the development team to detect major bugs and enrich the vocal recongition model, Analysis of end-users' needs and practices. Solutions benchmark. Collect the users experience feedback in live conditions. The service keeps constantky monitored. Dashboards containing high-level indicators are sent daily to the various interested teams (number Evaluation Contexts What is done? Evaluation context? Marketing experts Hosting providers Ergonomics experts Vocal experts U s e r expe r i e nce Platf o rm intera c tions Tech n ical p erf o rman c e SDS- U sers i nteract i ons Se r vice % 46,23 12 36 58 14 What is a Spoken Dialogue System? View points Métier Technical Developers Ergonomics Experts Marketing Experts Business Managers Hosting Providers Goals Evaluations Indicators Contracting Owners Hard/software platform delivering a service to the customer Automate routing service satisfying the customer relationship management Hard / software platform Interactive systems implying a human- amchine dialogue, a QoS & an individualised access to information A technology based service tailored to the various business customers' needs Hard / software platform Service automating the human operators low added-value tasks Enhance the customer service with automated vocal solutions Deliver a service to the customer with respect to the customer relationship policy Ensure the respect of the predefined QoS commitments and the end-user satisfaction. ROI (project financing decisions), QoS, dialogue performance, user satisfaction Evaluate the service quality and its adequacy to the organisation Track the end-user experience Expert evaluation campaign (field tests and real users) Measure of the service performance and the users' perception. Measure of the service performance, monitor live services and technical support Measure of the service performance and the users' perception Monitor the good functionning of the live services they host Responsible of the service hosting and maintenance. Enhance the organisation Compare the adequation of the service performance with the organisation needs Compare the adequation of the service performance with the users needs Answer the users' expectations. Tailor the development with a user-centric development approach Monitor the service perceived quality Control the service performance, monitor and correct anomalies. Indicators linked to the service design Develop the service Marianne Laurent, Philippe Bretier, Carole Manquillet Ad Hoc evaluations along the lifecycle of industrial spoken dialogues systems: heading to harmonisation? [email protected], [email protected], [email protected] Orange Labs, 2 avenue Pierre Marzin, 22307 Lannion, France