NL-Graphs: A Hybrid Approach toward Interactively Querying Semantic Data

• 1. NL-Graphs: A Hybrid Approach toward Interactively Querying Semantic Data Khadija Elbedweihy Suvodeep Mazumdar Stuart N. Wrigley Fabio Ciravegna OAK Group University of Sheffield Sheffield, UK k.Elbedweihy | s.mazumdar | s.wrigley | f.ciravegna | @ shef.ac.uk ovus00

2. Users and User Experience Semantic Search System Assist users satisfy their information needs User Satisfaction Query approach Performance of Search system Presentation of results . Usability and User Experience largely ignored Need to focus on users 3. Query Approaches Types Interaction mechanisms Formalisms (Natural language Forms Formal Query) Often Task or application Specific Keyword-based approach to quickly identify possible relevant instances Formal query for more complex tasks (negation, group-by etc.) Most SW applications focus on one approach 4. Esther Kaufmann and Abraham Bernstein. Evaluating the usability of natural language query languages and interfaces to semantic web knowledge bases. Journal of Web Semantics, 8(4):377393, 2010. Spectrum of Query Approaches 4 5. Issues + Motivations Comparison of Query Approaches (Expert and Casual) [2] NL Approach + Prior experience with Google-like approaches + Easy and quick to write questions/keywords - Less support for Complex queries View Based Approach + Good support in complex queries + Visualising the search space - Time and effort required formulating queries Learnability study with Graph based approach (Experts) [3] + Improvement in user satisfaction and performance over time - Still needed time and effort to formulate queries - Extensive use of keyword-search for concepts [2] K. Elbedweihy, S. N. Wrigley, and F. Ciravegna. Evaluating Semantic Search Query Approaches with Expert and Casual Users. In Proceedings of the Evaluations and Experiments Track, ISWC, 2012. [3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The visual appeal of linked data. Semantic web : interoperability, usability, applicability, 2013. In Press. 6. Affective Graphs 7. Affective Graphs Demo 8. HybridGraphs NL [4] + Graph [3] = Hybrid Query Approach Builds on Advantages of either approach Quick access to relevant data Easy and visual means to formulate complex queries Visually inspect search space Alleviates the disadvantages Lesser time taken to build complex queries [3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The visual appeal of linked data. Semantic web : interoperability, usability, applicability, 2013. In Press. [4] K. Elbedweihy, S.N. Wrigley, F. Ciravegna, and Z. Zhang (2013). Using BabelNet in bridging the gap between natural language queries and linked data concepts. NLP & DBpedia Workshop, 12th International Semantic Web Conference (ISWC2013), 21-25 October, Sydney, Australia. 9. Hybrid Approach 10. Evaluation Previous evaluations on NL component and Affective Graphs [3,4] (performance, usability, effectiveness etc.) Focus on hybrid approach as a new query mechanism and the support provided Methodology Expert (SW experts in Computer Science + Software development firm) and Casual users (staff and students at the University of Sheffield) 24 subjects (12 + 12) Aged between 18 and 53, mean age 31 years Short demonstration of a scenario based example Familiarisation with the system [3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The visual appeal of linked data. Semantic web : interoperability, usability, applicability, 2013. In Press. [4] K. Elbedweihy, S.N. Wrigley, F. Ciravegna, and Z. Zhang (2013). Using BabelNet in bridging the gap between natural language queries and linked data concepts. NLP & DBpedia Workshop, 12th International Semantic Web Conference (ISWC2013), 21-25 October, Sydney, Australia. 11. Evaluation Tasks 5 questions derived from QALD dataset [5] Different levels of difficulty 1. When was Capcom founded? 2. What did Bruce Carver die from? 3. Who was the wife of U.S. president Lincoln? 4. Give me all cities in Alaska with more than 10000 inhabitants. 5. Show me all songs from Bruce Springsteen released between 1980 and 1990. [5] V. Lopez, C. Unger, P. Cimiano, and E. Motta. Evaluating question answering over linked data. Web Semantics: Science, Services and Agents on the World Wide Web, 2013. 12. Evaluation Analysis Objective and Subjective data collected Input Time, number of attempts, Success rate Post evaluation questionnaires System Usability Scale (SUS), Extended questionnaire The query construction process was X. (5-point Likert scale, ranging from Laborious to Effortless) What did you like about the hybrid approach as a mechanism for expressing your query? and why? What things you didn't like about the hybrid approach as a mechanism for expressing your query? and why? Aesthetics questionnaire 13. Results Overall, experts (73.75) liked NL Graphs more than casuals (61.25) Graph based approach more complicated Experts more familiar with Semantic Web and graph data Success rate 100% (all users were able to successfully complete the tasks) mostly on one attempt Query Construction process scored effortless Most users commented positively on the usability and support provided by the hybrid approach Four users (1e + 3c) Complicated and non-intuitive Why are there more steps required to build queries than with Google? It seemed an extra step to get to your answer rather than just typing in a search and it appearing in results. 14. Results Contd. More assistance needed/desired Need for more Training Experts took longer (94.48s) to formulate queries than casual (76.88s) users Follow logic and prior understanding of SW concepts Increased interest in the interface, trying out new features, exploring technical aspects (e.g. SPARQL query generated) 15. Results Contd. 16. Results Contd. Q4 Give me all cities in Alaska with more than 10000 inhabitants Use of terms like Alaskan DbPedia properties such as isPartOf Numerical constraints >10000 Q5 - Show me all songs from Bruce Springsteen released between 1980 and 1990 Setting dates as a constraint is a manual process 17. Summary Hybrid approach to querying Linked Data It works ! High SUS scores indicate usability and usefulness Future work Several re-designs identified/suggested in the evaluation A longer term study directly comparing Hybrid approach to other approaches in identical conditions Study with larger set of user groups Improving NL system [3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The visual appeal of linked data. Semantic web : interoperability, usability, applicability, 2013 Aesthetic design in Semantic Web Applications [3] 18. Thank You Questions ?