My PhD defense slides -- A Semantics-based User Interface Model for Content Annotation, Authoring and Exploration

A Semantics-based UI Model for Content Annotation, Authoring & Exploration

Faculty of Mathematics and Computer Science University of Leipzig

Ali Khalili

Supervisors:

PhD Defense

Prof. Dr. Ing. habil. Klaus-Peter Fähnrich, University of Leipzig Prof. Dr. Sören Auer, University of Bonn

January 26, 2015

Outline

A Semantics-based UI Model for Content Annotation, Authoring & Exploration 2Ali Khalili

IntroductionResearch QuestionsContributions

State-of-the-art AnalysisWYSIWYM User Interface ModelFrom WYSIWYG to WYSIWYMLightweight Text AnalyticsAuthoring of E-Learning ContentAuthoring of Semantic Medical Prescriptions

ConclusionPublications & Awards

PhD DefenseJanuary 26, 2015

Introduction


User Interface & Applications

A Semantics-based User Interface Model for Content Annotation, Authoring & Exploration

Motivation - User Scenario


Introduction

Extracting the most demanded Data Science skills for IT jobs in Europe

Alice (Journalist)

Job Posting Websites

TextCo

Text AnalysisAnalysis Results

Job Poster

UIs for Semantic Content Exploration

UIs for Semantic Content Authoring

Difficult Time-consuming Error-prone Requires knowledge

representation expertise

Challenges


Introduction

Semantic Content Authoring (SCA) is cumbersome!

There is a lack of approaches, technologies and tools to facilitate collaboration.

There is a lack of incentives and instant gratification for users to adopt SCA.

There is a lack of standardization of UI technologies in the domain of SCA.

Research Questions


Introduction

How can we enable user-friendly manual and semi-automatic creation of rich semantic content?

What are existing approaches for user-friendly semantic content authoring? How to classify existing approaches for semantic content authoring? What types of user interfaces are used by each approach? What are the features supported by the proposed user interfaces? What types of users are targeted in each approach? How is the user interface evaluated?

How can we bind user interface elements to semantic representation data models? How can we integrate semantic content authoring features into the current authoring tools on the Social Web?

How can we exploit semantically-enriched content for content analysis? Can we utilize semantic content benefits to incentivize users contributing to the process of manual semantic content authoring?

How can we apply crowdsourcing & collaborative content authoring techniques to the process of semantic content authoring?

How can we apply semantic content authoring to a domain-specific use case for achieving content interoperability?

Contributions


Introduction

WYSIWYM UI Model

from WYSIWYG to WYSIWYM

WYSIWYM for Lightweight text Analytics

WYSIWYM for Authoring of E-Learning Content

State of the Art Analysis

WYSIWYM for Authoring of Semantic Medical

Prescriptions

guides extending

existing UIs

instant user gratification

support of collaboration

customization & content interoperability

RDFaCE

SlideWiki

conTEXT

Pharmer

- Approaches for Semantic Content Authoring (SCA) - A set of UI types & properties for creating efficient SCA systems

- Different views for content authoring - Combining NLP services for automatic content annotation

- Bindings between semantic representation data models & UI elements for content visualization, exploration & authoring

- Intelligent & Interoperable medical prescriptions

- Text analytics based on Linked Data services - Incorporating user feedback for NLP

- WikiApp data model - Crowdlearning

Contributions


Introduction

WYSIWYM UI Model






Prescriptions

guides extending

existing UIs




RDFaCE

SlideWiki

conTEXT

Pharmer







[Khalili and Auer, 2013a]

Approaches for Semantic Content Authoring


deliverableNature

deliveryDate

leading partner

work package

label

seeAlso

Deliverable LOD

2 O

ntology

Article

Person

...

...

...

Dom

ain O

ntology

Location

Organization

Person

label

label

label

...

...

...

...

Top-Down

Bottom-Up

OntologiesContent

Ontology Population

State of the Art

RDF OWL

Semantic Markup/ Annotation

RDFa Microdata Microformats

Quality Attributes for SCA System Design


Quality Attribute Realization

Usability

Customizability

Generalizability

Collaboration

PortabilityAccessibility

Proactivity

Automation

Evolvability

Evolvability

Scalability

State of the Art

WYSIWYM UI Model


WYSIWYM UI Model






Prescriptions

guides extending

existing UIs




RDFaCE

SlideWiki

conTEXT

Pharmer







[Khalili and Auer, 2014]

What You See Is What You Get - WYSIWYG


WYSIWYM UI Model

What You See Is What You Get - WYSIWYG


WYSIWYM UI Model

What You See Is What You Mean - WYSIWYM


User Interface

Semantics

WYSIWYM UI Model

What You See Is What You Mean - WYSIWYM


User Interface

Semantics

WYSIWYM UI Model

WYSIWYM Model


WYSIWYM UI Model

WYSIWYM Model


WYSIWYM UI Model

Semantic Representation Data Models


WYSIWYM UI Model

Tree-based (e.g. Taxonomies)

Graph-based (e.g. RDF)

Hypergraph-based (e.g. Topic Maps)

Item

Item

type

Item

-sub

item

Item

pro

perty

val

ue

Rel

ated

Item

s

Inst

ance

Cla

ss

Rel

atio

nshi

ps b

etw

een

entit

ies

Literal property values

Topi

c

Topi

c ty

pe

Topi

c as

soci

atio

ns

Topi

c ro

le in

ass

ocia

tion Topic

Occurrences

Valu

e

Lang

uage

tag

Dat

atyp

e

Valu

e D

atat

ype

Visualization Techniques


WYSIWYM UI Model

Structure encoded in: UI categories UI techniques Vi

sual

izat

ion

text Highlighting

Framing and segmentation (borders, overlays, backgrounds)

Text formatting (color, font, size etc.) Marking (appended icons)

Associating Line connectors Arrow connectors

Detail view Callouts (infotips, tooltips, popups)

images Highlighting


Image color effects Marking (appended icons)



videos

Highlighting


Image color effects Marking (appended icons)

Bleeping Speech


Detail view Callouts

(infotips, tooltips, popups) Subtitle

Exploration & Authoring Techniques


WYSIWYM UI Model

Expl

orat

ion text

Zooming Faceting

On-demand highlighting Expanding & Drilling down

images Zooming Faceting

videos Faceting (excerpts)

Aut

horin

g

text, images, videos

Form editing Inline edit Drawing

Drag and drop Context menu

(Floating) Ribbon editing Voice commands

(Multi-Touch) Gestures


WYSIWYM UI Model Binding


WYSIWYM UI Model Binding* If value is available in the text/subtitle.

No binding Partial binding Full binding

Tree-based(e.g. Taxonomies)

Graph-based(e.g. RDF)

Hypergraph-based(e.g. Topic Maps)

Item

Item

type

Item

-sub

item

Item

pro

perty

val

ue

Rel

ated

Item

s

Inst

ance

Cla

ss

Rel

atio

nshi

ps b

etw

een

entit

ies


Topi

c

Topi

c ty

pe

Topi

c as

soci

atio

ns

Topi

c ro

le in

ass

ocia

tion Topic

Occurrences

Val

ue

Lang

uage

tag

Dat

atyp

e

Val

ueD

atat

ype

Structure encoded in: UI categories UI techniques

Visu

aliz

atio

n

text

Highlighting


C1 C11 C13 C1 C1

Text formatting (color, font, size etc.) C2 C11 C14 C2 C2

Marking (appended icons) C3 C15 C3 C3

AssociatingLine connectors * * *

Arrow connectors * * *


images

Highlighting


C1 C11 C13 C1 C1

Image color effects C4 C11 C16 C4 C4


Associating Line connectorsArrow connectors


videos

Highlighting


C1 C11 C13 C1 C1


Marking (appended icons) C3C15 C3 C3

Bleeping C5 C5 C5

Speech



Detail viewCallouts

(infotips, tooltips, popups)Subtitle

Expl

orat

ion text

ZoomingFaceting

On-demand highlighting C5 C12 C5 C5

Expanding & Drilling down

images ZoomingFaceting


Auth

orin

g


Form editingInline editDrawing C7 C7 C7 C7

Drag and drop C8 C8 C8 C8

Context menu

(Floating) Ribbon editingVoice commands C9 C9 C9 C9

(Multi-Touch) Gestures C10 C10 C10 C10


WYSIWYM UI Model Binding* If value is available in the text/subtitle.

No binding Partial binding Full binding

Tree-based(e.g. Taxonomies)

Graph-based(e.g. RDF)

Hypergraph-based(e.g. Topic Maps)

Item

Item

type

Item

-sub

item

Item

pro

perty

val

ue

Rel

ated

Item

s

Inst

ance

Cla

ss

Rel

atio

nshi

ps b

etw

een

entit

ies


Topi

c

Topi

c ty

pe

Topi

c as

soci

atio

ns

Topi

c ro

le in

ass

ocia

tion Topic

Occurrences

Val

ue

Lang

uage

tag

Dat

atyp

e

Val

ueD

atat

ype

Structure encoded in: UI categories UI techniques

Visu

aliz

atio

n

text

Highlighting


C1 C11 C13 C1 C1

Text formatting (color, font, size etc.) C2 C11 C14 C2 C2





images

Highlighting


C1 C11 C13 C1 C1



Associating Line connectorsArrow connectors


videos

Highlighting


C1 C11 C13 C1 C1


Marking (appended icons) C3C15 C3 C3

Bleeping C5 C5 C5

Speech



Detail viewCallouts

(infotips, tooltips, popups)Subtitle

Expl

orat

ion text

ZoomingFaceting

On-demand highlighting C5 C12 C5 C5

Expanding & Drilling down

images ZoomingFaceting


Auth

orin

g


Form editingInline editDrawing C7 C7 C7 C7

Drag and drop C8 C8 C8 C8

Context menu

(Floating) Ribbon editingVoice commands C9 C9 C9 C9

(Multi-Touch) Gestures C10 C10 C10 C10

Best practices to facilitate, enhance & customize the WYSIWYM model For example:

Automation Recommendation Customization (Personalization) Collaboration & Crowdsourcing Accessibility etc.

Helper Components

WYSIWYM Interface


WYSIWYM UI Model



WYSIWYM UI Model






Prescriptions

guides extending

existing UIs




RDFaCE

SlideWiki

conTEXT

Pharmer







[Khalili et al., 2012a] [Khalili and Auer, 2013b]

RDFaCE - RDFa Content Editor


As a plugin for TinyMCE (http://tinymce.moxiecode.com) rich text editor Generates RDFa/Microdata annotations in the background.

V1: framing of named entities in the text V9: a callout showing additional type information X2: faceted browsing T6: command bar T5: a context menu for revising annotations T1: form editing H1: automatic content annotation

http://rdface.aksw.org


WYSIWYM Instantiation


RDFaCE Views for SCA


from WYSIWYG to WYSIWYM RDFaCE

RDFaCE Views for SCA



Automatic Content Annotation



Combining the results of different NLP APIs

Evaluation


from WYSIWYG to WYSIWYM RDFaCE NLP APIs Combination

31 articles were collected (and manually annotated) in the three categories news articles, weblog posts and Wikipedia articles.

Results of the RDFaCE automatic annotation feature was evaluated using this gold standard.

Evaluation


from WYSIWYG to WYSIWYM RDFaCE NLP APIs Combination

31 articles were collected (and manually annotated) in the three categories news articles, weblog posts and Wikipedia articles.

Results of the RDFaCE automatic annotation feature was evaluated using this gold standard.

RDFaCE Variations



Semantic Blogging in WordPress

Data Journalism using rNews

SEO using Schema.org

Semantic Blogging in WordPress


from WYSIWYG to WYSIWYM RDFaCE Use Cases

https://wordpress.org/plugins/rdface

https://wordpress.org/plugins/rdface

Data Journalism using rNews



http://dev.iptc.org/rNews

rNews - approved standard developed by IPTC (a consortium of the world's major news agencies, news publishers and news industry vendors).

- using semantic markup to annotate news-specific metadata in HTML documents.

http://dev.iptc.org/rNews




Schema.org initiated by the popular search engines Bing, Google and Yahoo! a broad, Web-scale and shared vocabulary focusing on popular

concepts suitable for search results.




Schema.orginitiated by the popular search engines a broad, Web-scale and shared vocabulary focusing on popular

concepts suitable for search results.

WYSIWYM for Lightweight Text Analytics


WYSIWYM UI Model






Prescriptions

guides extending

existing UIs




RDFaCE

SlideWiki

conTEXT

Pharmer







[Khalili et al., 2014]

Analytical Information Imbalance


Lightweight Text Analytics

Analytical Information Imbalance



Empower people to find out what patterns can be discovered and what conclusions can be drawn from the information they share.




Unstructured* Structured* Semi.structured*

•  A"ensity)•  Trendminer)•  MashMaker)•  Thomson)Data)

Analyzer)

•  IBM$Content$Analy.cs$pla2orm$•  GATE$•  Apache$UIMA$

•  Zoho$Reports$•  SAP$NetWeaver$•  Jackbe$•  Rapidminer$

•  Alchmey(•  OpenCalais(

•  Excel&•  DataWrangler&•  Google&Docs&Spreadsheets&•  Google&Refine&

•  Facete&•  CubeViz&

•  TweetDeck(•  Topsy(•  Flumes(




Unstructured* Structured* Semi.structured*

•  A"ensity)•  Trendminer)•  MashMaker)•  Thomson)Data)

Analyzer)

•  IBM$Content$Analy.cs$pla2orm$•  GATE$•  Apache$UIMA$

•  Zoho$Reports$•  SAP$NetWeaver$•  Jackbe$•  Rapidminer$

•  Alchmey(•  OpenCalais(

•  Excel&•  DataWrangler&•  Google&Docs&Spreadsheets&•  Google&Refine&

•  Facete&•  CubeViz&

•  TweetDeck(•  Topsy(•  Flumes(

Lack of tools dealing with unstructured content, catering non-expert users and providing extensible analytics interfaces.



T2: inline editing UI V1: framing of named entities in the text V2: text margin formatting for visualizing hierarchy V7: line connectors to show the relation between entities V9: a callout showing additional type information X2: faceted browsing H3: recommendation for NLP feedback.

http://context.aksw.org

WYSIWYM




Lightweight Text Analytics WYSIWYM

CollectingRSS, Atom, RDF Feeds REST APIs SPARQL Endpoints Web Crawlers

Processing

Enriching Mixing

Annotation Refinement Exploring & Visualizing

feedback

RDFaCEExhibitD3.js

BOA

conTEXT

Architecture & Workflow

Data Visualization & Exploration


Lightweight Text Analytics WYSIWYM conTEXT Workflow

Annotation Refinement & Feedback


Lightweight Text Analytics WYSIWYM conTEXT Workflow

Lightweight text analytics as an incentive for users to revise semantic annotationsRDFaCE for refining annotationsFeedback to NLP services NLP calibration

DBpedia Spotlight

FOX http://spotlight.dbpedia.org/rest/feedback

http://139.18.2.164:4444/api/ner/feedback

Feedback APIs

http://spotlight.dbpedia.org/rest/feedback

http://spotlight.dbpedia.org/rest/feedback

Evaluation


Lightweight Text Analytics WYSIWYM conTEXT

Task-driven usability & usefulness study 25 Users 10 questions pertaining to knowledge discovery in corpora of unstructured data

e.g. What are the five most mentioned countries by Bill Gates tweets?

usability evaluation platform http://context.aksw.org/evaluation

http://rdface.aksw.org/usability/

Evaluation Results


Lightweight Text Analytics WYSIWYM conTEXT Usefulness

Measuring +me & Jaccard similarity for answers using/without conTEXT

second

Avg. 136% more time without conTEXT

Evaluation Results


Lightweight Text Analytics WYSIWYM conTEXT Usability

System Usability Scale (SUS) 82

Evaluation Results


Lightweight Text Analytics WYSIWYM conTEXT Usability

System Usability Scale (SUS) 82



WYSIWYM UI Model






Prescriptions

guides extending

existing UIs




RDFaCE

SlideWiki

conTEXT

Pharmer







[Khalili et al., 2012b] [Tarasowa et al., 2013] [Tarasowa et al., 2014]

SlideWiki - OpenCourseWare Authoring


E-Learning WYSIWYM

V2: text margin formatting for visualizing content tree V7: line connectors to show the relation between slides and decks X4: expanding & drilling down to explore content T4: drag & drop to change the order of slides and decks T6: floating ribbon editing to author slide content H5: collaboration and crowdsourcing helper components

http://slidewiki.org



WikiApp Data Model


E-Learning WYSIWYM SlideWiki

WikiApp Data Model



WikiApp Data Model



Usage



January 25, 2015

Usage



Semantic Web lecture series created collaboratively on SlideWiki.

Usability Evaluation



22 subjects 252 new slide revisions were created by users. Originally the whole course had 130 questions, and students

changed 13 of them, fixing the typos or adding additional options to multiple-choice questions. SUS 69

Evaluation

WYSIWYM for Authoring of Semantic Medical Prescriptions


WYSIWYM UI Model






Prescriptions

guides extending

existing UIs




RDFaCE

SlideWiki

conTEXT

Pharmer







[Khalili and Sedaghati, 2013a] [Khalili and Sedaghati, 2013b]

Pharmer


Semantic Medical Prescriptions

V1: highlighting of drugs through framing V9: additional information about a drug in a callout T1/T2: combined form and inline editing of electronic prescriptions H2: Real-time drug tagging

http://bitili.com/pharmer

Authoring of Semantic Medical Prescriptions


http://bitili.com/pharmer

Contributions in a Glance


Conclusions

WikiApp

Community of users Semantically-enriched

documents

Binding

Semantic representation models

UI elements & techniques

Helper components

Anlaytics

Inte

rope

rabi

lity

WYSIWYM

use author

provide incentivize

Publications


Ali Khalili and Sören Auer. WYSIWYM ‐‐ Integrated Visualiza5on, Explora5on and Authoring of Seman5cally enriched Un‐structured Content. Seman1c Web Journal, 2014. Ali Khalili and Sören Auer. User Interfaces for Seman5c Authoring of Textual Content: A Systema5c Literature

Review. Journal of Web Seman1cs: Science, Services and Agents on the World Wide Web, 22(1), 2013. Ali Khalili and Bita Sedagha1. A WYSIWYM Interface for Seman1c Enrichment of E‐prescrip1ons using Linked

Open Drug Data. Interna1onal Journal On Advances in Life Sciences, 5(3,4), 2013. Darya Tarasowa, Sören Auer, Ali Khalili, and Jörg Unbehauen. Crowdsourcing (seman5cally) Structured

Mul5lingual Educa5onal Content (cosmec). Open Praxis Journal, 6(2), 2014.

Ali Khalili, Sören Auer and Axel C.N. Ngomo. conTEXT ‐‐ Lightweight Text Analy5cs using Linked Data. 11th Extended Seman1c Web Conference (ESWC 2014). Ali Khalili and Sören Auer. WYSIWYM Authoring of Structured Content based on Schema.org. The 14th

Interna1onal Conference on Web Informa1on Systems Engineering (WISE 2013). Darya Tarasowa, Ali Khalili, Sören Auer, and Jörg Unbehauen. Crowdlearn: Crowd‐sourcing the Crea5on of

Highly‐Structured E‐learning Content. 5th Intl. Conference on Computer Supported Educa1on (CSEDU 2013). Ali Khalili and Bita Sedagha1. Seman5c Medical Prescrip5ons ‐‐ Towards Intelligent and Interoperable Medical

Prescrip5ons. In IEEE Seventh Interna1onal Conference on Seman1c Compu1ng (ICSC 2013). Ali Khalili, Sören Auer, Darya Tarasowa, and Ivan Ermilov. Slidewiki: Elicita5on and Sharing of Corporate

Knowledge using Presenta5ons. The 18th Interna1onal Conference on Knowledge Engineering and Knowledge Management (EKAW 2012). Ali Khalili, Sören Auer, and Daniel Hladky. The RDFa Content Editor ‐ from WYSIWYG to WYSIWYM. In

Computer So\ware and Applica1ons Conference (COMPSAC 2012).

Awards


RDFaCE

Pharmer

Best-paper award of the 36th IEEE Signature Conference on Computers, Software, and Applications (COMPSAC) 2012.

Creative Innovation Project Award 2014 for OpenCourseWare Excellence from OCW Consortium.

Nominated for best-paper award at 5th Internat ional Conference on Computer Supported Education (CSEDU 2013) and 18th International Conference on Knowledge Engineering and Knowledge Management (EKAW 2012).

1st Prize of the AI Mashup Challenge 2014.Best-application prize at WoLE2013 challenge

(Doing Good by Linking Entities), WWW2013 workshops.

Best-poster prize at Leipzig Research Festival for Life Sciences 2012.

WYSIWYM


hank yu for yur atention…

January 26, 2015

My PhD defense slides -- A Semantics-based User Interface Model for Content Annotation, Authoring and Exploration

Engineering

content annotation

content analysis

semantic content benefits

content interoperability

enriched content

learning content state

current authoring tools

ui technologies