Cognitive Work Assistants - Vision and Open Challenges

© 2014 IBM Corporation

Cognitive Work Assistants: Vision and Open Challenges

Hamid R. Motahari Nezhad

IBM Almaden Research Center,

San Jose, CA, USA

Cognitive Systems Institute


The Work Practices of Human Administrative Assistants

Human assistant activities

– Calendaring• Scheduling, information formatting

and preparation

– Task Management

– Email Management• Filtering emails,

• Email classification

Interruption management

– Mediating interruption

– Prioritizing interruptions

Taking care of routine tasks

– Tracking

– Following up

– Travel arrangement, and

preparation

– Reminding, and organizing

– Managing work of human• Pre-processing

• Filtering

• Prioritizing

• Compiling information and reports

2

An assistant “will remove much of the burden of administrative chores from its human user and

provide guidance, advice, and assistance in problem solving and decision making.” Gutierrze and

Hilfdalgo, 1988


Human Administrative Assistants: conceptual framework

3

T. Erickson, etc.: Assistance: The Work Practice of Human Administrative Assistants and their Implications for IT and Organizations, CSCW’08.

Blocking, Doing, Redirecting

Key to the performance of assistants


Cognitive Assistance for knowledge workers

Cognitive case management is about providing cognitive support to knowledge workers

in handling customer cases in domains such as social care, legal, government services,

citizen services, etc.

Handling and managing cases involves understanding policies, laws, rules, regulations,

processes, plans, as well as customers, surrounding world, news, social networks, etc.

A cognitive agent would assist employees and customers (from each perspective)

– Assisting employees/workers by providing decision support based on understanding

the case, context, surrounding world and applicable laws/rules/processes.

– Helps employees/workers to be more productive, and effective

– Assists citizens by empowering them to know their rights and responsibilities, and

helping them to expedite the progress of the case

4

Users

Assistant

CustomersEmployees/

agents

Plansworkflows

Rules

Policies

Regulations

Templates

Instructions/

Procedures

ApplicationsSchedules

Communications such as

email, chat, social media,

etc.

Organization

Cog. Agent

Unstructured Linked Information


Cognitive Assistance: Application Domains

Cognitive Assistance for different Occupations

– Finance

– Education

– Retail

– Healthcare (physician assistant)

– Government (case management, civilian services, intelligence, defense, etc.)

Health Education Assistance

– Assistance to patients

– People well-being

– Public health education

Retail

– Assistance to buyers

5


Cognitive Assistant: what is it?

A software agent that

– “augments human intelligence” (Engelbart’s definition1 in 1962)

– Complements human by offering capabilities that is beyond the ordinary power and

reach of human (intelligence amplification)

– Performs tasks and offer assistance to human in decision making and taking actions

A more technical definition

– Cognitive Assistant offers computational intelligence capabilities typically based on

Natural Language Processing (NLP), Machine Learning (ML) and reasoning, and

provides cognition powers that augment and scale human intelligence (Jim Spohrer)

Getting us closer to the vision painted for human-machine partnership in 1960:

– “The hope is that, in not too many years, human brains and computing machines will be

coupled together very tightly, and that the resulting partnership will think as no human

brain has ever thought and process data in a way not approached by the information

handling machines we know today”

“Man-Computer Symbiosis , J. C. R. Licklider IRE Transactions on Human Factors in

Electronics, volume HFE-1, pages 4-11, March 1960

6 1 Augmenting Human Intellect: A Conceptual Framework, by Douglas C. Engelbart, October 1962


History of Cognitive Assistants from the lens of AI

7

1945

Memex (Bush)

1962

NLS/Augment

(Engelbart)

1955/6

Logic Theorist

(Newwell, Simon, 1955)

Checker Player

(Samuel, 1956)

Touring Test,

1950

Thinking machines

1966

Eliza

(Weizenbaum)

1965-1987 DENDRAL

1974-1984 MYCIN

1987 Cognitive Tutors

(Anderson)

Apple’s Knowledge

Navigator System

Expert Systems

1965-1987 1992-1998

Virtual Telephone

Assistant

Portico, Wildfire,

Webley;

Speech Recognition

Voice Controlled

2002-08

DARPA PAL

Program

CALO

IRIS


Modern Cognitive Assistants: State of the art (2008-present)

Commercial

Personal Assistants– Siri, Google Now, Microsoft

Cortana, Amazon Echo,

– Braina, Samsung's S Voice,

LG's Voice Mate, SILVIA, HTC's

Hidi, Nuance’ Vlingo

– AIVC, Skyvi, IRIS, Everfriend,

Evi (Q&A), Alme (patient

assistant)

– Viv (Global Brain as a Service)

Cognitive systems and platforms– IBM Watson

– Wolfram Alpha (a computational

engine with NLP interface)

– Saffron 10

– Vicarious (Captcha)

Open Source/Research

OAQA

DeepDive

OpenCog

YodaQA

OpenSherlock

OpenIRIS

iCub EU projects

Cougaar

Inquire* (intelligent textbook)

8

* Curated knowledge base


COGNITIVE AGENTS’ ABILITIES

What Capabilities Cognitive Agents need to have?

9


Human Intelligence in terms of Cognitive Abilities

10

Ability to Achievable by

machines today?

draw abstractions from particulars. Partially, semantic graphs*

maintain hierarchies of abstraction. Partially, semantic graphs*

concatenate assertions and arrive at a new conclusion. Partially, relationships present

reason outside the current context. No

compare and contrast two representations for

consistency/inconsistency.

Limited

reason analogically. Not automated, require

domain adaptation

learn and use external symbols to represent numerical,

spatial, or conceptual information.

Better than human in

symbolic rep. & processing

learn and use symbols whose meanings are defined in

terms of other learned symbols.

Uses and processes, limited

learning

invent and learn terms for abstractions as well as for

concrete entities.

No language development

capability

invent and learn terms for relations as well as things Partially, using symbols, not

cognitiveGentner, D. (2003), In D. Getner & S. Goldin-Meadow (eds.), Language in Mind: Advances in the Study of Language and Thought. MIT Press. 195--235 (2003)


Human Intelligence vs Machine Intelligence:Analytical Cognition, vs. Synthetic Cognition

Analytical Skills

Cognitive skills that machines excel

at would take intellectual efforts

from human– Mathematical calculations, making

logical decisions in complex

situations, chess

Computational Intelligence– Manipulation of symbols through

algorithmic information processing

– The processing units (processing

device) does not know or care about

the “meaning” of symbol

– Cognition by “information

processing”, or cognition as

computation

Synthetic Skills

Cognitive skills that human performs

effortlessly but hard for machines with

current AI– Interpretation of subtle facial

expressions, engaging in creative

conversations, etc.

Conscious intelligence– Symbol manipulation also happens in the

lowest level of hierarchical structure of brain

function

– The higher levels of hierarchical structure of

brain function involve emergent concepts

where higher level concepts/ideas combine,

and form complex organisms (analogy with

‘cloud’, a whole, relation to air and water

molecules, component)

– It is at this level of cognition that

“understanding of meaning” arise

11 Ref.: Eric Lord, Science, Mind and Paranormal Experience, 2009


Cognitive Assistant Vision: Augmenting Human Intelligence

12

CognitiveCapability

• Create new insights and new value

Discovery

• Provide bias-free advice semi-autonomously, learns, and is proactive

Decision

• Build and reason about models of the world, of the user, and of the system itself

Understanding

• Leverage encyclopedic domain knowledge in context, and interacts in natural language

Question Answering

Natural Language Processing and Interaction Skills

Em

oti

on

al In

tell

ige

nc

e S

kil

ls

So

cia

l In

tera

cti

on

Sk

ills

Touring Test

Analytical Abilities needed for a Cognitive Agent for higher order tasks


Not only personal work assistant, but a society of cogs

13

Cognitive Agent to

Agent

Outage Model

Consequence Table

Financial Cog

Enterprise Process

Cog

Objective Identification

Sensitivity Analysis

PR Cog

Personal Avatar

Deep Thunder

Crew Scheduler

Feeds

Human to Human

Cognitive Agent to Human

Sara’s Cog

Mobile Analytics

and Response

Two main type of cognitive

assistants: personal work

assistants, and expert cogs,

collaborate to support

human activities.

Interactions types need to be

supported:

• cog-to-cog interactions,

• human-cog interactions, and

• cog-backed human-to-

human interactions

Cogs need degrees of emotional

intelligence, and social

interaction skills to support cog-

human, and cog-backed human-

to-human interactions

Sara

Debra


A major challenge in passing Touring Test and building Cogs:building domain knowledge bases

“For an artifact, a computational intelligence, to be able to behave with high levels of

performance on complex intellectual tasks, perhaps surpassing human level, it must have

extensive knowledge of the domain”

The challenge of AI in making progress toward building human-like artifacts:

– Knowledge representation, and (especially) knowledge acquisition

Approaches

– Build a large knowledge base by reading text

– Distilling from the WWW a huge knowledge base

Semantic Web and Linked Data methods over the last decade extensively has explored

building models, ontologies and rule-set that contributes to WWW knowledge representation

– Manual, and semi-automated, focused on curated ontologies

– Community participation in building ontologies have resulted in creation of large

knowledge bases: DBPedia, Yago, Wikidata, Freebase, MediaWiki, etc.

– Ontologies are expensive to build and scale, and are generic in nature

14

EDWARD A. FEIGENBAUM, Some Challenges and Grand Challenges for

Computational Intelligence, Journal of the ACM, Vol. 50, No. 1, January 2003, pp. 32–40


Lesson Learned from Jeopardy in Watson (1)

“The Watson program is already a breakthrough technology in AI. For many years it had

been largely assumed that for a computer to go beyond search and really be able to perform

complex human language tasks it needed to do one of two things: either it would

“understand” the texts using some kind of deep “knowledge representation,” or it would have

a complex statistical model based on millions of texts.”– James Hendler, Watson goes to college: How the world’s smartest PC will revolutionize AI, GigaOm, 3/2/2013

Breakthrough:

1. Developing a systematic approach for scalable knowledge building over large, less

reliable data sources

• Building and curating a robust, and comprehensive knowledge base and ruleset has

been a key challenge in expert systems

• Watson approach for building on massive, mixed curated and not-curated and less

reliable information sources with uncertainty has proved effective

15

Source:

Inquire Intelligent

Book


Lesson Learned from Jeopardy in Watson (2)

16

Comparison of two QA

systems with and without

confidence estimation. Both

have an accuracy of 40%.

With perfect confidence estimator

Without confidence estimator

2. Leveraging a large number of not always accurate techniques but delivering

higher overall accuracy through understanding and employing confidence levels


Opportunity and challenge (1): explosive amount of data

17

80%of the world’s data

today is unstructured

90% of the world’s data was created in the

last two years

1 Trillionconnected devices

generate 2.5 quintillion bytes

data / day

3M+Apps on leading

App stores


Cognitive Computing as a Service: Watson in IBM BlueMix

18

Visualization RenderingGraphical representations of data analysis for easier understanding

User ModelingPersonality profiling to help engage users on their own terms.

Language IdentificationIdentifies the language in which text is written

Machine TranslationTranslate text from one language to another.

Concept ExpansionMaps euphemisms to more commonly understood phrases

Message ResonanceCommunicate with people with a style and words that suits them

Question and AnswerDirect responses to users inquiries fueled by primary document sources

Relationship ExtractionIntelligently finds relationships between sentences components

Coming

• Concept Analytics

• Question Generation

• Speech Recognition

• Text to Speech

• Tradeoff Analytics

• Medical Information Extraction

• Semantic Expansion

• Policy Knowledge

• Ontology Creation

• Q&A in other languages

• Policy Evaluation

• Inference detection

• Social Resonance

• Answer Assembler

• Relationship identification

• Dialog

• Machine Translation (French)

• Smart Metadata

• Visual Recommendation

• Industry accelerators

Available today

Opportunity and challenge (2): cognitive methods and tools


Open Challenges (1)

19

Building the knowledge base and Training Cognitive Agents

– How does User Train the Cog?

– How does User Delegate to the Cog?

Adaptation and training of Cogs for a new domain

– How to quickly train a cog for a new domain? Current approaches is laborious

and tedious.

Performance Dimensions, and Evaluation Framework

– Metrics, testing and validating functionality of Cog

– Are controlled experiments possible?

– Do we need to test in Real environment with Real users

User adoption/trust, and privacy– Can I trust that the Cog did what I told/taught/think the Cog did?

– Is the Cog working for me?

– Issues of privacy, privacy-preserving interaction of cogs.

Team vs. Personal Cogs – Training based on best practices vs. personalized instruction

– Imagine Teams of Cogs working with teams of Human Analysts

Symbiosis Issues– What is best for the human to do? What is best for the cog?


Open Challenges (2)

Teaching the Cog what to do– Learning from demonstration, Learning from documentation

– Telling the Cog what to do using natural language

– Interactive learning where the Cog may ask questions of the trainer

– How does the Cog learn what to do, reliably?

– Active learning where the Cog improves over time

• Moving up the learning curve (how does Cog understand the goal/desired end

state?)

• Adapts as the environment (e.g., data sources and formats change)

– On what conditions should the Cog report back to the Human?

– Task composition (of subtasks) and reuse

– Adaptation of past learning to new situation

Proactive Action taking – Initiating actions based on learning and incoming requests

• E.g., deciding what information sources to search for the request , issuing

queries, evaluating responses

– Deciding on next steps based on results or whether it needs further guidance from

Human

Personal knowledge representation and reasoning– Capturing user behavior, interaction in form of personal knowledge

– Ability to build knowledge from various structured and unstructured information

– AI Principle: expert knows 70,000+/- 20,000 information pieces, and human tasks

involves 1010 rules (foundation of AI, 1988)


Open Challenges (3)

Context understanding, and context-aware interaction

– Modeling the world of the person serving, including all context around the

work/task, and being able to use the contextual and environmental awareness

to proactively and reactively act on behalf of the user

Learning to understand the task and plan to do it

– Understanding the meaning of tasks, and coming up with a response (e.g..

How many people replied to an invite over email, accepting the offer, without

asking the Cog to do so), or suggestions on how to achieve it (based on any

new information discovered by the Cog)

Cognitive Speech recognition, or other human-computer interfaces for communicating with

Cogs

– Improving the speech-to-text techniques, and personalized, semantic-enriched

speech understanding

– Non-speech based approaches for communicating with humans


Learning from Jeopardy Challenge

Back in 2006, DeepQA (Question Answering) involved addressing key challenges

Feb 27-28, 2008, a group of researchers and practitioners from industry, academia and

government met to discuss state of the Question Answering (QA) field

The result was the development of a document (published in 2009) that included

– Vision for QA systems, and DeepQA

– Development of challenge problems with measurable dimensions

– Approach to open collaboration

– Open collaboration model

Defining Performance

Dimensions

Challenge Problem Set

Comparison

22


Call for Enabling Open Collaboration Model on Cognitive Assistants

The Open Collaboration Model enables sharing

knowledge, expertise, datasets, progress and

devising interoperable solution components

– Challenge Problem Set Comparison

– Defining and Developing Performance

Dimensions

– Open platform for sharing data, testbed and

comparative analysis

Building on Watson Ecosystem for Partners and

Watson University Program for academic partners

for towards Cognitive Assistant Open Collaboration

Platform, or similar open platforms for collaboration

– Building on open source cog projects

23

Building open platforms similar to Watson Content

Marketplace, Watson Ecosystem, and Watson University

Programs


THANK YOU!Questions?

24


BACKUP

25


Example: Automatic Task Extraction and ManagementOver Unstructured Communications

26

Email, Chat, and Calendaring apps are

the most used channels for doing work

in the enterprise

The goal of this project was to monitor

Communication channels (email, chat)

To capture and organize work of an

Employee (tasks)

Lifecycle of a typical task:

Create, Active, Complete, Cancel


Cognitive Assistant for Task management

Processing text of conversations (email, chat, etc) to extract and manage task

lifecycle

27Anup K. Kalia, Hamid R. Motahari Nezhad, Claudio Bartolini, Munindar P. Singh: Monitoring Commitments in

People-Driven Service Engagements. IEEE SCC 2013: 160-167

Deep Parsing of text

Where did it acquire knowledge?

• Wikipedia

• Time, Inc.

• New York Time

• Encarta

• Oxford University

• Internet Movie Database

• IBM Dictionary

• ... J! Archive/YAGO/dbPedia…

• Total Raw Content

• Preprocessed Content

28

• 17 GB

• 2.0 GB

• 7.4 GB

• 0.3 GB

• 0.11 GB

• 0.1 GB

• 0.01 GB

XXX

• 70 GB

• 500 GB

Three

types of

knowledge

Domain

Data(articles, books,

documents)

Training and test

question sets

w/answer keys

NLP Resources(vocabularies,

taxonomies,

ontologies)


Use Case: Work Assistant Example for knowledge workers

Assume an executive admin is managing an event organization process for their

department

– Step 1: sending invite to an event to employees in their department, through

email and requests for RSVP

• Cog (1): Q&A ability for the admin: How many have confirmed, how many

pending, how many not answered

• Cog (2): Predictive analytics: how many will eventually RSVP?

• Cog (3): Diagnostic analytics: why some not accepted (customers in case

of marketing case)?

– Step 2: Ordering place, food, transportation, etc

• Cog (1): tracking of the process steps, which vendor have replied, which

ones pending, have questions, etc.

• Cog (2): keeping track of synchronization and consistency (dates, amounts,

numbers, etc.) among different steps

– Step 3: Pre-event steps (self-discipline, and organization)

• Reminding people who have RSVPed

• Compiling and sending logistic information (from different steps)

– Learning changes to the process

29

Cognitive Work Assistants - Vision and Open Challenges

Data & Analytics

cognitive agent

cognitive support

complements human

human user

managing work of human

performance of assistants

government services

social care