Voice User Interface for Mobile Applications

Building a Voice User Interface Android Speech Recognition and Text-To-Speech

Tuesday, January 29, 13

Building a Voice User Interface Android Speech Recognition and Text-To-Speech

Tuesday, January 29, 13

http://wolfpaulus.comTuesday, January 29, 13

Star Trek© 2012-2013 Wolf Paulus - http://wolfpaulus.com

Tuesday, January 29, 13

Tuesday, January 29, 13

Red Planet© 2012-2013 Wolf Paulus - http://wolfpaulus.com

Tuesday, January 29, 13

Tuesday, January 29, 13

2001 Space Odyssey © 2012-2013 Wolf Paulus - http://wolfpaulus.com

Tuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

If a computer could think, how could we tell?

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

In 1950, Alan Turing suggested:

“If the responses from the computer were indistinguishable from that of a human,the computer could be said to be thinking.”

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Loebner Prize Solid 18 Carat Gold Medal

Grand Prize of $100,000 and a Gold Medal for the first computer whose responses were indistinguishable from a human's.

Each year an annual prize of $2000 and a bronze medal is awarded to the most human-like computer.

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

How can we create a Chat bot ?

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Capture Speech Input

Convert Speech into Text

Create Text Response

Message or Command ?

Execute Command

Message or Action ?

Perform Action

Synthesize Voice

(Message)

SpeekMessage

Cmd

Msg

Action

access Web Serviceperform on Device

Msg

ai

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Capture Speech Input

Convert Speech into Text

Synthesize Voice (Message)

SpeekMessage

access Web Serviceperform on Device

Echo Bot

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Capture Speech Input

void startVoiceRecognitionActivity() {Intent intent = new Intent( RecognizerIntent.ACTION_RECOGNIZE_SPEECH );

intent.putExtra( RecognizerIntent.EXTRA_PROMPT, "Speak to Bot1");intent.putExtra( RecognizerIntent.EXTRA_MAX_RESULTS, 1);

intent.putExtra( RecognizerIntent.EXTRA_CALLING_PACKAGE, getClass().getPackage().getName() );

intent.putExtra( RecognizerIntent.EXTRA_LANGUAGE_MODEL, ! ! ! ! !!                          RecognizerIntent.LANGUAGE_MODEL_FREE_FORM );

startActivityForResult( intent, VOICE_RECOGNITION_REQUEST_CODE );

}

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

@Override

protected void onActivityResult( int requestCode, int resultCode, Intent data ) {

switch (requestCode) {

case VOICE_RECOGNITION_REQUEST_CODE:

if (resultCode == RESULT_OK) {

ArrayList<String> matches = data.getStringArrayListExtra( RecognizerIntent.EXTRA_RESULTS );

say( matches.get(0) );

} else {

mTV_STT.setText("");

}

break;

}

}

... speech has been converted into text ...

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Convert Speech into Text

Capture Speech Input

t

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

private TextToSpeech mTts;..mTts = new TextToSpeech( this, this ); !    // Context, TextToSpeech.OnInitListener ..

// Implement TextToSpeech.OnInitListener@Overridepublic void onInit( final int status ) {! if ( status == TextToSpeech.SUCCESS && mTts != null ) {

! ! startVoiceRecognitionActivity();

! ! mTts.setOnUtteranceCompletedListener(new TextToSpeech.OnUtteranceCompletedListener() {

! ! ! @Override! ! ! public void onUtteranceCompleted( final String s ) {! ! ! ! startVoiceRecognitionActivity();! ! ! }});

! } else {! ! mTV_TTS.setText("Could not initialize TextToSpeech.");! }}

Synthesize Voice (Message)

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

private void say(final String s) {

final HashMap<String, String> map = new HashMap<String, String>(1);

map.put(TextToSpeech.Engine.KEY_PARAM_UTTERANCE_ID, UTTERANCE_ID);

mTts.speak(s, TextToSpeech.QUEUE_FLUSH, map);

}

SpeekMessage

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Code and DemoCapture Speech Input

Convert Speech into Text

Synthesize Voice (Message)

SpeekMessage

access Web Serviceperform on Device

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Capture Speech Input

Convert Speech into Text

Execute Command

Synthesize Voice (Message)

SpeekMessage

access Web Serviceperform on Device

“stock quote for ...”

Stock Quote Bot

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

@Override protected void onActivityResult( int requestCode, int resultCode, Intent data ) {

switch (requestCode) {

case VOICE_RECOGNITION_REQUEST_CODE: if (resultCode == RESULT_OK) {

ArrayList<String> matches = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS); say( matches.get(0) );

} else {

mTV_STT.setText(""); }

break;

} }

... speech has been converted into text ...

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

@Override protected void onActivityResult( int requestCode, int resultCode, Intent data ) {

switch (requestCode) {

case VOICE_RECOGNITION_REQUEST_CODE: if (resultCode == RESULT_OK) {

ArrayList<String> matches = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);final String s = matches.get(0);

final int k = s.indexOf( KEY_WORD );

! if (0 <= k) {! final String ticker = s.substring( k + KEY_WORD.length() ).trim();

! ! ! new YQuote(mHandler).execute( ticker );! ! } else {

say( s );

} } else {

mTV_STT.setText(""); }

break;

}

}

... speech has been converted into text ...

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Code and DemoCapture Speech Input

Convert Speech into Text

Execute Command

Synthesize Voice (Message)

SpeekMessage

access Web Serviceperform on Device

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Create Text Response

What is cheese ?What is chocolate?

How old are you? Who is the President?Where is Atlantis? What’s up?

Did you have dinner already?

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Artifical Intelligence Markup Language (AIML)

<?xml version="1.0" encoding="ISO-8859-1"?> <aiml>

<category><pattern>WHAT IS AIML</pattern><template>

AIML is short for Artifical Intelligence Markup Language

</template></category>

</aiml>

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

<?xml version="1.0" encoding="ISO-8859-1"?> <aiml>

<category><pattern>TELL ME WHAT AIML IS</pattern><template>

<srai>WHAT IS AIML</srai></template>

</category></aiml>

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

<?xml version="1.0" encoding="ISO-8859-1"?> <aiml>

<category><pattern>WHAT IS AIML</pattern><template>

<random><li>First response</li><li>Second response</li><li>3rd response</li>

</random></template>

</category></aiml>

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

<?xml version="1.0" encoding="ISO-8859-1"?> <aiml>

<category><pattern>TELL ME WHAT * IS</pattern><template>

I don't know what <star/> is.</template>

</category></aiml>

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

AIML Spec.• http://www.alicebot.org/TR/2011/

AIML Primer• http://www.alicebot.org/documentation/aiml-primer.html

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Program D v4.6 http://aitools.org/

last updated:14-Mar-2006

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

CharlieBot 4.1.8 http://sourceforge.net/projects/charliebot/

Forked from Program D v4.1.5 works on Mac OS X or any Java 1.3 or better VM

last updated:14-Dec-2002

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

ChatterBean http://www.geocities.ws/phelio/chatterbean/

ChatterBean is an AIML interpreter (also known as "Alicebot") written in pure Java.

Fully AIML 1.0.1 compliant

last updated:11-May-2006

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

AIML Sets

• http://aitools.org/Free_AIML_sets

• http://code.google.com/p/aiml-en-us-foundation-alice/

• http://www.square-bear.co.uk/aiml/

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

http://myAIMLServer:PORT/talk?botid=xyz..

XML-RPC: http://myAIMLServer:PORT/talk-xml

HTTP-POSTbotid=”xzy..”input=”Hello”custid=”d22..”

RESPONSE:<result status="0" botid="xyz.."

custid="d2228e2eee12d255"> <input>Hello</input> <that>Hi there!</that></result>

HTML:

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Capture Speech Input

Convert Speech into Text

Create Text Response

Message or Command ?

Execute Command

Synthesize Voice (Message)

SpeekMessage

Cmd

Msg

access Web Serviceperform on Device

Msg

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

@Overrideprotected void onActivityResult(final int requestCode, final int resultCode, final Intent data) {! switch (requestCode) {! ! case VOICE_RECOGNITION_REQUEST_CODE:! ! if (resultCode == RESULT_OK) {! ! ! final ArrayList<String> matches = data.getStringArrayListExtra( RecognizerIntent.EXTRA_RESULTS);! ! ! final String s = matches.get(0);! ! ! new AIML_RPC(mHandler).execute(s);! ! ! mTV_STT.setText(s);! ! } else {! ! ! mTV_STT.setText("");! ! }! ! break;! }}

Speech has been converted into text ...

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

@Overridepublic void handleMessage(final Message msg) {! if (msg.getData() != null && !msg.getData().isEmpty()) {! ! String s = msg.getData().getString(Bot3.BUNDLE_KEY_NAME_FOR_MSG);! ! if (s != null && 0 < s.length()) {! ! ! int i = s.indexOf("CMD ");! ! ! if (0 <= i) {! ! ! ! s = s.substring(i + 4, s.endsWith(".") ? s.length() - 1 : s.length());! ! ! ! String cmd;! ! ! ! int k = s.indexOf(" ");! ! ! ! if (0 < k) {! ! ! ! ! cmd = s.substring(0, k);! ! ! ! ! s = s.substring(k + 1).replace(" ", "");! ! ! ! ! if (Bot3.KEY_WORD.equals(cmd)) {! ! ! ! ! ! new YQuote(mHandler).execute(s);! ! ! ! ! }! ! ! ! }! ! ! } else {! ! ! ! Bot3.this.say(s);! ! ! }! ! }! }}

Handler

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Code and DemoCapture Speech Input

Convert Speech into Text

Create Text Response

Message or Command ?

Execute Command

Synthesize Voice (Message)

SpeekMessage

Cmd

Msg

access Web Serviceperform on Device

Msg

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Summary

Tuesday, January 29, 13

Capture Speech Input

Convert Speech into Text

Synthesize Voice (Message)

SpeekMessage

access Web Serviceperform on Device

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Capture Speech Input

Convert Speech into Text

Execute Command

Synthesize Voice (Message)

SpeekMessage

access Web Serviceperform on Device

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Capture Speech Input

Convert Speech into Text

Create Text Response

Message or Command ?

Execute Command

Synthesize Voice (Message)

SpeekMessage

Cmd

Msg

access Web Serviceperform on Device

Msg

AIML Bot

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Cora, your imaginary friendTechcasita Productions

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

© 2012-2013 Wolf Paulus - http://wolfpaulus.comTuesday, January 29, 13

Tuesday, January 29, 13