Cognitive Services: Building Smart Apps with Speech, NLP & Vision

Nick LandrySenior Technical Evangelist – Microsoft

[email protected]

Blog: AgeofMobility.com

@ActiveNick | github.com/ActiveNick

Microsoft Cognitive Services:

Building Smart Applications

with Speech, NLP & Vision

Microsoft Cognitive ServicesGive your apps a human side

Microsoft Cognitive ServicesGive your apps a human side

Cognitive Services API Collection

Bringing it all together The Seeing AI App

Computer Vision, Image, Speech Recognition, NLP,

and ML from Microsoft Cognitive Services

Watch Video HereRead Blog Here

EmotionSpeaker

Recognition

Speech

Custom

RecognitionComputer Vision

Face

Video

microsoft.com/cognitive

Linguistic Analysis

Language

Understanding

Bing Spell Check

Entity Linking

Knowledge

Exploration

Academic

Knowledge

Bing

Image Search

Bing

Video Search

Bing

Web Search

WebLM

Text Analytics Recommendations

Bing

Autosuggest

Bing

News SearchTranslator

EmotionSpeaker

Recognition

Speech

Custom

RecognitionComputer Vision

Face

Video

microsoft.com/cognitive

Linguistic Analysis

Language

Understanding

Bing Spell Check

Entity Linking

Knowledge

Exploration

Academic

Knowledge

Bing

Image Search

Bing

Video Search

Bing

Web Search

WebLM

Text Analytics Recommendations

Bing

Autosuggest

Bing

News SearchTranslator

Computer Vision API

Distill actionable information from

images

Video API

Analyze, edit, and process videos within

your app

Face API

Detect, identify, analyze, organize, and

tag faces in photos

Emotion API

Personalize experiences with

emotion recognition

Vision

Updated Computer Vision API

Content of Image:

Categories v0: [{ “name”: “animal”, “score”: 0.9765625 }]

V1: [{ "name": "grass", "confidence": 0.9999992847442627 },

{ "name": "outdoor", "confidence": 0.9999072551727295 },

{ "name": "cow", "confidence": 0.99954754114151 },

{ "name": "field", "confidence": 0.9976195693016052 },

{ "name": "brown", "confidence": 0.988935649394989 },

{ "name": "animal", "confidence": 0.97904372215271 },

{ "name": "standing", "confidence": 0.9632768630981445 },

{ "name": "mammal", "confidence": 0.9366017580032349, "hint": "animal" },

{ "name": "wire", "confidence": 0.8946959376335144 },

{ "name": "green", "confidence": 0.8844101428985596 },

{ "name": "pasture", "confidence": 0.8332059383392334 },

{ "name": "bovine", "confidence": 0.5618471503257751, "hint": "animal" },

{ "name": "grassy", "confidence": 0.48627158999443054 },

{ "name": "lush", "confidence": 0.1874018907546997 },

{ "name": "staring", "confidence": 0.165890634059906 }]

Describe0.975 "a brown cow standing on top of a lush green field“

0.974 “a cow standing on top of a lush green field”

0.965 “a large brown cow standing on top of a lush green field”

Start coding

microsoft.com/cognitive

Speech

Bing Spell Check API

Detect and correct spelling mistakes within your app

Language Understanding Intelligent Service

Teach your apps to understand

commands from your users

Web Language Model API

Leverage the power of language models trained on web-scale

data

Linguistic Analysis API

Easily parse complex text with language

analysis

Text Analytics API

Detect sentiment, key phrases, topics, and language from

your text

Language

Reduce labeling effort with interactive featuring

Seamless integration to Speech API

Deploy using just a few examples with active learning

Supports 5 languages (English, Chinese, Italian, French, Spanish)

Language Understanding Models

{“entities”: [

{“entity”: “flight_delays”,“type”: “Topic”

}],“intents”: [

{“intent”: “FindNews”,“score”: 0.99853384

},{

“intent”: “None”,“score”: 0.07289317

},{

“intent”: “ReadNews”,“score”: 0.0167122427

},{

“intent”: “ShareNews”,“score”: 1.0919299E-06

}]

}

“News about flight delays”

Language Understanding Models

Your conversation logic

Logic

Web Service

Your Bot

LUIS

Start coding

microsoft.com/cognitive

https://www.microsoft.com/cognitive-services/en-us/text-analytics-api

Start coding

microsoft.com/cognitive

https://www.microsoft.com/en-us/translator/products.aspx

Start coding

microsoft.com/cognitive

Easy

F lex ib le

Qual i ty

Why Microsoft Cognitive Services?

Developer Resources

Preview Pricinghttps://www.microsoft.com/cognitive-services/en-us/pricing

Documentationhttps://www.microsoft.com/cognitive-services/en-us/computer-vision-api/documentation

Client SDKshttps://github.com/Microsoft/ProjectOxford-ClientSDK

https://github.com/felixrieseberg/project-oxford (nodejs)

https://github.com/southwood/project-oxford-python

Example Codehttps://github.com/jsturtevant/happy-image-tester-django

https://github.com/jsturtevant/happy-image-tester-nodejs

Join Our Communityhttps://stackoverflow.com/questions/tagged/microsoft-cognitive

https://social.msdn.microsoft.com/forums/azure/en-US/home?forum=mlapi

https://cognitive.uservoice.com/

https://blog.xamarin.com/performing-ocr-for-ios-android-and-windows-with-microsoft-cognitive-services/

https://github.com/pierceboggan/smarter-apps/tree/master/ExpenseIt/

https://blog.xamarin.com/adding-facial-recognition-to-your-mobile-apps/

https://github.com/pierceboggan/smarter-apps/tree/master/talk/employee-directory/

https://github.com/xamarin/mini-hacks/tree/master/microsoft-cognitive-services/solutions

https://github.com/DotNetToscana/See4Me

https://github.com/ActiveNick/BingTranslateDemo

https://github.com/ActiveNick/CognitiveServices-Demo-Face

https://github.com/ActiveNick/CognitiveServices-Demo-EntityLinking

https://github.com/ActiveNick/CognitiveServices-Demo-TextAnalytics

https://channel9.msdn.com/Events/Build/2016/B878

https://channel9.msdn.com/Events/Build/2016/B855

https://channel9.msdn.com/Events/Build/2016/T620

https://channel9.msdn.com/Events/Build/2016/B834

https://channel9.msdn.com/Events/Build/2016/B833

http://developer.microsoft.com/cortana

http://AgeofMobility.com/Cortana

Online Microsoft training delivered by experts to help technologists continually learn

Hundreds of courses for developers, IT Pros, students, entrepreneurs and enthusiasts

11 different languages

3M+ students registered

Build your own Learning Plan

All free!

http://mva.microsoft.com

• Universal Windows App Developmentwith Cortana and the Speech SDK

• Available for on-demand viewing now:http://aka.ms/CortanaMVA

• Channel 9 Show

• Visual Studio Toolboxwith Robert Green• New Voice Commands

• Integration with Cortana’s canvas

• Background Voice Commands

• Continuous dictation

• Poutine in Montreal!

https://channel9.msdn.com/Shows/Visual-Studio-Toolbox/App-Development-with-Cortana

More Cortana Dev on Windows 10

Thank You!Slides are in SlideShare. Demos are on GitHub.Contact me and let me know what you build, I will be happy to help promote your apps.

Blog: AgeofMobility.com

Twitter: @ActiveNick

Email: [email protected]

Apps: www.bigbaldapps.com

LinkedIn: linkedin.com/in/activenick

GitHub: github.com/ActiveNick

Slideshare: slideshare.net/ActiveNick