ConnectJS 2015: Video Killed the Telephone Star

Putting Real-Time Voice, Video and Text into Rails

Video Killed the Telephone Star

CAN YOU SPEAK MAGIC?

2

Ben Klang


2

Ben Klang


2

Ben Klang


2

Ben Klang


3

Web RTC


WHAT WEBRTC IS

4


WHAT WEBRTC IS

•Javascript access to Camera and Mic

4


WHAT WEBRTC IS


•High Quality Video & Audio Codecs

4


WHAT WEBRTC IS



•Techniques for traversing NAT

4


WHAT WEBRTC IS




•Peer-to-Peer

4


WHAT WEBRTC IS




•Peer-to-Peer

•A Data Channel

4


WHAT WEBRTC IS




•Peer-to-Peer

•A Data Channel

•A Tool for Developers

4


WHAT WEBRTC IS NOT

5


WHAT WEBRTC IS NOT

•…a Polished, End-User-Ready Product

5


WHAT WEBRTC IS NOT


•…the same thing to every application

5


WHAT WEBRTC IS NOT



•…required to interop…but may

5


WHAT WEBRTC IS NOT




•…finished.

5


WHAT WEBRTC IS NOT




•…finished.

5

But don’t sweat it


6


6


7


7

X X


7

Temasys Plugin skylink.io/plugin

http://skylink.io/plugin/


8

How WebRTC Works


9

Communication Topology


10

Alice BobThe Trapezoid


11

Alice BobThe Triangle


12

The WebRTC TriangleAlice Bob

Signaling

Media


12

The WebRTC TriangleAlice Bob

Signaling

Media


13

WebRTC Infrastructure


14


14


14


14

Signaling


14

Signaling


14

Signaling

STUN & TURNNAT Traversal


14

Signaling

STUN & TURN

SFU or MCU

NAT Traversal

Larger Multiparty Video Conferences


14

Signaling

Recording Conferencing PSTN Gateway

STUN & TURN

SFU or MCU

NAT Traversal

Larger Multiparty Video Conferences

Media Server


15


WEBRTC SIGNALING

16


WEBRTC SIGNALING•Can be anything you like

16



16



16



16



16


WEBRTC SIGNALING•Can be anything you like•Select based on application requirements

16


WEBRTC SIGNALING•Can be anything you like•Select based on application requirements•Integrate with existing comms or greenfield?

16


WEBRTC SIGNALING•Can be anything you like•Select based on application requirements•Integrate with existing comms or greenfield?•Federate? Or Not?

16


WEBRTC SIGNALING•Can be anything you like•Select based on application requirements•Integrate with existing comms or greenfield?•Federate? Or Not?•Anonymous, Social,or Legal Identities?

16

CAN YOU SPEAK MAGIC?CAN YOU SPEAK MAGIC?CAN YOU SPEAK MAGIC?

17


17

Alice


17

Alice Bob


17

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Alice Bob


17

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Alice Bob


17

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0

s=- t=0 0 m=audio 61001 RTP/SAVPF 109

Alice Bob


17

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0


Alice Bob


17

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0


Alice Bob


17

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0


Alice Bob


ICE/STUN/TURN

18


19

Alice Bob

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0



19

Alice Bob

SRTP

SRTP

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0



19

Alice Bob

SRTP

SRTP

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0



19

Alice Bob

SRTP

SRTP

XHav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0



20

Designing Communications Apps


THE FIVE TENETS

21


THE FIVE TENETS

21

A Modern Voice Application should be…


THE FIVE TENETS

•Adaptive

21



THE FIVE TENETS

•Adaptive

•Fluid

21



THE FIVE TENETS

•Adaptive

•Fluid

•Contextual

21



THE FIVE TENETS

•Adaptive

•Fluid

•Contextual

•Trustworthy

21



THE FIVE TENETS

•Adaptive

•Fluid

•Contextual

•Trustworthy

•Reference-able21



22

Adaptive


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23


BEING ADAPTIVE

23

/


24

Fluid


BEING FLUID

25


•Start with Chat

BEING FLUID

25


•Start with Chat•Add audio

BEING FLUID

25


•Start with Chat•Add audio•Invite more participants

BEING FLUID

25


•Start with Chat•Add audio•Invite more participants•Upgrade to video

BEING FLUID

25


•Start with Chat•Add audio•Invite more participants•Upgrade to video•Downgrade back to Chat

BEING FLUID

25


•Start with Chat•Add audio•Invite more participants•Upgrade to video•Downgrade back to Chat•Switch devices and continue

BEING FLUID

25


26

Contextual


“Communicating isn’t going to be what you’re doing - it’s what you’ll be doing

while you’re doing something else”

- Geoff Hollingworth Ericsson Head of AT&T Foundry

27


28How many sales reps are still in homes?

How much have we sold so far this month?

How many callers are in the queue right now?

Add my manager to this call

When is the next available appointment?

BEING CONTEXTUAL


29

Trustworthy


TRUSTWORTHY

30


TRUSTWORTHY•Don't surprise the user!

30


TRUSTWORTHY•Don't surprise the user!•User data stays property of user

30


TRUSTWORTHY•Don't surprise the user!•User data stays property of user•Help users make smart choices where required (with sane defaults)

30


TRUSTWORTHY•Don't surprise the user!•User data stays property of user•Help users make smart choices where required (with sane defaults)•Asserted identity should beuser-centric and strong

30


31

Reference-able


REFERENCEABILITY

32


REFERENCEABILITY•Make Interactions Shareable

32


REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:

32


REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique

32


REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request

32


REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request•May contain content alternatives: recording + transcription + annotation

32


REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request•May contain content alternatives: recording + transcription + annotation•Is searchable & downloadable

32


REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request•May contain content alternatives: recording + transcription + annotation•Is searchable & downloadable•Can be shared (respecting privacy limits)

32


33

What Should We Build?


33

What Should We Build?3 Ideas


1. Contextual, Remote Job Interviews

34


35

http://interview.example

Interview Me

Katharine Klang

CAN YOU SPEAK MAGIC?CAN YOU SPEAK MAGIC?

JOB INTERVIEW APP

36


JOB INTERVIEW APP•Self Service

36


JOB INTERVIEW APP•Self Service•Candidate signs up, uploads résumé

36


JOB INTERVIEW APP•Self Service•Candidate signs up, uploads résumé•Easy timeslot scheduling

36


JOB INTERVIEW APP•Self Service•Candidate signs up, uploads résumé•Easy timeslot scheduling•No exchange of phone numbers or Skype names -works on any device

36



•Contextual Interviews

36



•Contextual Interviews•Text chat alongside video conversation

36



•Contextual Interviews•Text chat alongside video conversation•See resume next to candidate

36



•Contextual Interviews•Text chat alongside video conversation•See resume next to candidate•Track and save notes during interview

36



•Contextual Interviews•Text chat alongside video conversation•See resume next to candidate•Track and save notes during interview•Record and share interviewee responses

36


2. Incident Response App

37


38


INCIDENT RESPONSE

39


INCIDENT RESPONSE•Timely, Contextual Information

39


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop

39


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication

39


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication•Inherit from existing organization groups

39


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication•Inherit from existing organization groups•Allow ad-hoc participants (guests)

39


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication•Inherit from existing organization groups•Allow ad-hoc participants (guests)•Federate with external services

39



•Incident recording/logging

39



•Incident recording/logging•“Lessons learned” and process improvement

39



•Incident recording/logging•“Lessons learned” and process improvement•Links from/to issue tracking systems

39


3. Medical Records Patient Services

40


41


MEDICAL ADVICE/HISTORY

42


MEDICAL ADVICE/HISTORY

42


MEDICAL ADVICE/HISTORY•Secure Caller Authentication

42


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website

42


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics

42


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics•Cross-check against caller location

42



•Automate Medical Claims

42



•Automate Medical Claims•Call recording/transcription

42



•Automate Medical Claims•Call recording/transcription•Medical advice given to patient automatically added to patient file

42



•Automate Medical Claims•Call recording/transcription•Medical advice given to patient automatically added to patient file•Easy Auditing/Service Quality Assurance

42


43

WebRTC & Rails


44


45


46


RESOURCES & QUESTIONS•WebRTC •http://webrtc.github.io/samples/ •http://www.webrtc.org/

•Voice in Ruby •https://talkingstick.io/ •http://www.adhearsion.com/

47

@bklang [email protected]

http://webrtc.github.io/samples/

http://www.webrtc.org/

https://talkingstick.io

http://www.adhearsion.com/

http://twitter.com/bklang

mailto:[email protected]




47











47








ConnectJS 2015: Video Killed the Telephone Star

Software