Open APIs for Open Minds
The Stream Oriented Generic Enabler – SO GE
Developing rich multimedia applications with FI-WARE.
Speakers
2
Luis López
• Leader of the Kurento project
• As. Prof. at URJC
• Coordinator of the Stream Oriented GE at FI-WARE
Jose Antonio Santos
• Chief Architect of Kurento project
• Senior Developer at Naevatec
• Researcher at FI-WARE project
The Stream Oriented Generic Enabler – SO GE
3
• Send and receive media (Multidevice/multiprotocol)• Process media (Computer vision, augmented reality, media indexing, etc.)• Transform and adapt media (H.264, H.263, VP8, Ogg, and others)
Provides multimedia capabilities to the FI-WARE infrastructure
• No need of protocol/codec low level understanding
Exposes those capabilities through a simple to use API
• LGPL 2.1
Is distributed through a flexible FOSS licence
• Sorry for the instabilities…
Is still work in progress
How media APIs (usually) work
4
Client side- WWW browser- Native API
Server side- Media repositories- Media capabilities
Hey, I want to send/receive media in this way
Here you have the media you requested
API API
The SO-GE API is a WWW inspired API
5
Dynamic WWW page development
• Lots of capabilities (ex. Java EE)
Multimedia development with the SO-GE
• Don’t want to loose any capability
Create HTML- DDBB access- Transactions- Security tools- Etc.
HTTP request: I want this resource
HTTP response:The resource
Create Media- Media API- DDBB access- Transactions- Security tools- Etc.
HTTP request: I want this media
HTTP response:The media is
here
The SO-GE Media API: media elements and pipelines
6
Media Element
• Provides a specific media functionality
› Building block
› Send/receive media
› Process media
› Transform media
• The Media API provides a toolbox of media elements ready to be used.
• New media elements can be added
Media pipeline
• Chain of media elements implementing the desired media logic.
• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox
Media Element
Sink
SRC
The global vision of a SO-GE application
7
SO-GE Media Server Infrastructure
DecodeVideo
AugmentedReality
Encode Video
Computer Vision
Video Playingand Recording
Examples of media elements: HttpEndPoint
8
HttpEndPoint
Function
• Sends/receives media basing on HTTP transports (HTML5 video tag)
• Associates media to a temporary-one-time-use URL
Formats
• WebM
• H.264 support under development
Input parameters
• None
HttpEndPoint
Media Source
Media Sink
HTTP POST requests
HTTP answers
Example of media elements: RtpEndPoint
9
RtpEndPoint
Function
• Sends/receives media basing on RTP transports (Real Time Communications).
• Negotiates media parameterization through SDPs
Formats
• H.263, MPEG4, H.264, VP8
Input parameters
• Input SDP
RtpEndPoint
Media Source
Media Sink
RTP inputmedia
RTP outputmedia
Example of media elements: PlayerEndPoint
10
PlayerEndPoint
Function
• Plays media from file or URI.
Formats
• WebM
• MP4 (H.264)
• 3GPP
• AVI
• Etc.
Input parameters
• File URI
PlayerEndPoint
Media Source
Media fromfile or URI
Example of media elements: RecorderEndPoint
11
RecorderEndPoint
Function
• Records media into a file.
Formats
• WebM
• H.264 formats under development
Input parameters
• File path
RecorderEndPoint
Media Source
Media Sink
Media tofile
Example of media elements: JackVaderFilter
12
JackVaderFilter
Function
• Example of filter
• Recognizes faces
• Adds a Jack Sparrow or Darth Vader hat onto faces
Formats
• Raw
Input parameters
• None
JackVaderFilter
Sink
SRCMedia
StreamAugmented
Media stream
Example of media elements: ZBarFilter
13
ZBarFilter
Function
• Example of filter
• Recognizes Bar and QR codes
• Generates events
Formats
• Raw
Input parameters
• None
ZBarFilter
Sink
SRCMedia
StreamMediastream
Event
Developing media application with the SO-GE
Runtime environment
• Install Kurento Media Server (soon available at FI-LAB)
• Install JBoss Java EE container (soon available at FI-LAB)
Development environment
• Maven (repository: http://repository.kurento.com/archiva/repository/snapshots)
• Soon available at maven central
› kmf-content-api
› Provides handers
› kmf-media-api
› Provides media elements and pipelines
• Eclipse
• Javascript libraries (available at the same repository)
› kws-content-api
14
My first application with the SO-GE: play a video
15
HttpEndPoint
Media Source
Media Sink
PlayerEndPoint
Media Source
Media fromfile or URI
HTTP mediastreaming
Client side Server side
My first application with the SO-GE: play a video
Source code …
16
My second application: including a simple filter
17
HttpEndPoint
Media Source
Media Sink
PlayerEndPoint
Media Source
HTTP mediastreaming
JackVaderFilter
Sink
SRC
Client side Server side
My second application: including a simple filter
Source code …
18
Making it more fun: introducing real time media
19
Media Source
Media Sink
Real timemedia
HTTP mediastreaming
JackVaderFilter
Sink
SRC
RtpEndPoint
Media Source
Media Sink
Client side Client sideServer Side
Making it more fun: introducing real time media
Source code …
20
Programming with media events: the ZBarFilter
21
Real timemedia
ZBarFilter
Sink
SRC
RtpEndPoint
Media Source
Media Sink
HttpEndPoint
Media Source
Media Sink
Media events to browser
Client side Client sideServer Side
Programming with media events: the ZBarFilter
Source code …
22
Digging into the media API: implementing your own filters: requirements Requirements
• GStreamer
› Based on GObject and GLib
› C programing skills needed
• OpenCV or your preferred image processing software
• C++ basic knowledge
• Thrift basic knowledge
23
Digging into the media API: implementing your own filters: GStreamer media element implementation Download media server source code
Look for sub-module gst-kurento-plugins
Enter into src/filters folder
Create your filter as couple of filtername.c filtername.h files
• Your filter must inherit from GstVideoFilter
• You have to implement virtual method transform_frame_ip
› This method gives you each frame to be processed
• Be aware of init and finalize functions if you use attributes that need to be freed
24
Digging into the media API: implementing your own filters: Integration in media server Modify kms-interface/mediaServer.thrift file adding a new value to enum FilterType
Create a C++ file to manage your GStreamer filter (by convention we use the same name in camel case)
• This class inherits from kurento::Filter class
• Its constructor creates a “filterelement” gstreamer element and add your filter name as a parameter (it will be created automatically)
• Destructor should free this element
Modify kurento::MediaPipeline::createFilter method in order to add your filter type in the switch statement
25
Facing the FI-WARE challenge using the SO-GE
Getting started• Use the FI-WARE Catalogue (ready since 15 minutes ago hopefully!)
› http://catalogue.fi-ware.eu/enablers/stream-oriented-kurento/• Ask the SO-GE team around the FI-WARE stand
Augmenting videos in novel ways• Sensor information on live stream videos
› Pollution/noise/traffic Red-filtering level• Combine audio-visual & sensor information through augmented reality
› Thermometers, charts, gauges on top of a media flow.
Make a camera to become an advanced sensor using computer vision• Traffic status
› Car counter• Weather
› Sky/ground color
Conceive novel WWW and Smartphone interaction GUIs• Up to your imagination
26
http://fi-ppp.eu
http://fi-ware.eu
Follow @Fiware on Twitter !
Thanks !
27
Technical details: Javascript KwsContent API
Creating a media connection
• conn = new KwsContent(uri, options);
• Options
› remoteVideoTag: id of the video tag rendering the remote stream
› Others for WebRtc (currently not available)
• Semantics
› Connects to URI and requests media
› Upon media reception, renders it on the specified video tag
Adding event handlers
• conn.onstart( function …
• conn.onremotestream (function …
• conn.onterminate( function …
• conn.onmediaevent( function …
28
The SO-GE Content API: handlers
Handler
• Is implemented by the application developer (equivalent to a Servlet)
• Defines the sequence of instructions executed when a request for media is received
Four types
• PlayerHandler
› Handles requests for playing media with HTTP transport
› Compatible with the HTML5 video tag
• RecorderHandler
› Handlers request for recording media with HTML transport
› Compatible with HTTP file upload mechanism
• RtpMediaHandler
› Handlers request for establishing RTP media sessions
• WebRtcMediaHandler (under development)
29
Technical details: the SO-GE Architecture
30
Pla
yerH
anld
er i
nte
rnal
det
ails
: se
qu
ence
dia
gra
m
31
