1 Sam Jha Director, Product Management Qualcomm Technologies, Inc. Creating better mobile user experiences with Qualcomm Technologies’ advanced technologies
1
Sam Jha Director, Product Management Qualcomm Technologies, Inc.
Creating better mobile user experiences with Qualcomm Technologies’ advanced technologies
2
Immersive experiencesThe experiences worth having, remembering, and reliving
Draw you in…Take you to
another place…Keep you present in the moment…
3
Immersion and intelligence are the new “User Experiences”
Experiences become more realistic, engaging, and satisfying
Connecting your devices at home and work, making them more rewarding
On-device applications that learn, object, face and scene
detection
Smooth, interactive, cognitive user interfaces
Augmented reality-based experiences
New virtual reality applications
Realistic gaming experiences
Theater-quality movies and live sports
4
Better Display & Camera
Better Audio & Voice
Qualcomm® Snapdragon™ processors – enhancing user experienceFocusing on broader dimensions of user experience
Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.
High resolution audioSampling rate and precision3D surround sound and clarity
Pixel quantity and qualityResolution and frame rate Color accuracy, contrast, brightness
“True to life” soundReach out and Touch with your voice
Contextual interactionsIntelligent and personalized
Better content creationSensor, ISP, Resolution, fps, Computational camera, DSC quality
Natural user interfacesResponsive, accurate, and seamless
Visual quality
Sound quality
Intuitive interactions
Immersion&
Intelligence
5
Snapdragon – enhancing visual qualityFocusing on broader dimensions
High resolution audioSampling rate and precision
“True to life” sound 3D surround sound and clarity
Contextual interactionsIntelligent and personalized
Natural user interfacesResponsive, accurate, and seamless
Visual quality
Sound quality
Intuitive interactions
Immersion&
Intelligence
Pixel quantity and qualityResolution and frame rate Color accuracy, contrast, brightness
Better content creationSensor, ISP, Resolution, fps, Computational camera, DSC quality
6
Visual quality is much more than resolution and frame rate Color accuracy, contrast, and brightness are also critical elements
Pixel quantity Pixel quality
Resolution
Increased definition and sharpness
Frame rate
Reduced blurring and latency
Color accuracy
More realistic colors through an expanded color gamut, depth, and temperature
Contrast and brightness
Increased detail through a larger dynamic range and lighting enhancements
7
Power management is very important on any displayColor accuracy, contrast, and brightness are also critical elements
Qualcomm EcoPix is a product of Qualcomm Technologies.
Sunlight viewabilityQualcomm®
EcoPix™ features
Assertive Display/ SVI
Better viewing under bright light
Variable RefreshImproved battery life
CABL/ FOSS
Power savings for 30 ~ 50% on LCD or AMOLED
Display Stream Compression
High refresh
CABL/FOSS OFF CABL/FOSS ON
Same Image Quality at lower Backlight or Intensity
Low refresh
Reduce power and cost with VESA DSC
DD
ICW
ith RA
M
LCD
Controller
No C
ompression
FHD panelw/o 4Gbpsw/ 2Gbps
MSM
w/o 16 MB; w/ 8 MB
8
Display related complaints
AMOLED display power is high
Difficult to view in bright conditions
Natural imagery looks unnatural on some large color gamut displays
Display Jank under high system load
Mechanical design challenges for QHD displays
VESA DSC compression
Hardware composition (8 layers)
Color Gamut mapping
Assertive DisplayFOSS
Qualcomm Technologies’ solution
9
Taking an end-to-end approach to improve visual qualityThe journey of light from the mobile camera to a correct pixel on the display is challenging
End-to-end solutionsColor management, artifact removal, optimized click-to-shoot time
Component tuning | Image processing | System optimization
Mobile size constraints create challenges:• Poor light gathering• Increased noise
Lens and sensorAddresses camera and display
challenges while enhancing images
SoCPhysical limitations of LCDs and LEDs:• Partial color gamut• Limited contrast and brightness
Display
SoC
10
Qualcomm Spectra™ camera ISP
Qualcomm Spectrs is a product of Qualcomm Technologies QTI research..
• Improved image quality, with more natural skin tones• Pre & Post processing on HVX for new UX• Improved power efficiency and higher throughput
Benefits
What’s new
• Improved image quality • Enhanced de-mosaicing for new ~1.0u image sensors• New hardware for radial noise reduction & color artifact
correction• Camera sensor pre-processing streaming to DSP
• Improved hybrid autofocus framework (PDAF/Contrast/Laser)• GPGPU Optical Zoom• Universal Bandwidth Compression (UBWC)• Serial C-PHY interface
WithoutQualcomm Spectra ISP
Using Qualcomm Spectra ISP
11
Radial noise reduction
QTI research.
• New Diffusion Bayer Noise Filter• Removes Radial Noise from Image
Radial Noise, makes woodsurfaces look unnatural
12
Color artifact correction
QTI research.
Competition SDRawInput
Pink spots
No pink spots
13
35%
28%
28%
28%
27%
Camera quality influences over half of future smartphone selectionsOwners want capabilities that ensure quality pictures
Qualcomm survey with n=1065 smartphone ownersQ16. How important will the camera quality be in influencing the selection of your next smartphone? 5pt – extremely, very, moderately, slightly, not at all. Total N=1065Q5. Which of the following camera features would influence your next smartphone purchase? . N=1065
56%believe camera quality is very/
extremely important in influencing
their next smartphone selection
Top 5 Camera CapabilitiesInfluencing SP Purchase
1
2
3
4
5
Auto-Focus
Fast Auto-Focus
Low-Light Shooting Mode
Image Stabilization
Digital Zoom
14
Smartphones are the most used cameras – even for DSLR & P&S owners
Qualcomm survey with n=1065 smartphone owners
77%
76%
75%
72%
53%
46%
Easy to share
Convenience
Easy to use
Portability
Software/apps
On device editing
Top Smartphone Camera Advantages
72%Smartphone owners take
smartphone pictures at least weekly
1
Device UsageAve. % of pictures taken with device
61%
67%27%
29%
94%
DSLR Owner
P&S Owner
SP Only Owner2
3
4
5
6
15
Qualcomm Technologies, Inc. has a solution for the top issues users facePoor clarity, low light, zoom and focus are the most mentioned pain points
Q6. When taking pictures on your smartphone camera, which if any problems do you have? Total N=1567Qualcomm OptiZoom and Qualcomm ChromaFlash are products of Qualcomm Technologies, Inc.
Blurry picturesBad in low light Limited zoom Flash ruinsPoor/slow focus Overall slow toopen/zoom/take
36% 35% 28% 25% 22% 19%
Top SP Camera Pain PointsWhile taking pictures
SDLLVSDLLP
Qualcomm® Chromaflash™ camera feature
Qualcomm® OptiZoom™ camera
feature
Dual Camera AFPDAF
ZSLFast Start
Touch to TrackDual Camera AF
PDAF
Qualcomm Technologies’ solution
16
Re-focus
Segmentation with various background replacement
Qualcomm Technologies’ dual camera
Bokeh with variable aperture
Instant AF
17
Low light video & photo: Let there be light
• An algorithm for adaptively brightening areas of video/images that would otherwise appear too dark
BenefitsWhat’s new• Improves the quality of scenes that are
generally bright, but include dark regions• Fast and efficient noise-reduction for areas that
were under-exposed in the original frame
Low Light Video Snapdragon Low Light Video
18
Produce stable video despite moving hands/camera
Without EIS 2.0 With EIS 2.0
• Electronic Image Stabilization (EIS) is an image stabilization feature for video• The feature takes the sensor input and augments the Digital Image Stabilization (DIS). Resultant Hybrid solution
is substantially better than DIS
− EIS 2.0 compensates for 3 axis shake − Removes rolling shutter distortion
19
• Flash images usually suffer from over-exposedness and bluish tint (unnatural color).• Our ChromaFlash technology overcomes these problems algorithmically and produces and image that preserves
color, texture, and brightness.
Capture a real life image with ChromaFlash
No-flash image Flash image ChromaFlash Image
20
Dual camera features with even single camera - Qualcomm® UbiFocus™ camera feature
Qualcomm UbiFocus is a product of Qualcomm Technologies, Inc.
Out of focus Out of focus
Input Image 1 Input Image 2 UbiFocus
In focus In focus In focus
This is a post processing feature which takes five captured images as input
How it works:
Each image has its own desired pre-determined lens position as a capture requirement
The algorithm combines five images and output one fused image
The algorithm also generates a “map” that can be used for dynamic focusing
UbiFocus is a camera feature that generates an all-in-focus image from a mobile camera with a limited depth of field. Additionally, this feature allows users to selectively refocus their image in post-processing.
21
Snapdragon – enhancing sound qualityFocusing on broader dimensions
Pixel quantityResolution and frame rate
Contextual interactionsIntelligent and personalized
Pixel qualityColor accuracy, contrast,
and brightness
Natural user interfacesResponsive, accurate, and seamless
Visual quality
Sound quality
Intuitive interactions
Immersion&
Intelligence
High resolution audioSampling rate and precision
“True to life” sound3D surround sound and clarityReach out and Touch with your voice
22
Enriching sound by focusing on broader dimensionsProviding truly immersive audio experiences optimized for how humans hear
Sampling rateIncreased sampling rates to match human hearing
3D surround soundAccurate 3D capture and playback of audio
PrecisionIncreased bits-per-sample for increased audio fidelity
High resolution audio “True to life” sound
Clear audioZoom and focus on the important sound while filtering out the noise
23
Using multiple dimensions to capture vivid 3D sound
Understand environmentAutomatically adjust settings based on environment detection and voice identification
Isolate sounds of interestZoom, focus, and track the sounds of interest while separating out other sounds
Capture realistic soundCapture high-resolution, 3D surround sound through object-based or scene-based audio
Object-based sound captureScene-based sound capture
24
Zoom and focus on the important sound while filtering out the noise
Automatically tracks the dominant speaker• Audio source tracking in a 360° plane− 180° tracking with 2-mics
Useful outside of voice calls
• Gaming (player 1 vs. player 2)
Source localization & trackingConfigurable listen and quiet zones• Useful for camcorder
− Automatically only capture audio from in front of the device, and not the user holding the device
− Use Audio Zoom to increase the volume of the captured audio as you zoom in
Sound focus & Audio Zoom
1
24
3Zoom-In Area
25
End-to-end solutionsHigh-res audio, noise suppression, and 3D surround sound
Component tuning | Audio processing | System optimization
Size and cost constraints result in:• Limited number of mics• Increased noise
MicAddresses mic and speaker challenges
while enhancing audio
SoCSize and cost constraints result in:• Limited number of speakers• Reduced quality and fidelity
Speaker
Optimizing sound quality across the systemThe journey of audio from microphone to speaker is challenging
SoC
26
76%68% 66%
60%
49%
36%28%
21%
12%8% 8% 7%
Text Call Browse web
Email SocialNetwork
Listen to Music
Take Pictures
Watch TV/Videos
UseMaps
Record Videos
Mobile Bank
Read eBooks
Voice still the primary function of Smartphones!Nearly three quarters of Smartphone owners make calls daily
Qualcomm survey with n=1065 smartphone ownersQ3. For your current smartphone, how often do you do each of the following activities? 8pt – Several times/day, once/day, several times/week, once/week, 2-3 times/month, once/month, less than once/month, never
Smartphone Activity FrequencySeveral times/Once a day
Base: Total SP O N=1065
27
People still complain about voice on mobile devicesQualcomm technologies to address the challenges
Linear Analog Codec/ HD
Voice/ Full HD Voice
HD Voice/ Full HD Voice
Full HD Voice ECNS Dual/ Multi Mic NS
FENS ANC
Modem Centric Voice Solution/
VoLTE
EVS Channel Aware Mode
SRVCC, VCC, IMS PRO
eAMR, BeAMR, SRVCC
Qualcomm Technologies’ solution
Unnatural voice (lack of listening
comfort)
Voice clarity/
intelligibility
Sense of presence
Echo Background or ambient
noise
Cross talking
Call (set up) delay
Call drops Non-uniform voice quality
28
Always wideband voice (BeAMR) – Qualcomm Technologies customized codec
Received – Side Bandwidth Extension
GSM/WCDMA/TD-SCDMA/FDD/TTD-LTENone-TrFO, Legacy CS RAB & Device Interop
PSTN
AWV (BeAMR)HD Voice User Experience
Cellular Network Wide BandNarrow Band
Validation• Blind Enhanced AMR (BeAMR) is a receiver bandwidth extension (BWE) of the Enhanced AMR (eAMR)• BeAMR completes our HD Voice “Always HD” marketing initiatives• Minimizes impact of lack of AMR-WB coverage to mobile users at all times including in-network non-TrFO M2M, out-of-network M2M, VoLTE SRVCC,
IRAT and even L2M calls
• Can be a key turn-key voice differentiation feature for operators and OEM vendors by providing uniform HD Voice quality across the entire network
Technology and product development• BeAMR aka eAMR2.0 belongs to the same codec concept class as eAMR and uses the same codebase
• BeAMR operates in voice packet domain as compared with typical Blind Bandwidth Extension which generally operates in PCM domain• BeAMR is commercially available for both CS and VoLTE calls on MSM8994 and onwards.
29
Standard vs. Qualcomm Technologies customized speech codecs Relative voice quality comparison
HDOn is a product of Qualcomm Technologies, Inc.
AMR (12.2)
AMR-WB (12.65)
BeAMR (12.2)
eAMR (12.2)
EVS (13.2)
Standard Codecs HDOn™ codecs
Rela
tive
Subj
ectiv
e V
oice
Qua
lity
• High CAPEX deployment cost• Major radio & network upgrades• Extensive IoT & field testing
• Minimal CAPEX deployment cost• Only handset upgrades required• Interop to legacy RAB & network
WB
NB
SWB
WB
30
Crystal clear voice with Fluence™ noise cancellation technology
Fluence is a product of Qualcomm Technologies, Inc.
Dual-microphone noise cancellation• High stationary and non-stationary
noise suppression• Lower avg. vocoder bitrate in
noisy environmentsReceive voice enhancement• Render far end reference signal more
intelligible in noisy environments• Increase playback volume in noisy
environments
Wind noise rejection• Substantially reduces wind noise interference• Maintains full intelligible communication
in outdoor winds
Far-end noise suppression• Suppress stationary and non-stationary
noise coming from far-end and enhance DL voice quality
• Level of noise suppression can be adjusted on the fly
WideVoice• Creates wideband-like
extended speech• Blind Bandwidth Extension applied
to all narrowband speechEZHearing (Slow Talk)• Slows down speech without
affecting perceived characteristics • Increases speech intelligibility
for seniors and foreigners
31
Hear HD quality voice with Fluence HD™Fluence Solution – part of Fluence HD package
Snapdragon Voice Activation (SVA)• Use voice to start a device’s voice
natural user interface– No need to press a button, Just say
“Hey Snapdragon” to start the voice NUI
• Voice Activation detects the keyword and starts OS’s voice NUI
• Designed to only respond to the owner of the device
Adaptive Active Noise Cancellation• Adaptive handset ANC cancels out
background sound using the earpiece
• Improves voice intelligibility while in a loud environment
• AANC offers robustness across different handset positioning
• Low Power, Low cost implementation
“Always Wideband Voice” – Anytime, Any device, Any network• HD voice on legacy narrow band
• Customized end-to-end differentiated turnkey codec technologies which seamlessly can transform legacy narrowband network to HD Voice
• You “always” hear HD on Snapdragon
32
Ultimate voice experience with Fluence Pro
Robust echo cancellation & noise suppression solution for tablets & smartphones• Voice calls, VoIP calls, Voice recognition,
Video conferencing etc.• Handheld, desktop, round table
speakerphone use cases• Portrait and landscape mode usages
• Supports 3 & 4 mic configurations
- Flexible mic placement for NS (broadside or front-back)
Generic speakerphone support• Automatically tracks speaker movements
and provides consistent voice quality in dynamic scenarios
• Consistent non-stationary noise reduction that is robust to holding pattern variations- No degradation of non-stationary noise
reduction and retains similar performance as vertical holding
Surround sound recording• 5.1 audio capture for full HD Camcorder
experienceSource position tracking Sound Focus and Audio Zoom• Details on next page
33
Snapdragon – enhancing device interactionFocusing on broader dimensions
High resolution audioSampling rate and precision
Pixel quantityResolution and frame rate
“True to life” sound 3D surround sound and clarity
Contextual interactionsIntelligent and personalized
Pixel qualityColor accuracy, contrast,
and brightness
Natural user interfacesResponsive, accurate, and seamless
Visual quality
Sound quality
Intuitive interactions
Immersion&
Intelligence
34
Intuitive interactions immerse us in the moment
Natural user interfaces• Seamless• Responsive• Accurate
Contextual interactions• Intelligent• Personalized
Providing the appropriate interfaces based on the user, device, and
application
Providing the right level of immersion based on context
35
Natural user interfaces for intuitive interactions
Adaptive, multimodal, user interfaces
Voice recognition Use natural language
processing
Motion & gesture recognitionUse computer vision, motion sensors, or touch
Face recognitionUse computer vision to recognize facial expressions
Personalized interfacesLearn and know user preferences
based on machine learning
Eye tracking Use computer vision to measure point of gaze
Bringing life to objectsGive user interfaces to headless objects
36
Providing the right level of immersion based on contextTrusting your device to take appropriate actions
Contextual awareness• Scene recognition• Sensors• Learned preferences
Personalized experiences• Reduced friction• Personal assistant
Intelligent notifications• Right information at the right time• Interrupt immersion appropriately
37
Enabling new user experiences with Snapdragon
38
Devices, machines and things are becoming more intuitive Through new levels of on-device intelligence and cognitive capabilities
Reasoning Action Perception Act intuitively based on context
and user preferences Learn, anticipate, infer context
through machine learning Hear, see, and observe like humans
through sensors, computer vision
39
Offering a broad range of on-device cognitive capabilities
Qualcomm Snapdragon Sense and Qualcomm Snapdragon Smart Product are products of Qualcomm Technologie’s, Inc.
Intuitive security• Qualcomm® Snapdragon™ Smart Protect• Qualcomm® Snapdragon Sense™ ID biometrics
Visual perception• Scene Detect & Gallery
Classification• Snapdragon Text Activation
Voice and audio understanding
• Qualcomm® Snapdragon Sense™ ID Voice
Always-on awareness• Smart Connectivity
Immersive multimedia• Immersive Audio
On-device machine learning
Computer vision
Ultra-low powerSensor processing
Cognitive technologies
40
Supporting cognitive technologies on the device is key for many use cases
Enhanced reliability
Immediate response
Efficient use of network bandwidth
Security and user privacy
41
Smart audioImmersive audio
• Immersive 3D surround sound
Benefits
What’s new
• Perceived multi-channel surround through directional speaker virtualization
42
Smart visualsScene detect & gallery classification
• Near real-time image classification• No data sent off device
Benefits
What’s new
• Machine-learning automatically and intelligently recognize scenes
• Recognition done on-device, quickly and privately
43
Smart visualsSnapdragon text activation
• Intelligently engage with text as if it were already a link in your phone
Benefits
What’s new
• Recognize and Act on text in the real world• Support for Korean, English, Chinese with
more languages planned
Call NumberCompose E-mailOpen URL
Save ContactSearch WebNavigate
44
• Advanced water rejection and wet multi-finger tracking• Ultra low power mode• Improved touch responsiveness• Context awareness and sensor fusion• Improved UX and advanced algorithms through
Snapdragon host processing
What’s new
• Passive stylus and gloved touch• Superior cost to performance ratio• Improved time to market
Benefits
Qualcomm® improveTouch™ technologyCreating the next generation touch experience
Qualcomm improveTouch technology, a feature of Qualcomm Snapdragon processors, is a product of Qualcomm Technologies, Inc.
45
Snapdragon Smart Protect technologyReal-time, on-device behavioral analysis platform
Qualcomm Zeroth is a product of Qualcomm Technologies, Inc.
• Machine learning-based behavioral analysis• Based on the Qualcomm® Zeroth™ cognitive
computing platform• Almost instant detection and classification of
previously unknown malware
What’s new
Benefits
• Helps protect private, valuable data• Seamless user experience with negligible overhead• Strengthens signature-based anti-malware apps• Automatically adapts to each device
Traditional signature-based
detection
Snapdragon Smart Protect
behavioral-based detection
Comprehensive Protection
Safe content
Transformed malware
Unknown malware
Known malware Abnormal behavior
Photos taken while display is off
Abnormal behavior
SMS sent without user interaction
46
Adaptive smart connectivitySwitch between best connection intelligently and automatically with Connectivity Engine (CnE)
Signal Quality
LTE
Wi-Fi
Network Load vs. Capacity
Internet Reachability
Captive Portal
Internet
AGREE
Terms
Backhaul Throughput
Internet
100 Mbps 5 Mbps
Many variables impact which connection is “best”
3G/LTE
Wi-Fi
Operator Policy
3G/LTE HW
Wi-Fi HW
Internet-connected App
CnE
47
Wrap up
48
• One of the most advanced connectivity features in a mobile SoC including Cat 12/13 LTE and 802.11ad Wi-Fi supports high performance mobile computing
• Zeroth – the on-device machine learning platform powering cognitive applications to simplify and enhance user experiences
• Images & camcorder footage consistently delighting end users with quality, crispness, color and simplicity to capture− Intelligently focused on the subject, regardless of setting or situation
− Optimized settings and simplified operation for improved footage and image capture quality
− Highly efficient components working together to achieve higher performance per milliwatt, through tight hardware and software integration while delivering feature innovation
− Immersive mobile movie experiences
• High performance display processing subsystem delivers peak display graphics while reducing power and thermals− 4K wireless display performance rivaling modern living room experiences
− Console quality graphics
• Platform with strong support for advanced applications like Tango as well as other AR and VR integrations
Snapdragon – mastering Smartphone user experienceCombining immersion and intelligence to provide OEM’s a more creative platform solution
49
Efficient heterogeneous computing architecture
Within device constraints
Development time / Sleek form factor / Power and thermal efficiency / Cost
Qualcomm Technologies, Inc. is uniquely positioned to deliver superior immersive experiencesProviding an efficient, comprehensive solution in the Snapdragon 820
Visual quality
Immersive experiences
Sound quality
Intuitive interactions
Commercialization
Consistent, accurate color
In-focus images
Low-light video & photography
3D surround sound
Noise removal
Enlarged sweet spot
Seamless, responsive, smooth user interfaces
Intelligent, contextual interactions
Via Snapdragon™ solutions
Via ecosystem enablement
Comprehensive solutions across tiers
Custom designed processing engines
Snapdragon development platforms
Ecosystem collaboration
App developer tools
50
For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog
Nothing in these materials is an offer to sell any of the components or devices referenced herein.
©2013-2015 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Snapdragon, ChromaFlash and Fluence are trademarks of Qualcomm Incorporated, registered in the United States and other countries, used with permission. EcoPix, Qualcomm Spectra, OptiZoom, UbiFocus, HDOn, Snapdragon Sense, improveTouch and Zeroth are trademarks of Qualcomm Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable.
Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.
Thank youFollow us on: