CONFIDENTIAL 1 | © 2019 MIPI Alliance, Inc. Evolution of MIPI Imaging Conduit October 18, 2019 Haran Thanigasalam MIPI Camera & Imaging WG Chair
CONFIDENTIAL 1 | © 2019 MIPI Alliance, Inc.
Evolution of MIPI Imaging Conduit
October 18, 2019
Haran ThanigasalamMIPI Camera & Imaging WG Chair
2 | © 2019 MIPI Alliance, Inc.
Agenda • The Big Why
o Two key discoverieso Machine human frictiono Two pathways
• MIPI Imaging Conduit Solutiono Multi-phase developmento What’s Doneo What’s WIPo What’s Next
• Summary
3 | © 2019 MIPI Alliance, Inc.
The Big Why
URL
• Enrich lives by better understanding the world around us
• Knowledge limitations
• Scope of perception
• Challenges with abstraction
• Evolution of intelligence
• Two key discoveries
1. Individual & Collective Dysfunction
2. Superior Intelligence
URLURL
4 | © 2019 MIPI Alliance, Inc.
MIPI CSI-2 Imaging Conduit 2019
Machine human friction• Future is deeper human machine interaction
with connected devices
• Two Pathways:
1. Making us become more like machines
2. Making machines become more like us (machine awareness, emotional vision)
URLURL
5 | © 2019 MIPI Alliance, Inc.
Evolving MIPI Imaging Solution
IngredientsPhoton Collectors, Transistors, Optics,
Emitters, Algorithms
ApplicationsBroad Spectrum of Imaging & Vision Use cases targeting
multiple platforms
CSI-2 Imaging Conduit
One World l One Imaging Conduit l MIPI CSI-2
URL
I. Mobile - Pristine photography & video streaming on mobile platforms [CSI-2 v1.x]- RES_FPS_BPP | PORT EXP | SNS SWITCHING
II. Platforms - Support broad range of imaging applications beyond photography on multiple platforms [CSI-2 v2.x, v3.x]- SCR | VCX | LRTE (PDQ, ALP) | USL | RAW-24 | DPCM | SROI
III. Awareness - Develop infrastructure for emotional intelligence and machine awareness for perception and decision making capabilities [CSI-2 v4.x]- AOSC | FSAF | ISEC |ADAPL
IV. Scale - Open System Cloud Imaging Applications utilizing vision analytics (Schools, Hospitals, Municipalities, States)
V. WIPComplex problems cannot be solved by an individual company
Need for global alliances and liaisons: MIPI | DMTF | IEEE 1722
Evolution of MIPI Imaging Conduit Infrastructure
6 | © 2019 MIPI Alliance, Inc.
Two different CSI architectures
M-PHYPHY
Transport
Applications
CSI-2UniPro
CSI-3
D-PHYC-PHYA-PHY
CSI-2
C-PHYD-PHYI3C A-PHY
• MIPI CWG is continuing to evolve and advance CSI-2 imaging conduit targeting multiple platforms:•I3C: Ultra-Low-Power AON•C/D-PHY: Performance •A-PHY: Long Reach
• CSI-3 is not a next-gen solution, nor superset, nor backwards compatible with CSI-2
7 | © 2019 MIPI Alliance, Inc.
What’s Done
SROI UNIFIED IMAGING DRIVER(CCS, DisCo*)
LRTE(PDQ, ALP)
USL(ENCAP, REPL, OPT_WIRE)
DPCM | RAW-24
PSD
8 | © 2019 MIPI Alliance, Inc.
PSD Reduction
CSI-2 over C-PHY PSD emission reduction with scrambling (embedded clock and data)
CSI-2 over D-PHY PSD emission reduction with scrambling (data lanes)
9 | © 2019 MIPI Alliance, Inc.
LRTE
• Dramatically Improve Sensor Aggregation
• Optimal Transport Preserving Integrity
• Real-time Perception & Decision Making
• Phase out EOS &IL Impediments
• Asymmetric SP/LP Spacers
10 | © 2019 MIPI Alliance, Inc.
11 | © 2019 MIPI Alliance, Inc.
Smart Region of Interest
https://fuse.wikichip.org/news/763/iedm-2017-sonys-3-layer-stacked-cmos-image-sensor-technology/
3-layer stacked image sensor
• Phase 2 Single Frame
• Phase 4 Multi Frame
• ARCH: Edge |Hybrid
12 | © 2019 MIPI Alliance, Inc.
DPCM Objective Qualification• Enable 10-bit compression of RAW-12 video image with better IQ than
prior version 12-8-12
• Reduce maximum absolute error of single-bit change in pixel value by a factor of 4.43x
• Qualified 5 degree slanted edge input image with low, medium, and high illumination levels:
– 12-10-12 virtually indistinguishable from original image
– MTF frequency response analysis closely tracks the original (HI/MI/LI - LC/MC/HC)
• Benefits include Link BW reduction cost savings
13 | © 2019 MIPI Alliance, Inc.
• Unified Imaging Software Driver
• Enable Image Sensor Capabilities for Vision Compute
• Accelerate Bring up on Reference Platforms
Algorithm is King
• Advance pathways to realize image sensor self-tuning
14 | © 2019 MIPI Alliance, Inc.
CSI-2 CCS
15 | © 2019 MIPI Alliance, Inc.
What’s WIP
ISEC• Develop provisions for end-to-end imaging conduit security
(interface and class)• Authentication - trustworthy / genuine sensor module
with certificate signing• Data signing - data is originating from trusted source
devoid of spoofing attack• Obfuscation - Data encryption at line rate
• External liaisons with Distributed Management Task Force Security Protocol & Data Model development• Authentication• Session key exchange
FSAFMission Critical Imaging Applications• End to End CRC Protection • Enhanced Frame Structure • Fault Injection Mode
AOSCVDSP
SNSMulti-Mode Image
APP
Point-To-PointCSI-2 over I3C
• Enable always-on surrounding awareness
• Infrastructure to accommodate rapid evolution of vision inferencing
• Ultra-low-power operations
A-PHY ADAPL• Enable native long-reach support for DISCRETE and
INTEGRATED image sensors
16 | © 2019 MIPI Alliance, Inc.
Always On Sentinel Conduit (AOSC)
VDSP
CSI-2 over C/D-PHYSNS
Dual-Mode Image Sensor(Cropped | Full-Frame)
APP
Application Processor
Point-To-PointCSI-2 over I3C
• Enable always-on surrounding awareness
• Infrastructure to accommodate rapid evolution of vision inferencing
• Ultra-low-power operations
Always On Imaging ApplicationsSNS Guidance
<2mw @ 3FPS, 8-10BPP, QVGA Monochrome
Select applications may require higher capabilities
17 | © 2019 MIPI Alliance, Inc.
End-to-end Security • Develop provisions for end-to-end imaging
conduit security (interface and class)• Authentication - trustworthy / genuine
sensor module with certificate signing• Data signing - data is originating from
trusted source devoid of spoofing attack• Obfuscation - Data encryption at line rate
• External liaisons with Distributed Management Task Force Security Protocol & Data Model development• Authentication• Session key exchange
18 | © 2019 MIPI Alliance, Inc.
CSI-2 ISEC Considerations Device Manufacturing A device shall not reveal the device unique asymmetric private key to device manufacturing personnel during manufacturing.
Device shall only reveal the device unique asymmetric private key to authorized device manufacturing personnel while programming the device.
Platform Manufacturing A device shall not reveal the device unique asymmetric private key or a host shall not reveal the host unique asymmetric private key to platformmanufacturing personnel.A device shall only reveal device unique symmetric key to authorized platform manufacturing personnel while programming the device on the CSI-2 linkinterface.
Platform Refurbishing A device shall not reveal the device unique asymmetric private key to platform refurbishing personnel.
A device shall only reveal device unique symmetric key to authorized platform refurbish personnel while programming the device on the CSI-2 link interface.
Host Firmware Update The host firmware update shall not require a retransmit of the device unique symmetric key on the CSI-2 link.
Model Authentication Host shall be able to determine certified device model during platform manufacturing / refurbishing. The certification level shall be available throughmetadata published by the device vendor.
Device Authentication Host shall be able to determine that the device has not changed.Device Session Key Rotation Device shall support rotation of session keys for runtime activities.Device Firmware Measurement Device shall generate the mutable firmware measurement response using only its immutable code or HW
Device Firmware Measurement Replay Host shall be able to determine that device firmware measurement is replay protected.Device Firmware Rollback Host shall provide a policy to prevent device firmware rollback.Device Data Integrity Host shall be able to determine that received device data is the same as data captured from the image sensors on the device without modification or replay
of the previously captured data.Device Data Confidentiality Device may protect Privacy sensitive data from a rogue actor able to observe the CSI-2 link. For long reach IOT platforms Host/Device shall protect Privacy
sensitive data from a rogue actor able to observe the long reach cable.
Host Authentication Device shall be able to determine the identity of the host in a single -master environment. Device may be able to determine the identity of the host in amulti-master environment.
Host Command Integrity Device shall be able to determine the integrity of the host command including region of interest, virtual channel etc.
Device Revocation Host shall be able to revoke a vulnerable device.Device Tamper Detection Device shall indicate tamper evidence in its response.
19 | © 2019 MIPI Alliance, Inc.
CSI-2 Functional Safety
Mission Critical Imaging Applications (ISO-262262: 2018)• End to End CRC Protection • Enhanced Frame Structure • Fault Injection Mode
20 | © 2019 MIPI Alliance, Inc.
Importance of Machine (Emotional) Awareness
I. RES | FPS | BPP | CCI (CSI-2 v1.x)
II. VCX | PSD | LRTE | DPCM (CSI-2 v2.x)
II. USL | SROI | RAW-24 (CSI-2 v3.0 )
III. AOSC | FSAF | ISEC | A-PHY (CSI-2 v4.0)
SNS APPECV
CSI-2 SecureImaging Conduit
FFT
EMPATHY
HAPPY
BORED
SNS
21 | © 2019 MIPI Alliance, Inc.
CSI-2 v4.0 Imaging Conduit
• Always On Sentinel Controller (AOSC)– Optimal infrastructure for VDSP: ultra-low power, simple, always on CSI-
2 transport mechanism using I3C I/O – Target Platforms: Mobile, IoT & Client, Autonomous
• Functional Safety (FSAF)– Develop safety provisions for mission critical decision making
applications on autonomous platforms with areas of interest may include Message Counter (MC), SourceID, End-To-End CRC/MC protection, Enumeration of Rows & Columns.
– The solution shall also meet the needs of Automotive ISO-26262: 2018 requirements for ASIL-B to ASIL-D.
– Target Platforms: IoT, Autonomous (ADAS)
• Imaging Security (ISEC)– Develop provisions for end-to-end imaging conduit authentication &
data protection required for privacy.– Establish liaisons with DMTF – Target Platforms: Mobile, IoT & Client, Autonomous
• Adaptation Layer (ADAPL) for A-PHY– Support integrated and discrete solutions couple to CSI-2 over C/D-PHY– Long-reach SerDes solution for imaging applications.– Target Platforms: Autonomous (ADAS)
• Time Synchronization Provision (TSP)• Explore development of time synchronization including 1)
Transport and 2) Enablement• Target Platforms: Mobile, IoT & Client, Autonomous
• Multi-Sensor Support (MSS) • Explore supporting two or more C/D-PHY image sensors
coupled to single RX port on Application Processor• Target Platforms: Mobile, IoT (AR / Wearable Headgears),
Client, Autonomous
• Radar and LiDar Enhanced Support• Explore enhanced provisions to transport RAW data
from Radar and LiDAR image sensors for ADAS
• Unified Imaging SW Driver | Self Tuning• Continue to advance CCS aligned with CSI-2
• Develop DisCo for Imaging v1.0 with SWWG
• Alleviate Product Interoperability Risk• Continue to advance ETE CTS for CSI-2 (X-Matrix)
HW & System HW & System
SW & Drivers
Interoperability
PHYs & I/Os: A/C/D-PHY & I3C
22 | © 2019 MIPI Alliance, Inc.
Problem Definition (Use Cases)| Timing (Execution Vs Perfection)
Innovation Growth Maturity
Phase I(Mobile)
Phase II(Platforms)
Phase III(Awareness)
MIPI CWG
Time
Phase IV(Scale)
23 | © 2019 MIPI Alliance, Inc.
What’s Next – Computer Vision Systems
SNSModule
LocalPROC
CloudBackend
NetworkConnection
(IEEE 802.3 | IEEE 802.11)
STD | USL | AOSC | ADAPL
MIPI CSI-2
Open System Cloud Imaging Applications Vision Analytics | 3D Perception | Training model (self, supervised) | Deep Learning | Object Classification
24 | © 2019 MIPI Alliance, Inc.
PHYs & I/Os
CSI-2
IEEE-1722
IEEE 802.3(ETH)
IEEE 802.11(WiFi)
CSI-2
C-PHYD-PHYI3C
A-PHY (DIS)
CSI-2
C-PHYD-PHYI3C A-PHY(INT)
Natively Supported• I3C: Ultra-Low-Power• D-PHY: Performance• C-PHY: Performance • A-PHY: Long Reach
Potential Exploration / Evaluation• IEEE-1722 for ETH/WiFi
25 | © 2019 MIPI Alliance, Inc.
Closed System PHYs
• Closed System Mid-Range solutions targeting Q2 2019 development• Closed System Long-Range solution targeting Q1 2020 development• Potential IEEE 1722 evaluation for Open System developments
26 | © 2019 MIPI Alliance, Inc.
CSI-2 over C/D-PHYs (2 Lanes)
D-PHYRX
(6-pins)CSI-2
RX
D-PHYTX
(6-pins)CSI-2
TX
Periodic clock
Data
I2C Compatible 2-wire Camera Control
CSI-2 Imaging over D-PHY
Pin compatible
CSI-2D-PHY
ImageSensor
ApplicationProcessor
C-PHYRX
(6-pins)
C-PHYTX
(6-pins)
I2C Compatible 2-wire Camera Control
CSI-2 Imaging over C-PHY
CSI-2C-PHY
ImageSensor
ApplicationProcessor
Embedded clock & data
CSI-2TX
CSI-2RX
AF Gyro OIS MEMS
AF Gyro OIS MEMS
Flash
Flash
2.3x Effective Pixel BW
1x Effective Pixel BW
27 | © 2019 MIPI Alliance, Inc.
Unified Serial Link
D-PHYRX CSI-2
RX
D-PHYTXCSI-2
TX
Periodic FWD clock
Data0
Camera Control Interface (CCI) over I2C / I3C / SPI
ImageSensor Module
ApplicationProcessor
GPIOs
D-PHYTRX CSI-2
USL
D-PHYTRXCSI-2
USL
ImageSensor Module
ApplicationProcessor
C-PHYRX CSI-2
RX
C-PHYTXCSI-2
TX
EMB_CD_TRIO_0
Camera Control Interface (CCI) over I2C / I3C / SPI
ImageSensor Module
ApplicationProcessor
GPIOs
C-PHYTRX CSI-2
USL
C-PHYTRXCSI-2
USL
ImageSensor Module
ApplicationProcessor
CSI-2 USL over D-PHY CSI-2 USL over C-PHY
Periodic FWD clock
Data0
EMB_CD_TRIO_0
USL USL
• Builds on LRTE• Wire Reduction• Encapsulation• Replay Protection• Long reach
28 | © 2019 MIPI Alliance, Inc.
Latest CSI-2 v3.0 over C/D-PHY Port BW• CSI-2 v3.0 C/D-PHY BW capabilities per lane
– C-PHY v2.0: Standard Channel 6 GSps / Lane = 6 x 2.28 Gbps / 3 wires, Short Channel 8 GSps / Lane
– D-PHY v2.5: Standard Channel 4.5 Gbps / Lane = 4.5 Gbps using 4 wires, Short Channel 6.5 Gbps / Lane
– D-PHY v3.0 will provide higher BW– trending spec completion Q1 2020
CSI-2 v3.0 over C/D-PHY Imaging Conduit Effective (Usable) BW Solutions:
• 9 Wire CSI-2 over C-PHY: 41.1 Gbps [Standard Channel] (3 x 6 x 2.28)• 10 Wire CSI-2 over D-PHY: 18.0 Gbps [Standard Channel] (4 x 4.5)
29 | © 2019 MIPI Alliance, Inc.
CSI-2 over C-PHY - 2.3x Effective BW
• 3 terminated conductors
• CSI-2 policy: symbol transition each state for trivial CDR
• Theoretical BW gain: log2^5
• As implemented for CSI-2 = 16/7
30 | © 2019 MIPI Alliance, Inc.
CSI-2 over A-PHY• Discrete & Integrated• NRZ: Low Cost: NRZ (8b/10b)• PAM: Scalable Performance: PAM-X (i.e. PAM-4)• O: Optional | M: Mandatory• Cable Topologies (Image Sensor PCB à Processor ECU PCB)
– 50 Ohm Coax | 15 m | 4 inline connectors– 100 Ohm SDP | 10 m | 4 inline connectors
G1(2 Gbps)
G2(4 Gbps)
G3(8 Gbps)
G4(12 Gbps)
G5(16 Gbps)
NRZ
M M O
PAM
M M M
32/48/100 Gbps
PAM
31 | © 2019 MIPI Alliance, Inc.
Platform Considerations
APP
ISPCVE
AGG
SNS MODULE
IRXVCMGYROACCEPROMLEDLASER…
ARCH: Mobile | Client | IoT | Autonomous
CENTRAL
EDGE / DISTRIBUTED
HYBRID
PHY
DISCRETE
INTEGRATED
GFX
SNS
VDSP
AOSC
CSI-2 v4.0 FSAF DIS
CSI-2 v4.0 FSAF EN
CSI-2 v4.0 FSAF ENUSL ENLRTE EN
CSI-2 v2.1SCR EN
DPCM EN
CSI-2 v4.0AOSC
A-PHY SerDes
CSI-2 v4.0AOSC
CSI-2 v4.0 (INT) USL ENLRTE ENSCR_EN
CSI-2 v2.1SCR EN
DPCM EN
CSI-2 v3.0SROI ENUSL EN
CSI-2 v1.3SCR ENRAW 12
CSI-2 v4.0
CSI-2 v4.0 supports multiple features over 4 modes of transport
STD USL AOSC ADAPL
RAW ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص
DPCM ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص
SCR ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص
LRTE ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص X ملسو هيلع هللا ىلص
SROI ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص
FSAF ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص
ISEC ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص
STD – Unidirectional C/D-PHY, CCI over I2C / I3C / SPI, GPIOsUSL – Encapsulation using bidirectional C/D PHYAOSC – Encapsulation using bidirectional I3CAPHY – Integrated and Discrete SerDes transport
32 | © 2019 MIPI Alliance, Inc.
• New automotive white paper to be published this week• A-PHY to support broadest spectrum of speed needs using two profiles • Auto-related features included in CSI-2 v3.0, as well as next camera and display
specifications • Now identifying requirements for A-PHY v2.0
Recent activities:
33 | © 2019 MIPI Alliance, Inc.
• Information on new features and specification releases
• Use cases and applications of MIPI specifications
• Q&As with working group chairs and other experts
• Latest MIPI Alliance news
• Highlights and key takeaways from webinars and MIPI DevCon presentations
• Details of MIPI participation in industry events
In the blog:
mipi.org/blog
34 | © 2019 MIPI Alliance, Inc.
Summary• Resolve problems that cannot be solved by single company
• Advance provisions to enable machine awareness through sighto Methodical phased development: Mobile | Client | IoT | Automotive
• Democratize benefits of AI vision through standardization
• Together we go further o PHY WG: A/C/D-PHYso SWG: I3Co SWWG: DisCo for Imaging o Liaisons: DMTF | ETH/WiFi
CONFIDENTIAL 35 | © 2019 MIPI Alliance, Inc.
Thank you!
www.mipi.org
36 | © 2019 MIPI Alliance, Inc.
Backup Material
37 | © 2019 MIPI Alliance, Inc.
Virtual Channel Expansion
URL
System Considerations
• Proliferation of Image Sensors Require CSI-2 Aggregators and VC Expansion
• System is limited to 16 or lower VCs:– Supported by CSI-2 over C/D-PHY
• System requires more than 16 VCs:– Supported by CSI-2 over C-PHY
URL
38 | © 2019 MIPI Alliance, Inc.
Mobile
Target Platform: Smart Phones, Tablets
Target Imaging Application: Pristine still photography l
streaming video
Client & IoT
Target Platforms: Content Creation (AIO, Notebooks, Hub)
Target Imaging Application: Comprehensive E2E imaging conduit to enable machine
awareness
Autonomous
Target Platform: Automotive, Drones, Medical, Factory
Target Imaging Application: Real time perception and
decision making
IP Reuse / Engineering
Imaging System Platforms
39 | © 2019 MIPI Alliance, Inc.
Evolving CSI-2 Architecture for Automotive
CSI-2 Vision
Long Range Radar
Video Camera / LiDAR
UltraSonic
CSI-2 Aggregators may benefit from Virtual Channel Expansion and LRTE PDQ
CSI-2 streaming data from aggregated imagesensors for near real-time perceptionand decision making platform solutions
40 | © 2019 MIPI Alliance, Inc.
Deep Neural NetworksMyriad X Hybrid Arch Deep Neural Network 8 Cameras using 16 MIPI Lanes - 700 MP/s
Drone Collision Avoidance (DJI SPARK)
http://www.electronicdesign.com/industrial-automation/intel-s-myriad-x-vision-chip-incorporates-neural-network