Top Banner

of 66

20131102-Towards ECV-輔大

Jun 04, 2018

Download

Documents

Yuan-Kai Wang
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • 8/13/2019 20131102-Towards ECV-

    1/66

    Wang, Yuan-Kai()Electrical Engineering,Fu Jen University ()Email: [email protected],URL: http://www.ykwang.twSlideshare: Search Towards Embedded Computer Vision

    Towards

    EmbeddedComputer Vision

  • 8/13/2019 20131102-Towards ECV-

    2/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    2

    MIT Oxygen

    http://oxygen.lcs.mit.edu/ 1999

  • 8/13/2019 20131102-Towards ECV-

    3/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    3

    The Aware Home

    http://awarehome.imtc.gatech.edu/

  • 8/13/2019 20131102-Towards ECV-

    4/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Ambient Intelligence v.s.

    Object Intelligence

    Embedded system + Sensors

    4

    Wearable ComputingSmart Home, Smart

  • 8/13/2019 20131102-Towards ECV-

    5/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    55

    Embedded Systems

    "Without" Sensors

    MIC ITIS

  • 8/13/2019 20131102-Towards ECV-

    6/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    66

    Embedded Systems

    "With" Sensors

    GPS

    Navigation

    Wii

    Roomba

    Exoskeleton

  • 8/13/2019 20131102-Towards ECV-

    7/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    7

    Driver assist

    Front view

    Night vision

    Blind corner

    Blind spot

    Rear view

    Embedded Systems

    "With" Image Sensors

  • 8/13/2019 20131102-Towards ECV-

    8/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    88

    Embedded Systems

    "With" Image Sensors

    Google Glass

  • 8/13/2019 20131102-Towards ECV-

    9/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Smart Watch

    9

  • 8/13/2019 20131102-Towards ECV-

    10/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Microsoft

    SenseCam

    Embedded Systems

    "With" Image Sensors

    10

  • 8/13/2019 20131102-Towards ECV-

    11/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Observation

    Miniaturization

    High Resolution

    3D Capability

    11

  • 8/13/2019 20131102-Towards ECV-

    12/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    12

    Embedded Computer Vision

    ! Embedded System + Camera+ Computer Vision Algorithm

    Image

    Processing

    Image

    Recognition

    Image

    Capturing

  • 8/13/2019 20131102-Towards ECV-

    13/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    13

    Embedded Computer Vision

    ! Embeddedcompute vision" Implements computer vision algorithms

    on low-cost, low-power,

    constrained hardware! Constrainedhardware" Low-speed CPU" Low capacity memory" No floating-point (FPU)" Low-resolution image sensor

  • 8/13/2019 20131102-Towards ECV-

    14/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    A BResearch Status (1/3)

    14

  • 8/13/2019 20131102-Towards ECV-

    15/66

    T d E b dd d C Vi i 1616

  • 8/13/2019 20131102-Towards ECV-

    16/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    1616

    Research Status (3/3)

    DARPA

    Grand

    Challenge

    Google

    CarGoogle

    Taxi

    T d E b dd d C t Vi i 17

  • 8/13/2019 20131102-Towards ECV-

    17/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    17

    International Activities (1/2)!

    Special conferences" IEEE Int. Workshop on Embedded ComputerVision

    " IEEE Int. Conf. Distributed Smart Cameras! Special journal issues

    " IEEE Journal of Selected Topics in SignalProcessing, vol. 2, no. 4, Aug. 2008

    " EURASIP Journal on Embedded Systems! Short courses in important CV conferences

    " CVPR07&08: Distributed vision processing insmart camera networks" ESC07: Embedded CV and smart cameras" ICASSP09: Distributed processing in smart

    cameras

    T d E b dd d C t Vi i 18

  • 8/13/2019 20131102-Towards ECV-

    18/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    18

    International Activities (2/2)

    ! Research projects and Lab." Princeton Univ./Georgia Tech.:

    Embedded Systems Lab., Wayne Wolf" Stanford Univ.

    Wireless sensor networks Lab." UCLA, CMU, MIT" Delft Univ. of Technology

    SmartCam Project" Graz Univ. of Technology

    T d E b dd d C t Vi i 19

  • 8/13/2019 20131102-Towards ECV-

    19/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    19

    Three Ways for ECV

    ! CPU (Central Processing Unit)" ARM, PowerPC

    ! DSP (Digital Signal Processor)" TI, ADI, NXP

    ! FPGA (Field Programmable Gate Array)" Altera, Xilinx

    Towards Embedded Computer Vision p 20

  • 8/13/2019 20131102-Towards ECV-

    20/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    20

    CPU

    ! Stanford MeshEye" ARM 7 (55MHz), ZigBee node" 3 image sensors" 30x30 grayscale x 2" 640x480 color x 1

    Object detection

    (http://wsnl.stanford.edu/smartcam.html)

    "MeshEye:a hybrid-resolutionsmart camera mote for applicationsin distributed intelligentsurveillance", IPSN-SPOTS, 2007

    Towards Embedded Computer Vision p 21

  • 8/13/2019 20131102-Towards ECV-

    21/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    21

    DSP

    ! "Visual surveillance on DSP-basedembedded platform," Graz Univ. ofTechnology, 2008(Phd. dissertation)" TI C6414 (600MHz)" Applications: Adaboost face detection, vehicle

    detection, license plate detection

    Towards Embedded Computer Vision p 22

  • 8/13/2019 20131102-Towards ECV-

    22/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    22

    FPGA (2/3)

    "Hardware, Design andImplementation Issues on a

    FPGA-Based Smart Camera,"ICDSC, 2007

    Towards Embedded Computer Vision p 23

  • 8/13/2019 20131102-Towards ECV-

    23/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    23

    Hybrid: CPU+DSP

    ! "Distributed Embedded Smart Camerasfor Surveillance Applications,"IEEE Computer, 2006.

    Developed for traffic surveillance

    Towards Embedded Computer Vision p 24

  • 8/13/2019 20131102-Towards ECV-

    24/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Hybrid: CPU+DSP by SoC

    ! TI OMAP

    24

    Towards Embedded Computer Vision p 25

  • 8/13/2019 20131102-Towards ECV-

    25/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    25

    Hybrid: CPU+FPGA

    ! UCLA Cyclops" MCU: Atmega128" CPLD: image processing" ECV app.: Hand gesture recognition

    http://research.cens.ucla.edu/projects/2007/

    Multiscaled_Actuated_Sensing/

    Cyclops/

    Towards Embedded Computer Vision p 26

  • 8/13/2019 20131102-Towards ECV-

    26/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Hybrid: CPU+FPGA by SoC

    ! Xilinx Zynq

    26

    Towards Embedded Computer Vision p. 27

  • 8/13/2019 20131102-Towards ECV-

    27/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    27

    Challenges for ECV

    ! Algorithm refinement" Parallel computation" Function partition, Multi-threading

    " Stream processing" Memory hierarchy optimization

    ! Hardware design" Pipeline, SIMD, board design

    ! Optimized programming skills" Fixed-point arithmetic" Memory management" Intrinsic commands

    Towards Embedded Computer Vision p. 28

  • 8/13/2019 20131102-Towards ECV-

    28/66

    Towards Embedded Computer Vision p.

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    28

    Our Experiences

    Entertainment Robot (CPU)Vision Sensor (CPU)Video Surveillance (FPGA)

    Wearable Vision (DSP) Towards Embedded Computer Vision p. 29

  • 8/13/2019 20131102-Towards ECV-

    29/66

    p p

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Case 1Entertainment

    RobotCPU

    Towards Embedded Computer Vision p. 30

  • 8/13/2019 20131102-Towards ECV-

    30/66

    p p

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Sony AIBO

    Towards Embedded Computer Vision p. 31

  • 8/13/2019 20131102-Towards ECV-

    31/66

    p p

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Rolling Dice

    Towards Embedded Computer Vision p. 32

  • 8/13/2019 20131102-Towards ECV-

    32/66

    p p

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Face Detection

    Towards Embedded Computer Vision p. 33

  • 8/13/2019 20131102-Towards ECV-

    33/66

    p p

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Face Recognition

    Towards Embedded Computer Vision p. 34

  • 8/13/2019 20131102-Towards ECV-

    34/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Facial Expression Recognition

    Towards Embedded Computer Vision p. 35

  • 8/13/2019 20131102-Towards ECV-

    35/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Case 2Vision Sensor

    CPU

    Towards Embedded Computer Vision p. 36

  • 8/13/2019 20131102-Towards ECV-

    36/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    FJUCam

    Powerconsumption:1W

    Power sources:5V DC current5V Cell Battery

    Weight: 35gmSize:6 x 4.5 x 5 (cm)

    (W x H x D)

    Three Modules1. Main board, 2. Lens module

    3. Storage module

    Towards Embedded Computer Vision p. 37

  • 8/13/2019 20131102-Towards ECV-

    37/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Face Detection

    FJUCam

    Face Detection

    Image

    Display

    Vision Sensor Network

    Towards Embedded Computer Vision p. 38

  • 8/13/2019 20131102-Towards ECV-

    38/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Cyclops Robot

    ! Color tracking

    Towards Embedded Computer Vision p. 39

  • 8/13/2019 20131102-Towards ECV-

    39/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Case 3Video Surveillance

    FPGA

  • 8/13/2019 20131102-Towards ECV-

    40/66

    Towards Embedded Computer Vision p. 41

  • 8/13/2019 20131102-Towards ECV-

    41/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Background Subtraction

    Current

    Frame

    -

    Background

    Image Update

    Post Processing

    Motion Object Image

    Background Image

    1+kB

    kB

    1+kP

    1+kM

    ),(),(

    ),(

    1

    1

    yxByxP

    yxM

    kk

    k

    !=

    +

    +

    ),()1(),(

    ),(

    1

    1

    yxPyxB

    yxB

    kk

    k

    +

    +

    !+= ""

    Speed up by (1) Circuit design, (2) Paralization

    Towards Embedded Computer Vision p. 42

  • 8/13/2019 20131102-Towards ECV-

    42/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    FPGA Speed Up

    ! Comparisons" PC: 2.8GHz CPU, C implementation" FPGA can speed up 500 times

    2.8G

    25M

    51

    10

    Towards Embedded Computer Vision p. 43

  • 8/13/2019 20131102-Towards ECV-

    43/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Case 4Wearable Vision

    X-EYE

    ARM + DSP

    Towards Embedded Computer Vision p. 44

  • 8/13/2019 20131102-Towards ECV-

    44/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Goal

    44

    Towards Embedded Computer Vision p. 45

  • 8/13/2019 20131102-Towards ECV-

    45/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Demo Video

    Towards Embedded Computer Vision p. 46

  • 8/13/2019 20131102-Towards ECV-

    46/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Development

    SD USB-WIFI

    SB-RS232

    SB

    BeagleBoard

    CameraUSBHub

    Towards Embedded Computer Vision p. 47

  • 8/13/2019 20131102-Towards ECV-

    47/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Hardware

    4747

    Peripheral I/O

    USB HostJTAGDVI-D video outS-Video outSD/MMC+Stereo in/outRS-232 serial1Alternate powerUSB 2.0 HS OTG

    USER

    OMAP3530 Processor600MHz Cortex-A8

    NEON+VFPv316KB/16KB L1256KB L2

    430MHz C64x+ DSP32K/32K L1

    48K L1D32K L2

    Power VR SGX GPU64K on-chip RAMPOP Memory256MB LPDDR RAM256MB NAND flash

    RESET

    7.6 cm Towards Embedded Computer Vision p. 48

  • 8/13/2019 20131102-Towards ECV-

    48/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    System

    2010.04.25

    48 Towards Embedded Computer Vision p. 49

  • 8/13/2019 20131102-Towards ECV-

    49/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Software

    Towards Embedded Computer Vision p. 50

  • 8/13/2019 20131102-Towards ECV-

    50/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    FJUCam2

    Towards Embedded Computer Vision p. 51

  • 8/13/2019 20131102-Towards ECV-

    51/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Next : UAV

    Towards Embedded Computer Vision p. 52

  • 8/13/2019 20131102-Towards ECV-

    52/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Next : Sleep Monitoring

    Towards Embedded Computer Vision p. 53

  • 8/13/2019 20131102-Towards ECV-

    53/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Next : Vehicular

    Object Detection

    Towards Embedded Computer Vision p. 54

  • 8/13/2019 20131102-Towards ECV-

    54/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Heterogeneous Multicore

    ! Xilinx Zynq: ARM+FPGA

    Towards Embedded Computer Vision p. 55

  • 8/13/2019 20131102-Towards ECV-

    55/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Future ofEmbedded Vision

    Towards Embedded Computer Vision p. 56

  • 8/13/2019 20131102-Towards ECV-

    56/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Summary

    ! Embedded CPU is not appropriate forECV" Pros" Great flexibility" Fast development

    " Cons"

    No float-point arithmatics"Interrupt-based architecture

    ! Smart phone with 4/8 cores is OK? Towards Embedded Computer Vision p. 57

  • 8/13/2019 20131102-Towards ECV-

    57/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    I Think!

    !Embeddedcompute vision"Low-cost, low-power,

    minimal hardware

    ! High-Performancecomputer vision" Fast speed without cost,

    power, and hardware

    constraintsFrom contradictionto convergence ?!

    ConstrainedResource

    AbundantResource

    Yes by multicore

    Contradiction

    Towards Embedded Computer Vision p. 58

  • 8/13/2019 20131102-Towards ECV-

    58/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Challenges (1/2)

    ! Algorithm decomposition" Function decomposition" Partition serial part and parallel part" Data flow analysis and

    data dependency analysis" Parallelism" Loop unrolling"

    Multithreading" Pipeline

    Towards Embedded Computer Vision p. 59

  • 8/13/2019 20131102-Towards ECV-

    59/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Challenges (2/2)

    ! Performance analysis method" For efficiency improvement! Implementation efforts" Choose a good embedded platform for

    computer vision" Software issues" Hardware issues

    "Programming skills" Multi-threading

    Towards Embedded Computer Vision p. 60

  • 8/13/2019 20131102-Towards ECV-

    60/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Future Research Directions

    HeterogeneousMulticore

    ARM+ DSP

    ARM +FPGA

    ARM +GPGPU

    Towards Embedded Computer Vision p. 61

  • 8/13/2019 20131102-Towards ECV-

    61/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    I Think Again!

    ! Is there a development platform forembedded computer vision with" Built-in camera" AE, AF, AWB

    " High-performance processor" Heterogeneous multicore

    " Software development environment:"

    C, Matlab" OpenCV" Sample codes" Labs

    Towards Embedded Computer Vision p. 62

  • 8/13/2019 20131102-Towards ECV-

    62/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    Embedded Vision Alliance

    Towards Embedded Computer Vision p. 63

  • 8/13/2019 20131102-Towards ECV-

    63/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    ! Ramesh RaskarMIT Platform

    Towards Embedded Computer Vision p. 64

  • 8/13/2019 20131102-Towards ECV-

    64/66

    Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

    ! Andrew AdamsStanford Platform

  • 8/13/2019 20131102-Towards ECV-

    65/66

    Towards Embedded Computer Vision p. 66

  • 8/13/2019 20131102-Towards ECV-

    66/66

    !Creative Commons - 3.0 ! (1) (2)!