PowerPoint
SakuraSensor: Quasi-Realtime Cherry-Lined Roads Detection
through Participatory Video Sensing by CarsShigeya Morishita, Shogo
Maenaka, Daichi NagataMorihiko Tamai, Keiichi Yasumoto, Toshinobu
Fukukura, Keita SatoNara Institute of Science and TechnologyDENSO
CORPORATION
Thank you chairperson.Good afternoon, everyone.My name is
Shigeya Morishita from Nara Institute of Science and Technology.I
am very happy to see all of you today.Today, I would like to
present our research named sakura sensor.1
Latest car navigation systemsHelp drivers search comfortable
& efficient routes
CriteriaTraveling distanceTraveling timeToll/Toll-freeFuel
efficiencyScenic beauty2
NAVITIME(http://products.navitime.co.jp/function/2519.html)Toll-freeFuel-efficientMinimum
distanceToll Scenic
Latest car navigation systems help drivers with comfortable and
efficient driving. With these systems, we can search routes by
various () criteria.Among these criteria, we focus on scenic
beauty.2
Scenic route searchProblems of existing servicesInformation is
edited manuallySmall number of scenic spotsLow update
frequencyScenery information consists of only texts and
imagesinsufficient for users
3Our approachUse participatory sensing by carsCollect and share
videos of scenic spotsExample of scenic spot info.
However, existing scenic route search services have some
problems.First, information is edited manually.Second, scenery
information consists of only texts and images.To solve these
problems, approach, we use participatory sensing by cars and
automatically collect and share videos () of scenic spots.3
Related work4MethodProposed methodParkNet [12]SignalGuru
[15]Nericell [3]Participatory sensingCooperative
sensingReal-timeInformation detection from videos (ultrasound
signals)(traffic signals) (horn sounds)
[12] ParkNet: Drive-by Sensing of Road-Side Parking Statistics,
MobiSys10[15] SignalGuru: Leveraging Mobile Phones for
Collaborative Traffic Signal Schedule Advisory, MobiSys11[11]
Nericell: Rich Monitoring of Road and Traffic Conditions using
Mobile Smartphones, SenSys08Many existing studies on participatory
sensing (PS) by carsNo studies use both PS and real-time video
sensing
There are many existing studies on participatory sensing by
cars.However, as long as we know, no studies use both participatory
sensing and real-time video sensing.
4
SakuraSensor: automatically identifies scenic spots location and
collects videos using PS
we target cherry-lined roads automatically collect and update
scenic information gathering videos of scenic locationThe best
period of flowering cherriesis short and uncertain from year toyear
and from place to place
We propose Sakura Sensor, which automatically identifies scenic
spots location and collects videos using participatory
sensing.SakuraSensor targets flowering cherries called SAKURA in
Japanese, sincethe best period of flowering cherries is short and
uncertain from year to year and from place to place.So, up-to-date
information is mandatory.5
SakuraSensor App for iOS devices6Full size video -
https://youtu.be/2pRfDS7DeAcDemo at Hall C No.20
We have developed SakuraSensor application for iOS devices.Ill
show a demo video of sakurasensor.We are also demonstrating
SakuraSensor at hall C number twenty.
6
Key Idea7
Cloud
Cars with SmartphoneToo much cost for cellular bandwidth &
computation resource at cloudRecording videoAnalyzing &
sharingvideo with cherriesUpload whole recorded video
Recording videoAnalyzingvideoUpload only video with flowering
cherriesSharingvideo with cherries
One possible approach to realize SakuraSensor is as follows.
Cars with smartphone record videos and upload the whole recorded
video to cloud server for analysis and sharing.However, this
approach takes too much cost for cellular bandwidth and computation
resource at cloud.The key idea of SakuraSensor is analyzing video
at smartphones so that only video with flowering cherries are
uploaded to the cloud and shared.
Challenges and key ideas of SakuraSensor
7
Technical ChallengesTC1: Real-Time flowering cherry detection by
smartphone
TC2: Efficient load distribution among cars8
We have two technical challenges to realize SakuraSensor.First
challenge is how to realize real-time flowering cherry detection by
smartphone.Second challenge is how to realize efficient load
distribution among cars.
8
TC1: Real-time Cherry DetectionEmploy simple computer vision
techniquesSmart phone has lower computation power than PC/Cloud
Basic approachCount cherry-like color pixels in each image
Identify amount of flowering cherry as cherry intensity
Problem to solveArtificial objects with similar color must be
removed9
For the first challenge, we employ simple computer vision
techniques since smart phone has lower computation power.So, our
basic approach is just to count chery-like color pixels in each
image and identify amount of flowering cherry In each image called
cherry intensity ().Here, the problem to solve is that artificial
objects with similar color must be removed.9
Step1: Removing Artificial Objects
An input imageBinary image after edge detection
box counting method [5]fractal dimensions10Employ fractal
analysisNote: natural objects has higher fractal dimension
To remove artificial objects in each image, we employ fractal
dimension analysis.Here, note that natural objects has higher
fractal dimension. So, to an input image, we apply edge detection
algorithm, and box counting methodto calculate fractal dimension of
each square region.10
Real-time fractal dimension calculation11Red regions show
natural objects
This is Real-time fractal dimension calculation.Here red color
regions show natural objects.11
Step2: Detecting Cherry by Color Analysis
12Used 148 regions extracted from various scenesCreated color
histogram of flowering cherry in HSV color space
Then we detect flowering cherry by color analysis.We created
color histogram of flowering cherry in HSV color space. Here, we
used 148 regions extracted from various scenes.These are part of
the regions.12
HSV color spaceHHueSSaturationVValue of Brightness
From http://en.wikipedia.org/wiki/HSL_and_HSV
characterizes the color significantly varies depending on the
lighting condition13Our approachused only H-S color space
HSV color space consists of Hue, Saturation and Value of
brightness.From preliminary experiment, we found that V
significantly varies () depending on the lighting condition. So, we
used only H-S color space.13
H-S histogram for flowering cherry
HS01790255
Created from total of 148 cherry regions
The value at each coordinate is normalized between 0 and 114
This is the H-S histogram created from 148 cherry regions.The
value at each coordinate is normalized between 0 and 1.
14
Step3: Calculating cherry intensity of an image
HS00Pixels (H, S)=(30, 20)
The value of (30, 20) is 0.816
An input imageCherry intensity = average value of all pixelsUse
Backprojection method [6]
Then we calculate cherry intensity of an image by using
backprojection method.For each pixel, the value is retrieved in the
H-S histogram Finally, cherry intensity of the image is calculated
as the average value of all pixels.15
Real-time cherry intensity calculation16Red boxes show high
cherry intensity regions
This is real-time cherry intensity calculation.Here, red boxes
show high cherry intensity regions.16
TC2: Load Distribution among CarsWhen all cars always conduct
image analysis & uploadstoo much cost (battery consumption,
bandwidth, etc)
Possible approacheach car senses at a fixed intervalmay miss PoI
(cherry locations)17
The second technical challenge is load distribution among
cars.When all cars always conduct image analysis and uploads of
videos, the cars will take too much cost.Possible approach is that
each car senses at a fixed interval.However, it may miss PoI.17
k-stage sensing18location where sensingis performed
Narrows sensing interval step-by-step when new PoI is found
Fixed interval(1st stage)PoI is detected!The preceding car
So we propose k-stage sensing which narrows sensing interval
step-by-step when new PoI is found by preceding cars.This is the
example of k-stage sensing.The preceding car travels and sensing is
performed at an initial fixed interval.18
k-stage sensing19
Shorter Interval(2nd stage)
PoI is detected!Sensing is performed in this Radius
PoI detected by preceding carThe following car traveling the
same roadNarrows sensing interval step-by-step when new PoI is
found
location where sensingis performed
After that, when a following car enters the same road.The car
narrows its sensing interval and radius, respectively, because a
PoI is found on the road.Then, this car performs sensing at the
shorter interval while the car is in the circle centered at the PoI
with radius. 19
Evaluation of SakuraSensorInvestigate effectiveness of cherry
intensityCompare the results of manual classification and automatic
classification by cherry intensity
Videos
manual classification(used as ground truth)classificationby
cherry intensityCompute accuracy by comparison 20
We conducted some experiments to evaluate Sakura Sensor.The
first experiment is to investigate the accuracy of cherry
intensity.We compare the result of manual classification and
automatic classification by cherry intensity.20
Videos used for experimentsRecorded videos in 8 different scenes
(routes) using SakuraSensor app for iOS by multiple carsscene
namedatevehicleareaLength (min.)S1Mar. 31V1Aichi Pref.17S2Apr.
5V2Nara Pref.12S3Apr. 10V2Nara Pref.66S4Apr. 10V3Nara
Pref.261S5Apr. 10V4Nara Pref.186S6Apr. 11V1Gifu Pref.72S7Apr.
12V2Osaka Pref.137S8Apr. 18V1Aichi Pref.89
extracted 1-second videos at random starting time from each
scene21
We recorded videos in 8 different scenes using SakuraSensor
application for iOS devices by multiple cars.We extracted 1-second
videos at randomly selected starting time from each scene.
21
1-Second Videos Manual ClassificationClass nameCriteriaC1cherry
ratio (in image) < 5%C25% cherry ratio < 25%C325% cherry
ratio
SceneC1C2C3S1791710S2931017S3372433S416139645S5116760S62614772S788810S8521107Total4994230154
22Classification results with the same decision by two persons
were used
We defined three classes where C1s cherry ratio in image is less
than 5%, C2 between 5 and 25%, C3 more than 25%.Here, only
classification results with the same decision by two persons were
used.22
Videos of each class23
C1 (ratio < 5%)C2 (5% ratio < 25%)C3 (25% ratio)
These are example Videos of class C123
Evaluation Methodology
Dividing videos of each class into halvesTraining setTest
set24Set of 1 second videos
Manual classification by human
First, we divided the set of classified videos in each class to
the training set and the test set.
24
Evaluation Methodology
Training setTest setMedian of cherry intensity: M1 0.00033Median
of cherry intensity: M2 0.00791Median of cherry intensity: M3
0.03326
ViCherry intensity
A video25
Then, from training set, we calculated median of cherry
intensity for each class.Using the median values, 1-second videos
in the test set are automatically classified.25
Classification Accuracy (1-second videos)26
precision recall
0.970.900.740.830.240.65
This figure shows the classification result by Sakura Sensor. We
see that a good classification result is obtained for class C1 and
C3 videos. On the other hand, for class C2 videos result is not so
good.The main reason is that many videos included in class C1 were
classified to class C2.26
Evaluation of k-stage sensing27
Simulation by 600 cars (k=3, 300m150m50m)smaller sensing times
similar PoI discovery rate
We also evaluated the effectiveness of 3-stage sensing
method.Evaluation was done with simulation by 600 cars.These
results show the k-stage sensing achieves good PoI discovery rate
with smaller sensing times.
We just adopted hulistic. Or empilically
27
ConclusionsSakuraSensorParticipatory video sensing system by
carsConsisting of two key techniques
Flowering cherry detection by in-vehicle smartphoneColor
histogram analysis for identifying cherry-blossomsFractal dimension
analysis for removing artificial objects other than flowering
cherryCherry detection accuracy (C3) with 0.7 of Precision and 0.8
of Recall
k-stage sensingDistribute sensing load among carsSimilar PoI
discovery rate with about half sensing times compared with the
fixed interval sensing method28
ConclusionsWe proposed sakura sensor which is a Participatory
video sensing system by cars.As two key techniques, we proposed
flowering cherry detection by in-vehicle smart phone andK-stage
sensing.
this system consisting of two key techniquesFirst is flowering
cherry detection by in-vehicle smartphone.Color histogram analysis
for identifying cherry-blossomsFractal dimension analysis for
removing artificial objects other than flowering cherryCherry
detection accuracy with 0.7 of Precision and 0.8 of RecallSecond is
k-stage sensingthis method distributes sensing load among
cars.Similar PoI discovery rate with about half sensing times
compared with the fixed interval sensing method.28
29Thank you!Demonstration at Hall C No.20
We also demonstrate our system.Please wat
29