Towards “Eyes” for Sensor Network Systems Andreas G. Andreou Electrical and Computer Engineering and Whitaker Biomedical Engineering Institute Johns Hopkins University http://www.ece.jhu.edu/faculty/andreou/AGA/index.htm
Towards “Eyes” for Sensor Network Systems
Andreas G. AndreouElectrical and Computer Engineering andWhitaker Biomedical Engineering InstituteJohns Hopkins Universityhttp://www.ece.jhu.edu/faculty/andreou/AGA/index.htm
2004© Andreas G. Andreou 2
outline
• “Ears” for sensor network systems: a brief detour• Introduction
– Light, photons, noise, bandwidth– Current signal processing and translinear networks
• Systems– Polarization contrast chip– Spatial-temporal processing – Ego-motion compensation chip in a balloon observatory– Network architecture for distributed feature extraction.
• Conclusions
2004© Andreas G. Andreou 3
smart microphone project
Gradient Flow ASIC
Cross-Correlation ASIC
Auto-Correlation Wake-Up ASIC
4 MEMs Microphones
Power Strobe Circuit
4 chamber acoustic horn
http://www.signalsystemscorp.com/acoustic_surv.htm
2004© Andreas G. Andreou 4
and some related papers
P. Julian, A.G. Andreou, P. Mandolesi, D. Goldberg, “A low-power CMOS integrated circuit for bearing estimation,” Proceedings of the 2003 IEEE International Symposium on Circuits and Systems, (ISCAS 2003), Bangkok, Thailand, Vol. 5, pp. 305 -308, May 2003.
P. Julian, A.G. Andreou, L. Riddle, S. Shamma, G. Cauwenberghs, “A comparative study of sound localization algorithms for energy aware sensor network nodes, IEEE Transactions on Circuits and Systems: Part II: Analog and Digital Signal Processing, to appear, 2004.
D. Goldberg, A.G. Andreou, et.al., “A wake-up detector for an acoustic surveillance sensor network: algorithm and VLSI implementation”, submitted to IPSN-04.
M. Stanacevic and G. Cauwenberghs, ``Micro-power mixed-signal acoustic localizer,'' in Proc. European Solid State Circuits Conf. (ESSCIRC 2003), Estoril, Portugal, September 2003.
2004© Andreas G. Andreou 5
what did we learn?
• COTS can take you up to a point.• DSP and FPGA also take you up to a point, custom
analog or digital design is necessary.• Event based, one bit digital processing.• Interfaces are critical! –necessity for system level
design-• Algorithm exploration is necessary with real data and
the actual application environment.• Wireless data communication is expensive; do the
computation locally if you can!• Analog subthreshold CMOS works well if designed
properly!
2004© Andreas G. Andreou 6
CCxxxx: Chipcon Datasheetswww.chipcon.com
UWB: 03267r6P802-15_TG3aMulti-band-OFDM-CFPPresentation
the energy cost of bits –in wires and wireless-
2004© Andreas G. Andreou 7
the not-so-state-of-the-art not-eye
http://www.clairex.com/
CrossBow MTS310CASensorBoard
Clairex CdSe photoconductor~ 2 mW power (light ON)~ 47 uW power (light OFF)~ 10 kHz bandwidth5 Volts power supply (signal)10 bits ADC, 15 KS/s
13 nJ per bit of light data –NOT information-
http://www.xbow.com
2004© Andreas G. Andreou 8
“eyes” for sensor network systems
• Is there something interesting in the environment ?– in a specific class of objects
• Where is it ?• What is it ?
often it is about a few bits in the right place at the right time
Eyes: sensory structures capable of spatial vision, i.e. imaging the environment, no matter how crude the image is
Land and Nilsson, Animal Eyes
2004© Andreas G. Andreou 9
the way natural eyes see
• Continuous sensing
• Polarization sensitivity• Contrast sensitivity
• Local gain control• Spatial filtering
• Temporal filtering• Sampling on demand
2004© Andreas G. Andreou 10
light, photons, photon shot-noise, bandwidth …
2004© Andreas G. Andreou 11
analog, digital and all that …
Continuous-Value Discrete-Time
Discrete-Value Discrete-Time
CVDT
DVDT
CCDSwitched Capacitor
Continuous-Value Continuous-Time
CVCT
Linear and non-linear analog
Discrete-Value Continuous-Time
DVCT
Asynchronous digitalNeuron spikes
Anisochronous Pulse Time ModulationBinary digital
Multivalue digital
2004© Andreas G. Andreou 12
subthreshold CMOS
• Current is exponential function of the terminal voltages Vs, Vb, Vg, Vd• Large dynamic range• High gain (transconductance)• Low saturation voltage Vdsat ~100mV• Lossless channel and source/drain symmetry (diffusive networks)• Zero conductance control node (gate); possibility of floating gate for
long term charge storage• Mobility considerations• Frequency limitations
max 22m t
T
g Vf
C L
µπ π
= 1 100
0.25 1.6
m MHz
m GHz
µµ→
→
2004© Andreas G. Andreou 13
subthreshold MOS and bipolar characteristics
0
0
exp exp exp
exp exp exp
n GB SB DBD DS n
t t t
p GB SB DBD SD p
t t t
V V VI I S I
V V V
V V VI I S I
V V V
κ
κ
⎡ ⎤⎛ ⎞ ⎛ ⎞ ⎛ ⎞− −≡ = −⎢ ⎥⎜ ⎟ ⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠ ⎝ ⎠⎣ ⎦
⎡ ⎤−⎛ ⎞ ⎛ ⎞ ⎛ ⎞≡ = −⎢ ⎥⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⎝ ⎠ ⎝ ⎠ ⎝ ⎠⎣ ⎦
IC = IS expVBE
Vt
⎛
⎝ ⎜
⎞
⎠ ⎟ = IS exp
VB − VE
Vt
⎛
⎝ ⎜
⎞
⎠ ⎟
2004© Andreas G. Andreou 14
symmetric MOS model
221
2
1
2S D
S D
o ox depx eDQ
pSQ
d
kT kTQ Q
q
Q
C C
WI I I
C C q
Q
Lµ
⎡ ⎤⎛ ⎞ ⎛ ⎞≡ − = + − +⎢ ⎥⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎢ ⎥⎝ ⎠⎣ + ⎠⎦+ ⎝
DRIFT
DIFFUSION
( ) ( ), ,SD GB SB GB DBI F V V F V V∝ −
( ) ( ) ( )SD GB SB DBI G V H V H V∝ −⎡ ⎤⎣ ⎦
2004© Andreas G. Andreou 15
non-linear CMOS resistors and translinear grids
( )exph n C n rPQ Q P
v T
S V VI I I
S V
κ κ⎛ ⎞ ⎡ ⎤−= −⎜ ⎟ ⎢ ⎥⎣ ⎦⎝ ⎠
NMOS only “diffusor/conveyor”
Vr − VP = G1 IP
Vr − VQ = G1 IQ
VP − VQ = G2 IPQ
IPQ =G1
G2
⎛
⎝ ⎜
⎞
⎠ ⎟ IQ −IP( )
Linear conductances
A.G. Andreou and K.A. Boahen, “Translinear circuits in subthreshold CMOS,” Journal of Analog Integrated Circuits and Signal Processing, Vol. 9, pp. 141-166, March 1996.
2004© Andreas G. Andreou 16
1D spatial averaging network
I j∗ = Iij − I jk + I j
I j∗ = I j +
Sh
Sv
⎛
⎝ ⎜
⎞
⎠ ⎟ exp κ nvC − κ nv r( ) 2 I j − Ii − Ik( )
d2I
dx 2 ≈ 2I j − Ii − Ik
Ii∗, I j
∗, Ik∗, I∗(x)
Inputs:
Ii , I j , Ik , I(x)
Outputs:
At node Vj:
Normalizing inter-node distances to unity we write the above on the continuum
where
Find the smooth function I(x) that best fits the data I(x) with the minimum energy in its first derivative.
λ The reqularization parameter cost associated with energy in the derivative relative to the squared error of the fit to the data
2
2( ) ( )
d II x I x
dxλ∗ = +
2004© Andreas G. Andreou 17
sensor level processing: what and where
PIXEL
Column Select
Row
Sel
ect
Amplifiers
X Computational Circuits
Y C
ompu
tatio
nal C
ircui
ts
Pixel:transductionamplificationgain controlquantizationnon-uniformity correctionspatial filteringtemporal filtering
Periphery:ego-motionmomentsglobal contrastdata-communication
2004© Andreas G. Andreou 18
seeing in ways that we can’t!
Mantis Shrimp
L.B. Wolff and A.G. Andreou, “Polarization camera sensors,” Image and Vision Computing, Vol. 13, No. 6, pp. 497-510, August 1995.
2004© Andreas G. Andreou 19
doing things in front of the pixel: micropolarizers
Z. Kalayjian and A.G. Andreou, “Integrated imaging linear polarimeter," ISA Transactions, Vol. 38, pp. 203-209, 1999.
A.G. Andreou and Z.K. Kalayjian, “Polarization imaging: principles and integrated polarimeters,” IEEE Sensors Journal, Vol. 2, No. 6, pp. 566-576, December 2002.
2004© Andreas G. Andreou 20
current-mode translinear processing
T R ⊥
| |T R
| |
| |
T R T RP C
T R T R
⊥
⊥
−=
+
2004© Andreas G. Andreou 21
sensor level processing: what and where
PIXEL
Column Select
Row
Sel
ect
Amplifiers
X Computational Circuits
Y C
ompu
tatio
nal C
ircui
ts
Pixel:transductionamplificationgain controlquantizationnon-uniformity correctionspatial filteringtemporal filtering
Periphery:ego-motionmomentsglobal contrastdata-communication
2004© Andreas G. Andreou 22
spatial/temporal filter
Detection Buffering / Amplification
Spatial Filtering
Scanning
Temporal Filtering33 x 30 pixels 0.5 micron linear capacitor triple metal CMOS50 micron cell pitch (2 x 2 mm die)
Moving “t”
2004© Andreas G. Andreou 23
sensor level processing: what and where
PIXEL
Column Select
Row
Sel
ect
Amplifiers
X Computational Circuits
Y C
ompu
tatio
nal C
ircui
ts
Pixel:transductionamplificationgain controlquantizationnon-uniformity correctionspatial filteringtemporal filtering
Periphery:ego-motionmomentsglobal contrastdata-communication
2004© Andreas G. Andreou 24
doing things in the sides
A.G. Andreou, R.C. Meitzler, K. Strohbehn and K.A. Boahen, “Analog VLSI neuromorphicimage acquisition and pre-processing systems,” Neural Networks, Vol. 8, No. 7-8, pp. 1323-1347, 1995.
2004© Andreas G. Andreou 25
Flare Genesis Observatory
• Balloon based observatory• Truly autonomous – low bit rate link -
– A three stage hierarchical system of sun orientation and tracking
– Two “Eyes” for finding the sun and motion stabilization + Kodak MegaplusCCD camera
• Solar power; command and control power budget ~1W
http://sd-www.jhuapl.edu/FlareGenesis/flare.html
2004© Andreas G. Andreou 26
networks of nodes ….constraints ….
100s0.01Latency (ms)
10010Energy per bit (nJ)
10010Distance (m)
1200Data rate (Mbits/s)
COTS (Chipcon 24xx)UWB (Multiband OFDM)Rates
RF link constraints
Image sensor constraints (1280x1024 pixels, 24 bit/pixel, 10000 frames/s)
0.00001 (pixel)0.1 (frame rate)Latency (ms)
2020 x (bits/frame)Energy per bit (pJ)
0.010.01Distance (m)
variable24,000Data rate (Mbits/s)
Anisochronous event basedScannedRates
2004© Andreas G. Andreou 27
analog, digital and all that …
Continuous-Value Discrete-Time
Discrete-Value Discrete-Time
Continuous-Value Continuous-Time
CVDT CVCT
DVDT
CCDSwitched Capacitor Linear and non-linear analog
Discrete-Value Continuous-Time
DVCT
Asynchronous digitalNeuron spikes
Anisochronous Pulse Time ModulationBinary digital
Multivalue digital
2004© Andreas G. Andreou 28
PIXEL
Column Select
Row
Sel
ect
Amplifiers
X Computational Circuits
Y C
ompu
tatio
nal C
ircui
ts
Periphery:ego-motionmomentsglobal contrastdata-communication
Pixel:transductionamplificationgain controlquantizationnon-uniformity correctionspatial filteringtemporal filtering
event based systems
2004© Andreas G. Andreou 29
digital event based imager
2004© Andreas G. Andreou 30
high slew rate gain at low energy costs
2004© Andreas G. Andreou 31
the network is the architecture
LocalProcessor
Encoder
CommunicationProcessor
Decoder
ALL COMPUTATION DONE ON THE ADDRESSES OF THE EVENTS
2004© Andreas G. Andreou 32
distributed network processing
• Information is encoded in a stream of events, the address of each pixel node– Address Event Representation– Asynchronous on demand
• Programmable communication processors transform and route the events
• Local Processors perform spatial/temporal integration and normalization
• Point-to-point and broadcast links provide high speed interconnects
2004© Andreas G. Andreou 33
simulation …
2004© Andreas G. Andreou 34
feature extraction through projective fields
D.H. Goldberg, G.C. Cauwenberghs and A.G. Andreou, “Probabilistic synaptic weighting in a reconfigurable network of VLSI integrate-and-fire neurons,” Neural Networks, Vol. 14, No. 6-7, pp. 781-793, July 2001
Think of computation as part of the MAC layer
2004© Andreas G. Andreou 35
results
Input RectifedLaplacian(Matlab)
PrAER(Matlab)
PrAERchip
2004© Andreas G. Andreou 36
and something about biology: blow-fly photoreceptor
P.A. Abshire and A.G. Andreou, “Capacity and energy cost of information in biological and silicon photoreceptors ," Proceedings of the IEEE, Vol. 89, No. 7, pp. 1052-1064, July 2001.
2004© Andreas G. Andreou 37
some final thoughts
• One size perhaps does not fit all!• With multiple interacting points of view, is worth revisiting
“polarization vision”.• Asynchronous on demand systems may be preferable to random
access or scanned for giving information to the question: “is there anything of interest out there ?”
• CMOS imagers can be cheap and can be designed to specific applications; 0.5 micron CMOS may be a sweet spot for garden variety “eyes”.
• Increased physical complexity; more than just visible; large area sensor devices conformal to non planar surfaces.
• What good are “eyes” without optics or when they can’t move?…. Good questions! “eye” designers have plenty things to do.
2004© Andreas G. Andreou 38
acknowledgments:
• DARPA N0014-00-C-0315 Acoustic Microsensors• NSF-ECS-0010026: Microscale adaptive optical wavefront correction• NSF-EIA-0130812: A comparative study of information processing in
biological and bio-inspired systems: performance criteria, resources tradeoffs and fundamental limits.
• NVSED Smart focal planes
Teresa SerranoKim StrohbehnLarry RiddleGert CauwenberghsDavid GoldbergPedro JulianPablo MandolesiEugenio Culurciello
Philippe PouliquenRich MeitzlerZaven KalayjianPamela AbshireKwabena BoahenMarc CohenBernabe LinaresBarranco
2004© Andreas G. Andreou 39
subthreshold CMOS challenges: noise
2004© Andreas G. Andreou 40
energy costs per pixel
ker
: ( )
:
:
: ker
: ( )
: ( 10 100 )
:
nel
bg
comp
pd scale
P power Watts
pixel activity factor
N number of pixels
N number of pixels in the computational nel
I mean photocurrent A
I computational branch current typically nA
V V phototransduction b
ξ
−
= ( (0.3) 0.3 )
: ( (0.3 0.2) 0.5 )comp scale sat
ranch voltage typically Volts
V V V computational branch voltage typically Volts
=
= + + =
•Bandwidth scales linearly with computational branch current•Power will scale linearly with computational branch voltage
2004© Andreas G. Andreou 41
Transduction and Dynamic Range Compression (I): Temporal
1. Average signal in time and store the state -range- on a quasi floating gate (Vfb)2. Employ negative feedback to position the DC operating point.3. Amplifier-computational branch-: single stage biased in substhreshold (100nA)
Delbruck and Mead 96
8 7 7
( )
(0.1 10 0.3 10 1) 10
bg pd comp compP I V I V N
W per pixel
ξ− − −
= +
× × + × =
2004© Andreas G. Andreou 42
Non-Uniformity Correction Using FGMOS
Cohen & Cauwenberghs 2001
• An MOS mirror with FGMOS transistor (M2) injected using impact ionization (tunneling will work as well).• NO power cost during operation.• Technique can be applied to both current mode and voltage mode pixels• Energy cost only with initial calibration• Calibration takes ~2000 iterations for all pixels on the chip and each pixel takes about 1sec
8 8
7
( )
(10 0.3 10 10) 1
10
adapt in pd out compE P T N I V I V T
Joules per pixel
− −
−
= × = × + ×
= × + × ×=
2004© Andreas G. Andreou 43
Transduction and Dynamic Range Compression (I): Spatial -network based-
( )out comp in bg bg pd comp compI I I I P I V I V Nξ→ → = +
1. Average using a shunting network 2. Employ negative feedback and log-antilog amplifier to do the ratio computation3. Note! kernel size does not matter as we normalize everything to the computational
current and this gets steered from one pixel to the other.4. Compression function not tanh but something that can be synthesized in CM circuits!
V
Vs
= In
In + Isn
Vs = 0.6nA, 1.8nA
n =1.2
IS = 1.5nA, 3nA
Vdd
Vdd
Vnw
Vnw
Vnw
VIu
Vhh
VccVnw
Vpx
Vln
Iout
Iin
H
R
Ih
Boahen and Andreou 92
2004© Andreas G. Andreou 44
Center-ON-OFF surround with local competition and rectification
1. An alternative to resistive grids we can explicitelycompute the Laplacian using simple scaled mirrors and summing the currents.
2. Added local wiring complexity
C• C+0C−0
C+60
C−60
C+120
C−120
I•
I•I•
I•
I•I•
I+0
I−0
I+60
I−60
I+120
I−120
0 0 0 0 0 60 120 0 0 0 060 60 120 120 60 120 60 120C I I I I C C C C C Cα β⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞
⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠
• • •+ − + − + + − −= − + − − + − + + +
I − 120 I − 60 I + 60 I+ 120
C •120
C •60
C + 600
C − 600
C − 1200
C + 1200
C •0
RectificationNormalized-intensity gradient
computation
Local different-orientation
inhibition
VliV ni
V r
N eighbour same-orientation
inhibition
( 4 )comp compP I V Nξ= × × 8 7(4 10 1) 10 W per pixel− −× × =
Cauwenberghs and Waskiewicz 1999