Page 1
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Fast Atomic Force Microscopy Imaging
using Self-Intersecting Scans and Inpainting
Rodrigo Farnham, Nen Huynh, Travis MeyerAdvisors: Andrea Bertozzi, Jen-Mei Chang, Alex Chen
2011 UCLA CAM REU
Page 2
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Outline
1 Background
2 Noise RemovalDrift RemovalTilt RemovalStreak Detection
3 Scan Types
4 InpaintingPenalized Dictionary Inpainting
5 Results
Page 3
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Atomic Force Microscopy (AFM)
1
1Source: Wikipedia
Page 4
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Raster Scan
Raster Scanned Image
(a) The AFM scans top-downwith horizontal scan paths.Notice the different mean
intensities of each line.
Line-Flattening
(b) Each horizontal scan isline-fitted and the fit is
subtracted from the scan line.
Page 5
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Problems with Raster Scan
Scanning is slow due to resonant frequencies.Forces the mean of each line to be the same.Distorts image when a dark/light object is on the sides.
(a) (b)
Figure: (a) A simulated image. (b) Line-flattened. Notice the lossof the horizontal bar and the distortion in the foreground andbackground around objects.
Page 6
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Cycloid Scan
Step 1: Scan[1]
Cycloid Scan Path
Step 2: Collect Data
Data from Cycloid Scan
The Cycloid scan was studied as a feasible way to collect datausing a ”smooth” path.
Page 7
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Model of Distortion
S(t) = I(x(t),y(t)) + T(x(t),y(t)) + D(t) + χ(t) + η(t)
S: Signal from AFM.
I: Height of sample - function of bounded variation.
T: Tilt - approximately a plane in x and y.
D: Drift - continuous function with small secondderivative.
χ: Streaks - simple function of finite discontinuity.
η: Gaussian noise.
Page 8
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Poly-Flattening
Subtract polynomial fit from scan arcs in analogue withline-flattening.Issues:
Assumes mean is constant across flattening interval.
Does not enforce continuity between arcs.
Is multi-valued in intersections.
Distorts image when there is a high contrast.
Page 9
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Difference-Flattening
We can exploit self-intersections on the scan path.If (x(a), y(a)) = (x(b), y(b)) then∆i = S(bi)− S(ai) = D(bi) − D(ai) (ignoring χ and η).Let ` be the number of intersections.
Page 10
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Poly-Difference-Flattening
Approximate D(t) by an n-degree polynomialD(t) =
∑nk=1 αkφk(t) where φk(t) = tk .
For simplicity, represent α = [α1 α2 · · ·αn]T .
Using least squares, we minimizeE =
∑`i=1(D(bi)− D(ai)−∆i)
2.
We simplify the process by introducing a general basis forthe related function
d(ai , bi) = D(ai)− D(bi) =n∑
k=1
αk(φk(t1)− φk(t2))
=n∑
k=1
αkΦk = [Φ1(ai , bi) Φ2(ai , bi) . . . Φn(ai , bi)]α.
Page 11
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Poly-Difference-Flattening
Since E = ‖∆− Φα‖2,∂E
∂α= 2ΦT Φα − 2ΦT ∆ = 0 gives
ΦT Φα = ΦT ∆
where given the i th intersection, ai is the first visit to anintersection, bi is the second visit and Φ = [(bi)
j − (ai)j ]ij .
Page 12
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Poly-Difference-Flattening
Benefits:
Takes into account the intersections.
Enforces continuity between arcs.
Preserves shades, i.e. does not have line-flatteningdistortions.
Produces better results than Poly-Flattening.
Issues:
The polynomial requires a high degree for fitting severethermal drift; this causes numerical instability.
“Blows up“ at the ends.
Page 13
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Trig-Difference-Flattening
Same idea as poly differences, but using e ikx0<|k|≤n for our
basis and Φ = [e√−1jbi − e
√−1jbi ]ij .
Benefits:
Takes into account the intersections.
Fits the boundedness and oscillation of Tilt and Driftbetter than Poly-Flattening.
Issues:
Slower than the previous drift removal methods.
The trig functions require a high degree for fitting severethermal drift; this causes numerical instability.
Page 14
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Smoothing Spline over Differences
For splines, E = LSQ + λPenalty where LSQ is the leastsquares term and Penalty is a penalty term for ”waviness”of the spline.
Arbitrary splines can be constructed with a basis ofB-splines:
Ni ,0(t) =
1 ti ≤ t < ti+1
0 otherwise(Base case)
Ni ,j(t) =t − ti
ti+j − tiNi ,j−1(t) +
ti+j+1 − t
ti+j+1 − ti+1Ni+1,j−1(t)
where j is the degree of the spline polynomial and1 ≤ i ≤ m − 1− j with m being the number of knots.
Page 15
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Smoothing Spline over Differences
Figure: An example of a spline basis
Page 16
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Smoothing Spline over Differences
LSQ = ‖Φα−∆‖2 and
Penalty =
∫ T
0
‖Φ′′α‖2 dt
= αT
(∫ T
0
Φ′′T Φ′′ dt
)α
= αTMα where T is the ending time.
The functional is then
E = ‖Φα−∆‖2 + λαTMα
0 =∂E
∂α= 2ΦT Φα + 2ΦT ∆− 2λMα
ΦT ∆ = λMα− ΦT Φα
Page 17
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Smoothing Spline over Differences
Benefits:
Takes into account the intersections.
Fits the boundedness and oscillation of tilt and driftbetter than Poly-Flattening.
Does not require a high degree to work well.
Issue:
Requires a recursively defined basis.
Page 18
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Tilt Removal
Image with Tilt Tilt Removed
After drift is removed, a plane is fitted to the image andsubtracted off.
Page 19
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Streak Detection
A threshold is used to mark the locations of the smoothedsignal with high derivative. This then marks the streaks.
(a) Signal (b) Smoothed
(c) Derivative of Smoothed (d) Magnitude of Derivative
Page 20
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Streak Detection
Issues:
A threshold must be found.
Conservative thresholds have too many false positives.
Cannot be used alone to remove streaks.
Page 21
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Double Archimedean Spiral Scan
The problem of finding a curve φ that covers the most spacecan be written as an energy minimization:
E (φ) =∫
Ωmin
td(z , φ(t))dz
with φ having fixed length L > 0 and d(·, ·) being L2 distance.
Page 22
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Double Archimedean Spiral Scan
The expression mint
d(z , φ(t)) can be best visualized as a heat
map. Running simulated annealing on Matlab, we get:
Page 23
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Double Archimedean Spiral Scan
The image suggests a path similar to an Archimedean Spiral.To be able to remove drift, intersections are needed, so aDouble Archimedean Spiral Scan is used:
Page 24
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Cycloid vs. Double Archimedean Spiral
Benefit of Double Archimedean Spiral over Cycloid:
The Archimedean Spiral fits the Energy functionalE (φ) =
∫Ω
mint
d(z , φ(t))dz better than Cycloid. Hence,
it covers more space than Cycloid for the same length ofthe curve.
Benefit of Cycloid over Double Archimedean Spiral:
The Cycloid scan has short-term and long-termself-intersections in regular intervals which facilitatesgood drift removal.
Page 25
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Cycloid vs. Double Archimedean Spiral
Each connecting line represents a temporal connectionbetween intersections. A connection which starts from theright is when the scan hits the intersection and the leftindicates when it is revisited.
(a) Cycloid Path (b) Cycloid Bundle
Page 26
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Cycloid vs. Double Archimedean Spiral
(c) Spiral Path (d) Spiral Bundle
Page 27
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Penalized Dictionary Inpainting
Given image I, associate with it a neighborhood matrixB = [ ~B1 · · · ~B`]
T , where each ~Bi is a column vectorrepresentation of a neighborhood box.
Page 28
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Penalized Dictionary Inpainting
Assume there is a dictionary which succinctly describes theseneighborhoods. Then we can write B = PV where P are the coefficients,V is the dictionary, and B approximates B. Then we can minimize theleast squares error to simultaneously solve for P and V :
E = ‖B − PV ‖2
∂E
∂P= −2(B − PV )V T
∂E
∂V= −2PT (B − PV )
Gradient descent gives,
Pk+1 = Pk + τ(B − PkVk)V Tk
Vk+1 = Vk + τPTk (B − PkVk)
Notice that in the case of inpainting the difference B − PV is taken only
on known data.
Page 29
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Penalized Dictionary Inpainting
However, this formulation does not enforce a geometric constraint. Toregularize it, we include a penalty term. We require adjacentneighborhoods’ dictionary basis expression to be similar, i.e. if Bi = PiVis adjacent to Bj = PjV then ‖Pi − Pj‖2 is small.This corresponds to a high dimensional derivative and is analogous toHeat equation inpainting.Combined, these arrive at the evolution:
Pk+1 = Pk + τ(B − PkVk)V Tk − λMPk
Vk+1 = Vk + τPTk (B − PkVk)
Page 30
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Penalized Dictionary Inpainting
Figure: Banded Matrix M
Page 31
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
GUI
Figure: Afmshop
Page 32
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Results: Cycloid ScanSimulated Scan
TV Inpainting without DriftRemoval
TV Inpainting withPoly-Flattening
Page 33
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Results: Cycloid ScanSimulated Scan
TV Inpainting without DriftRemoval
TV Inpainting with Spline DriftRemoval
Page 34
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Results: Double Archimedean Spiral ScanCourtesy of Lawrence Berkeley Laboratory
TV Inpainting without DriftRemoval
TV Inpainting with Spline DriftRemoval
Page 35
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Results: Penalized Dictionary Inpainting
Figure: An example of Penalized Dictionary Inpainting
Page 36
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Results: Penalized Dictionary Inpainting
TV Inpainting Penalized Dictionary Inpainting
Page 37
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Future Work
Automate/Improve Streak Detection.
Investigate other scan paths.
Incorporate different regularization into penalizeddictionary inpainting.
Extend our inpainting method to include multi-resolutionconsiderations.
Finalize a GUI that can be used on actual AFM data.
Page 38
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Acknowledgements
Paul Ashby
We thank Paul Ashby, Dominik Ziegler,Andreas Frank (Lawrence BerkeleyNational Laboratory) for the samplepictures and assistance in understandinghardware and software related to theAFM. Also, we thank Christoph Brunefor his generous help on inpainting.
Page 39
Background Noise Removal Scan Types Inpainting Results Future Work Acknowledgements References
Reference
Y.K Yong, Moheimani.S.O.R., and I.R. Petersen.High-speed cycloid-scan atomic force microscopy.Nanotechnology, 2010.