Kriging and spatial design accelerated by orders of magnitude

Kriging and spatial design accelerated by orders of magnitu-de: combining low-rank covariance approximations with FFT-techniques

A. Litvinenko joint work with W. [email protected], 8. April 2013

CC

SCScien

tifiomputing

Problem definition Low-rank FFT methods for Kriging Numerics

Task 1: Simple Kriging

Let m be number of measurement points, n number of estimationpoints.Let s ∈ Rn be the (kriging) vector to be estimate with mean µs = 0and cov. matrix Qss ∈ Rn×n:

s = Qsy Q−1yy y︸︷︷︸ξ

, (1)

where y ∈ Rm vector of measurement values, Qsy ∈ Rn×m crosscov. matrix and Qyy ∈ Rm×m auto cov. matrix.

CC

SCScien

tifiomputing

Kriging, combining low-rank covariance approximations with FFT-techniques Seite 2


Task 2: Estimation of Variance σs ∈ Rn

Let Qss|y be the conditional covariance matrix. Then

σs = diag(Qss|y) = diag(Qss − QsyQ−1

yy Qys)

= diag (Qss) −

m∑i=1

[(QsyQ−1

yy ei) ◦QTys(i)

],

where QTys(i) is the transpose of the i-th row in Qys.

CC

SCScien

tifiomputing



Task 3: Geostatistical optimal design

Goal: to optimize sampling patterns from which the data values in yare to be obtained.The objective function to be minimized is:

φA = n−1 trace[Qss|y

]or

φC = cT Qss|yc , (2)

mostly called the A and C criteria of geostatistical optimal design.

φC = cT (Qss − QsyQ−1yy Qys)c

= σ2z − (cT Qsy)Q−1

yy (Qysc), (3)

with σ2z = cT Qssc.

CC

SCScien

tifiomputing



Toeplitz and circulant matrices

Any Toeplitz covariance matrix Qss ∈ Rn×n (first column denote byq) can be embedded in a larger circulant matrix Q ∈ Rn×n (firstcolumn denote by q).

To embed symmetric level-d block Toeplitz matrices in symmetriclevel-d circulant matrices, embed the Toeplitz blocks Ti in circulantblocks Ci , and then extend the series of the blocks to obtain aperiodic series of blocks.

CC

SCScien

tifiomputing



Sampling and Injection:

Consider the m × n sampling matrix H:

Hi,j =

{1 for xi = xj

0 otherwise, (4)

where xi are the coordinates of the i-th measurement location in y,and xj are the coordinates of the j-th estimation point in s.

sampling: ξm×1 = Hun×1 (5)

injection: un×1 = HTξm×1 (6)

CC

SCScien

tifiomputing



Sampling and Injection:

Qys = HQss (7)

Qsy = QssHT (8)

Qyy = HQssHT + R . (9)

Qsyξ = QssHTξ = Qss

(HTξ

)︸︷︷︸

u

. (10)

CC

SCScien

tifiomputing



Embedding and extraction

Let M be a n × n mapping matrix that transfers the entries of thefinite embedded domain onto the periodic embedding domain. Mhas one single entry of unity per column. Extraction of anembedded Toeplitz matrix Qss from the embedding circulant matrixQ as follows:

Qss = MT QM, (11)

q = Mq

CC

SCScien

tifiomputing



Superposition via FFT

(Qsyξ =)Qssu = MT QMu = MTF[−d](F[d] (q) ◦ F[d] (Mu)

).

(12)

within the Kriging context, this becomes

Qsyξ = QssHTξ = MTF[−d][F[d](q) ◦ F[d]

(MHTξ

)], (13)

where injection and embedding form one joint operation(

MHT)

of

injecting ξ into an n × 1 vector of zeros.

CC

SCScien

tifiomputing



Low-rank FFT methods for Kriging

CC

SCScien

tifiomputing



Tensor properties of FFT

Let

‖U − U(k)‖2 6 ε, where U(k) =

k∑j=1

d⊗i=1

uji , (14)

with factors uji ∈ Rni . Then

U := F[d](U) and U(:) =

(d⊗

i=1

Fi

)· U(:). (15)

CC

SCScien

tifiomputing



Tensor properties of FFT

Lemma

Let u = U(:) =∑ku

j=1

⊗di=1 uji , where u ∈ Rn, uji ∈ Rni and

n =∏d

i=1 ni . Then the d-dimensional Fourier transformationu = F[d](u) is given by

u = F[d] (u) =

(d⊗

i=1

Fi

)ku∑

j=1

(d⊗

i=1

uji

)=

ku∑j=1

d⊗i=1

(Fi(uji))

(16)

CC

SCScien

tifiomputing



FFT, Hadamard and Kronecker products

LemmaIf

u ◦ q =

ku∑j=1

d⊗i=1

uji

◦ kq∑

`=1

d⊗i=1

qì

=

ku∑j=1

kq∑`=1

d⊗i=1

(uji ◦ qì

).

Then

F[d](u ◦ q) =ku∑`=1

kq∑j=1

d⊗i=1

(Fi(uji ◦ qì

)).

CC

SCScien

tifiomputing



Accuracy

LemmaIf

‖q − q(kq)‖F 6 εq,‖q − q(kq)‖F‖q‖F

6 εrel,q.

Then

‖Qss − Q(kq)ss ‖F 6

√nεq,

‖Qss − Q(kq)ss ‖F

‖Qss‖F6 εrel,q .

CC

SCScien

tifiomputing



d-dimensional embedding/extraction

M[d] =

d⊗i=1

Mi .

u = M[d]u =

(d⊗

i=1

Mi

)·

ku∑j=1

d⊗i=1

uji =

ku∑j=1

d⊗i=1

Miuji =:

ku∑j=1

d⊗i=1

uji ,

CC

SCScien

tifiomputing



Lemma: Low-rank kriging estimate Qssu

Let u = M[d]u =∑ku

j=1

⊗di=1 uji with uji = Miuji ∈ Rni and u ∈ Rn.

Let also M[d]T QM[d] = Qss where Q ∈ Rn×n is level-d circulantcov.matrix with q =

∑kq`=1

⊗di=1 qì , q ∈ Rn, qì ∈ Rni and

qì = M[d]T qì . Then

Qssu ≈ Q(kq)ss u(ku) =

kq∑`=1

ku∑j=1

d⊗i=1

MTi F

−1i

[(Fi qì) ◦ (Fi uji)

]. (17)

with accuracy

‖Qssu − Q(kq)ss u(ku)‖F 6

√nεq‖u‖+ ‖Qss‖ · εu. (18)

The total costs is O(kukqd L∗ log L∗) instead of O(n log n), withL∗ = maxi=1...d ni and n =

∏di=1 n.

CC

SCScien

tifiomputing



Low-rank FFT Kriging estimator

Let u = HTξ, and j-th row hj ∈ R1×n of H: hj =⊗d

k=1 hjk , wherehjk ∈ Rnk×1.Then, the injection of ξ = (ξ1, .., ξm) can be written as

u = HTξ =

m∑j=1

(ξj

d⊗k=1

hjk

), and (19)

MHTξ =

(d⊗

i=1

Mi

)m∑

j=1

ξj

d⊗i=1

hji

=

m∑j=1

ξj

d⊗i=1

Mihji =:

m∑j=1

ξj

d⊗i=1

hji . (20)

CC

SCScien

tifiomputing



Kriging estimator s in low-rank

Remind that

s = Qsyξ = QssHTξ = MT QMHTξ. (21)

Use Eq. 20, obtain

Lemma

The Kriging estimator can be written in a low-rank tensor format

s = Qsyξ =

m∑j=1

ξj

kq∑`=1

d⊗i=1

(MT

i F−1i

[(Fi qì) ◦ (Fi hji)

]). (22)

CC

SCScien

tifiomputing



Low-rank FFT estimation variance

Let

σs = σ2s1n −

m∑p=1

(Qsy Q−1yy ep︸︷︷︸ξp

) ◦QTys(p)

, (23)

Then

σs = σ2s1n−

m∑p=1

m∑j=1

ξjp

kq∑`=1

d⊗i=1

(MT

i F−1i

[(Fi qì) ◦

(Fi hji

)]) ◦QTys(p)

,

where ξpj is the j-th element of ξp = Q−1yy ep, i.e., the (j, p)-th

element of Q−1yy ,

CC

SCScien

tifiomputing



where

Qsy(p) = Qsshp

=

kq∑r=1

d⊗i=1

(MT

i F−1i

[(Fi qri) ◦ (Fi hpi)

]), (24)

where hp ∈ Rn is the p-th row of H, hpi = Mihpi ∈ Rni×1.

CC

SCScien

tifiomputing



σs = σ2s1n

−

m∑p=1

m∑j=1

ξjp

kq∑r=1

kq∑`=1

d⊗i=1

(MT

i F−1i

[(Fi qì) ◦

(Fi hji

)])

. . . ◦(

MTi F

−1i


]) . (25)

Here one can exploit the fact that (Fi hpi) is the analytically knownp-th column of the one-dimensional discrete Fourier matrix.

CC

SCScien

tifiomputing



A-criterion of optimal design

Lemma

φA = σ2s

−

m∑p=1

m∑j=1

ξjp

kq∑r=1

kq∑`=1

d∏i=1

(1T

niMT

i F−1i

[(Fi qì) ◦

(Fi hji

)])

. . . ◦(1T

niMT

i F−1i


]) , (26)

in which the Kronecker product is applied to scalars (due to thesummation) and simplifies to a simple product, and where 1ni ∈ Rni is avector of all ones.

CC

SCScien

tifiomputing



Numerics

Domain: 20m × 20m × 20m, 25, 000× 25, 000× 25, 000 dofs. Wefeature 4,000 measurement data values, randomly distributedwithin the volume, with increasing data density towards the lowerleft back corner of the domain. The covariance model is anisotropicGaussian with unit variance and with 32 correlation lengths fittinginto the domain in the horizontal directions, and 64 correlationlengths fitting into the vertical direction.

CC

SCScien

tifiomputing



Numerics

The top left figure shows the entire domain at a sampling rate of 1:64 perdirection, and then a series of zooms into the respective lower left backcorner with zoom factors (sampling rates) of 4 (1:16), 16 (1:4), 64 (1:1) forthe top right, bottom left and bottom right plots, respectively. Color scale:showing the 95% confidence interval [µ− 2σ,µ+ 2σ].

CC

SCScien

tifiomputing



Numerics on computer with 16GB RAM:

2D Kriging with 270 million estimation points and 100measurement values (0.25 sec.),

to compute the estimation variance (< 1 sec.),

to evaluate the spatial average of the estimation variance (theA-criterion of geostat. optimal design) for 2 · 1012 estim. points(30 sec.),

to compute the C-criterion of geostat. optimal design for 2 · 1015

estim. points (30 sec.),

solve 3D Kriging problem with 15 · 1012 estim. points and 4000measurement data values (20 sec.)

CC

SCScien

tifiomputing



Memory requirements

Abbildung: Memory requirements of the four different methods dependingon the number of lattice points in a rectangular domain

CC

SCScien

tifiomputing



CPU time

Abbildung: CPU time of the four different methods depending on thenumber of lattice points in a rectangular domain

CC

SCScien

tifiomputing



CPU time

Method max. nodes (dec., log2) CPU time [s]Kriging

Standard 2.0972e+6 21 14.6942FFT/Kronecker 2.6843e+8 28 0.2551

w/o final 1.0995e+12 40 1.9176FFT-based 6.7108e+7 26 2.7603

Estimation varianceStandard 2.0972e+6 21 7.6671FFT/Kronecker 1.3422e+8 27 22.5566FFT-based 3.3554e+7 25 223.2980

A-criterionStandard 2.0972e+6 21 7.6881FFT/Kronecker 1.3744e+11 37 52.0253FFT-based 3.3554e+7 25 224.5097

C-criterionStandard 2.0972e+6 21 0.7300FFT/Kronecker 2.2518e+15 51 3.2447FFT-based 6.7108e+7 26 2.6745

CC

SCScien

tifiomputing



Literature

W. Nowak, A. Litvinenko, Kriging accelerated by orders of magnitude:combining low-rank covariance approximations with FFT-techniques,accepted to Mathematical Geosciences, 2013

CC

SCScien

tifiomputing


Kriging and spatial design accelerated by orders of magnitude

Education