Fast Fourier Transformjordicf/Teaching/AP2/pdf/14_Fast... · 2020. 2. 7. · •Using the Master Theorem we conclude: RuntimeFFT𝑛=O(𝑛log𝑛) •Gilbert Strang (MIT, 1994):

Fast Fourier Transform

Jordi Cortadella and Jordi Petit

Department of Computer Science

Sum of sinusoids

Fast Fourier Transform © Dept. CS, UPC 2

FFT application in Signal Processing


Converting a signal: time domain ↔ frequency domain

Fourier series

• Periodic function 𝑓(𝑡) of period 1:

𝑓 𝑡 =𝑎02+

𝑛=1

∞

𝑎𝑛 cos(2𝜋𝑛𝑡) +

𝑘=1

∞

𝑏𝑛 sin(2𝜋𝑛𝑡)

• Fourier coefficients:

𝑎𝑛 = 2න0

𝑇

𝑓(𝑡) cos(2𝜋𝑛𝑡) 𝑑𝑡, 𝑏𝑛 = 2න0

𝑇

𝑓(𝑥) sin 2𝜋𝑛𝑡 𝑑𝑡

• Fourier series is fundamental for signal analysis (to move from time domain to frequency domain, and vice versa)


Why Fourier Transform?


Discrete Fourier Transform (DFT)

A method for converting a sequence of 𝑁complex numbers 𝑥0, 𝑥1, … , 𝑥𝑁−1 to a new sequence of 𝑁 complex numbers:

𝑋𝑘 =

𝑛=0

𝑁−1

𝑥𝑛𝑒−2𝜋𝑖𝑘𝑛/𝑁

such that

𝑥𝑛 =1

𝑁

𝑘=0

𝑁−1

𝑋𝑘𝑒2𝜋𝑖𝑘𝑛/𝑁


Polynomials: coefficient representation


Polynomials: point-value representation

• Fundamental Theorem (Gauss): A degree 𝑛 polynomial with complex coefficients has exactly 𝑛 complex roots.

• Corollary: A degree 𝑛 − 1 polymonial 𝐴(𝑥) is uniquely identified by its evaluation at 𝑛 distinct values of 𝑥.


Polynomials: point-value representation


Lagrange formula: example


𝑃 𝑥 = { −1,−1 , 0, −2 , 2,2 }

𝑃 𝑥 = −𝑥 𝑥 − 2

3− 2

𝑥 + 1 𝑥 − 2

−2+ 2

𝑥 + 1 𝑥

6

𝑃 𝑥 = 𝑥2 − 2

Conversion between both representations


representation addition multiplication evaluation

coefficient O(𝑛) O(𝑛2) O(𝑛)

point-value O(𝑛) O(𝑛) O(𝑛2)

𝑎0, 𝑎1, ⋯ , 𝑎𝑛−1 𝑥0, 𝑦0 , ⋯ , 𝑥𝑛−1, 𝑦𝑛−1

evaluation

interpolationCoefficient representation Point-value representation

Could we have an efficient algorithm to move from coefficientto point-value representation and vice versa?

From coefficients to point-values


𝑝 𝑥 = 𝑎0 + 𝑥 𝑎1 + 𝑥 𝑎2 + 𝑥 𝑎3 +⋯+ 𝑥 𝑎𝑛−1 + 𝑥𝑎𝑛 ⋯

Horner’s rule:

Evaluation by divide-and-conquer• Credits: based on the intuitive explanation by Dasgupta,

Papadimitriou and Vazinari, Algorithms, McGraw-Hill, 2008.

• We want to evaluate 𝐴(𝑥) at 𝑛 different points. Let us choose them to be positive-negative pairs: ±𝑥0, ±𝑥1, … , ±𝑥 Τ𝑛 2−1

• The computations for 𝐴(𝑥𝑖) and 𝐴(−𝑥𝑖) overlap a lot.

• Split the polynomial into odd and even powers

3 + 4𝑥 + 6𝑥2 + 2𝑥3 + 𝑥4 + 10𝑥5 = 3 + 6𝑥2 + 𝑥4 + 𝑥 4 + 2𝑥2 + 10𝑥4

• The terms in parenthesis are polynomials in 𝑥2:

𝐴 𝑥 = 𝐴𝑒 𝑥2 + 𝑥𝐴𝑜(𝑥2)


Evaluation by divide-and-conquer

• The calculations needed for 𝐴 𝑥𝑖 can be reused for computing 𝐴 −𝑥𝑖 .

𝐴 𝑥𝑖 = 𝐴𝑒 𝑥𝑖2 + 𝑥𝑖𝐴𝑜 𝑥𝑖

2

𝐴 −𝑥𝑖 = 𝐴𝑒 𝑥𝑖2 − 𝑥𝑖𝐴𝑜 𝑥𝑖

2

• Evaluating 𝐴 𝑥 at 𝑛 paired points

±𝑥0, ±𝑥1, … , ±𝑥 Τ𝑛 2−1

reduces to evaluating 𝐴𝑒 𝑥 and 𝐴𝑜(𝑥) at just𝑛/2 points: 𝑥0

2, ⋯ , 𝑥𝑛/2−12




+𝑥0 −𝑥0 +𝑥1 −𝑥1 +𝑥𝑛/2−1 −𝑥𝑛/2−1⋯Evaluate: 𝐴 𝑥degree ≤ 𝑛 − 1

Evaluate:𝐴𝑒 𝑥 and 𝐴𝑜(𝑥)degree ≤ 𝑛/2 − 1

𝑥02 𝑥1

2 𝑥𝑛/2−12⋯

𝑇 𝑛 = 2 ⋅ 𝑇 𝑛/2 + O 𝑛 = O(𝑛 log 𝑛)

If we could recurse, we would get a running time:

But can we recurse?



+𝑥0 −𝑥0 +𝑥1 −𝑥1 +𝑥𝑛/2−1 −𝑥𝑛/2−1⋯Evaluate: 𝐴 𝑥degree ≤ 𝑛 − 1

Evaluate:𝐴𝑒 𝑥 and 𝐴𝑜(𝑥)degree ≤ 𝑛/2 − 1

𝑥02 𝑥1

2 𝑥𝑛/2−12⋯

The problem: ? We need 𝑥02 and 𝑥1

2 to be a plus-minus pair.But a square cannot be negative !

Selection of the evaluation points


+𝑥0 −𝑥0 +𝑥1 −𝑥1 +𝑥2 −𝑥2 +𝑥3 −𝑥3

𝑥02 𝑥1

2 𝑥22 𝑥3

2

𝑥04 𝑥2

4

𝑥08

+ 𝑖 − 𝑖

+1

+1 −1

+1 −1

+1 −1

−𝑖+𝑖

+𝑖 −𝑖 + −𝑖 − −𝑖

Note:

𝑖 = ±1

21 + 𝑖

−𝑖 = ±1

2(1 − 𝑖)

Selection of the evaluation points


+ 𝑖

− 𝑖

+1−1

+𝑖

−𝑖

+ −𝑖

− −𝑖

Complex numbers: review


𝑟

𝜃

𝑎

𝑏

Imaginary

Real

𝑧 = 𝑎 + 𝑏𝑖 𝑧 = 𝑟(cos 𝜃 + 𝑖 sin 𝜃) = 𝑟𝑒𝑖𝜃

Polar coordinates: (𝑟, 𝜃)

Length: 𝑟 = 𝑎2 + 𝑏2

Angle 𝜃 ∈ 0,2𝜋 : cos 𝜃 =𝑎

𝑟, sin 𝜃 =

𝑏

𝑟

𝜃 can always be reduced modulo 2𝜋

Some examples:

Number −1 𝑖 5 + 5𝑖

Polar coords (1, 𝜋) (1, 𝜋/2) (5 2, Τ𝜋 4)

Complex numbers: multiplication


(𝑟1, 𝜃1)

(𝑟2, 𝜃2)

(𝑟1𝑟2, 𝜃1 + 𝜃2)

𝑟1, 𝜃1 × 𝑟2, 𝜃2 = (𝑟1𝑟2, 𝜃1 + 𝜃2)

For any 𝑧 = 𝑟, 𝜃 :

−𝑧 = (𝑟, 𝜃 + 𝜋), since −1 = 1, 𝜋

If 𝑧 is on the unit circle, then 𝑧𝑛 = (1, 𝑛𝜃)

Complex numbers: the 𝑛th roots of unity


Angle 2𝜋/𝑛

4𝜋/𝑛

2𝜋

𝑛+ 𝜋

Solutions to the equation 𝑧𝑛 = 1(𝑛 = 16)

Solutions are 𝑧 = 1, 𝜃 ,for 𝜃 a multiple of 2𝜋/𝑛

All roots are plus-minus paired:

− 1, 𝜃 = (1, 𝜃 + 𝜋)

Roots of unity for 𝑛 = 8


+ 𝑖

− 𝑖

+1−1

+𝑖

−𝑖

+ −𝑖

− −𝑖

Divide-and-conquer step


Evaluate 𝐴(𝑥) at 𝑛th roots of unityEvaluate 𝐴𝑒(𝑥) and 𝐴𝑜 𝑥at Τ𝑛 2 nd roots of unity

Divide-and-conquer steps


FFT algorithm

function FFT(𝑨,𝝎)Inputs: 𝑨 = (𝒂𝟎, 𝒂𝟏, … , 𝒂𝒏−𝟏), for 𝒏 a power of 2

𝝎: A primitive 𝒏th root of unity

Output: 𝑨 𝟏 , 𝑨 𝝎 , 𝑨 𝝎𝟐 , … , 𝑨 𝝎𝒏−𝟏

if 𝝎=1: return 𝐴

𝑨𝒆 𝝎𝟎 , 𝑨𝒆 𝝎𝟐 , … , 𝑨𝒆(𝝎𝒏−𝟐) = FFT(𝑨𝒆, 𝝎

𝟐)

𝑨𝒐 𝝎𝟎 , 𝑨𝒐 𝝎𝟐 , … , 𝑨𝒐(𝝎𝒏−𝟐) = FFT(𝑨𝒐, 𝝎

𝟐)

for 𝒌 = 𝟎 to 𝒏 − 𝟏:𝑨 𝝎𝒌 = 𝑨𝒆 𝝎𝟐𝒌 +𝝎𝒌𝑨𝒐(𝝎

𝟐𝒌)

return 𝑨 𝟏 , 𝑨 𝝎 , 𝑨 𝝎𝟐 , … , 𝑨(𝝎𝒏−𝟏)


FFT algorithm

for 𝒌 = 𝟎 to 𝒏 − 𝟏: 𝑨 𝝎𝒌 = 𝑨𝒆 𝝎𝟐𝒌 +𝝎𝒌𝑨𝒐(𝝎𝟐𝒌)


Example (𝒏 = 𝟖):

𝐴 𝜔0 = 𝐴𝑒 𝜔0 + 𝜔0𝐴𝑜 𝜔0





𝐴 𝜔5 = 𝐴𝑒 𝜔10 + 𝜔5𝐴𝑜 𝜔10

𝐴 𝜔6 = 𝐴𝑒 𝜔12 + 𝜔6𝐴𝑜 𝜔12

𝐴 𝜔7 = 𝐴𝑒 𝜔14 + 𝜔7𝐴𝑜 𝜔14

𝝎𝟎

𝝎𝟏

𝝎𝟐

𝝎𝟑

𝝎𝟒

𝝎𝟓

𝝎𝟔

𝝎𝟕

𝝎𝟒 = 𝝎𝟏𝟐 𝝎𝟐 = −𝝎𝟔

FFT algorithm









𝐴 𝜔5 = 𝐴𝑒 𝜔10 + 𝜔5𝐴𝑜 𝜔10

𝐴 𝜔6 = 𝐴𝑒 𝜔12 + 𝜔6𝐴𝑜 𝜔12

𝐴 𝜔7 = 𝐴𝑒 𝜔14 + 𝜔7𝐴𝑜 𝜔14

𝝎𝟒 = 𝝎𝟏𝟐 𝝎𝟐 = −𝝎𝟔





𝐴 𝜔4 = 𝐴𝑒 𝜔0 − 𝜔0𝐴𝑜 𝜔0

𝐴 𝜔5 = 𝐴𝑒 𝜔2 −𝜔1𝐴𝑜 𝜔2



FFT algorithm












FFT shuffling









FFT algorithm












for 𝒌 = 𝟎 to 𝒏/𝟐 − 𝟏:

𝑨 𝝎𝒌 = 𝑨𝒆 𝝎𝟐𝒌 +𝝎𝒌𝑨𝒐 𝝎𝟐𝒌

𝑨 𝝎𝒌+𝒏𝟐 = 𝑨𝒆 𝝎𝟐𝒌 −𝝎𝒌𝑨𝒐 𝝎𝟐𝒌

FFT algorithm

function FFT(𝒂,𝝎)Inputs: 𝒂 = (𝒂𝟎, 𝒂𝟏, … , 𝒂𝒏−𝟏), for 𝒏 a power of 2

𝝎: A primitive 𝒏th root of unity

Output: 𝒂 𝟏 , 𝒂 𝝎 , 𝒂 𝝎𝟐 , … , 𝒂 𝝎𝒏−𝟏

if 𝝎=1: return a // 𝒏 = 𝟏, 𝒂 has only one element

𝒔𝟎, 𝒔𝟏, … , 𝒔 Τ𝒏 𝟐−𝟏 = FFT( 𝒂𝟎, 𝒂𝟐, … , 𝒂𝒏−𝟐 , 𝝎𝟐)

𝒔𝟎′ , 𝒔𝟏

′ , … , 𝒔 Τ𝒏 𝟐−𝟏′ = FFT 𝒂𝟏, 𝒂𝟑, … , 𝒂𝒏−𝟏 , 𝝎𝟐

for 𝒌 = 𝟎 to Τ𝒏 𝟐 − 𝟏: // FFT shuffling𝒓𝒌 = 𝒔𝒌 +𝝎𝒌𝒔𝒌

′

𝒓𝒌+ Τ𝒏 𝟐 = 𝒔𝒌 −𝝎𝒌𝒔𝒌′

return (𝒓𝟎, 𝒓𝟏, … , 𝒓𝒏−𝟏)


FFT: asymptotic complexity• The runtime of the FFT can be expressed as:

𝑇 𝑛 = 2 ⋅ 𝑇 Τ𝑛 2 + O 𝑛

• Using the Master Theorem we conclude:

Runtime FFT 𝑛 = O(𝑛 log 𝑛)

• Gilbert Strang (MIT, 1994):

“the most important numericalalgorithm of our lifetime”.

• Reference: Cooley, James W., and John W. Tukey, 1965,“An algorithm for the machine calculation of complex Fourier series,” Math. Comput. 19: 297-301.


Unfolding the FFT


𝐹𝐹𝑇 Τ𝑛 2

𝑎0𝑎2

𝑎𝑛−2

⋮

𝐹𝐹𝑇 Τ𝑛 2

𝑎1𝑎3

𝑎𝑛−1

⋮

𝑟𝑘

𝑟𝑘+ Τ𝑛 2+

+

𝜔𝑘

𝜔𝑘+ Τ𝑛 2


Unfolding the FFT (butterfly diagram)𝑎0

𝑎4

𝑎2

𝑎6

𝑎1

𝑎5

𝑎3

𝑎7

𝐴(𝜔0)

𝐴(𝜔1)

𝐴(𝜔2)

𝐴(𝜔3)

𝐴(𝜔4)

𝐴(𝜔5)

𝐴(𝜔6)

𝐴(𝜔7)

4

4

4

4

4

4

2

2

6

6

4

2

3

7

6

5

1

000

001

010

011

100

101

110

111

000

100

010

110

001

101

011

111

Why is it called a butterfly diagram?







𝑎0, 𝑎1, ⋯ , 𝑎𝑛−1 𝑥0, 𝑦0 , ⋯ , 𝑥𝑛−1, 𝑦𝑛−1

evaluation


values = FFT( coefficients , 𝜔)

O(𝒏 𝐥𝐨𝐠𝒏)

From point-values to coefficients


From point-values to coefficients







𝑎0, 𝑎1, ⋯ , 𝑎𝑛−1 𝑥0, 𝑦0 , ⋯ , 𝑥𝑛−1, 𝑦𝑛−1

evaluation


values = FFT( coefficients , 𝜔)

coefficients =1

𝑛FFT( values , 𝜔−1)

O(𝒏 𝐥𝐨𝐠𝒏)

Polynomial multiplication

Input: Coefficients of two polynomials 𝐴(𝑥) and 𝐵(𝑥),of degree 𝑑𝐴 and 𝑑𝐵, respectively. Let 𝑑 = 𝑑𝐴 + 𝑑𝐵.

Output: The product 𝐶 = 𝐴 ⋅ 𝐵.

1. Selection: Pick 𝜔 = (1, Τ2𝜋 𝑛), such that 𝑛 ≥ 𝑑 + 1 and 𝑛 is a power of two.

2. Evaluation (FFT): Compute 𝐴 1 , 𝐴 𝜔 , 𝐴 𝜔2 , … , 𝐴 𝜔𝑛−1 . Compute 𝐵 1 , 𝐵 𝜔 , 𝐵 𝜔2 , … , 𝐵 𝜔𝑛−1 .

3. Multiplication: Compute 𝐶 𝜔𝑘 = 𝐴 𝜔𝑘 ⋅ 𝐵(𝜔𝑘), for all 𝑘 = 0,… , 𝑛 − 1.

4. Interpolation (inverse FFT): Recover 𝐶 𝑥 = 𝑐0 + 𝑐1𝑥 + 𝑐2𝑥

2 +⋯+ 𝑐𝑑𝑥𝑑.


Example: from values to coefficients• Let us consider a polynomial:

𝑃 𝑥 = 𝑎0 + 𝑎1𝑥 + 𝑎2𝑥2 + 𝑎3𝑥

3

• We have 𝑛 = 4 and 𝜔 = 𝑖. Let us assume that the evaluation at four different points is:

𝑃 1 = 2𝑃 𝑖 = 1 − 𝑖𝑃 −1 = 4𝑃 −𝑖 = 1 + 𝑖

• We want to calculate the coefficients (𝑎0, 𝑎1, 𝑎2, 𝑎3) using the inverse FFT, i.e.,

[a0, a1, a2, a3] =1

4FFT( 2, 1 − 𝑖, 4, 1 + 𝑖 , 𝜔−1)


Example: from values to coefficients


FFT( 2,1 − 𝑖, 4,1 + 𝑖 , 𝜔 = −𝑖)

FFT( 2,4 , 𝜔 = −1) FFT( 1 − 𝑖, 1 + 𝑖 , 𝜔 = −1)

FFT( 2 ,𝜔 = 1) FFT( 4 ,𝜔 = 1) FFT( 1 − 𝑖 , 𝜔 = 1) FFT( 1 + 𝑖 , 𝜔 = 1)

2 4 1 − 𝑖 1 + 𝑖

𝑟0 = 2 + 4 = 6𝑟1 = 2 − 4 = −2

𝑟0 = 1 − 𝑖 + 1 + 𝑖 = 2𝑟1 = 1 − 𝑖 − 1 − 𝑖 = −2𝑖

𝑟0 = 6 + 2 = 8𝑟2 = 6 − 2 = 4𝑟1 = −2 + −𝑖 −2𝑖 = −4𝑟3 = −2 − −𝑖 −2𝑖 = 0

𝑘 = 0 ቊ

𝑘 = 1 ቊ

𝑎0, 𝑎1, 𝑎2, 𝑎3 =1

4[8,−4,4,0] 𝑃 𝑥 = 𝑥2 − 𝑥 + 2

Example: from coefficients to values


FFT( 2,−1,1,0 , 𝜔 = 𝑖)

FFT( 2,1 , 𝜔 = −1) FFT( −1,0 , 𝜔 = −1)

FFT( 2 ,𝜔 = 1) FFT( 1 ,𝜔 = 1) FFT( −1 ,𝜔 = 1) FFT( 0 ,𝜔 = 1)

2 1 −1 0

𝑟0 = 2 + 1 = 3𝑟1 = 2 − 1 = 1

𝑟0 = −1 + 0 = −1𝑟1 = −1 − 0 = −1

𝑟0 = 3 − 1 = 2𝑟2 = 3 + 1 = 4𝑟1 = 1 + 𝑖 ⋅ −1 = 1 − 𝑖𝑟3 = 1 − 𝑖 ⋅ −1 = 1 + 𝑖

𝑘 = 0 ቊ

𝑘 = 1 ቊ

𝑦0, 𝑦1, 𝑦2, 𝑦3 = [2,1 − 𝑖, 4,1 + 𝑖]𝑦0, 𝑦1, 𝑦2, 𝑦3 = FFT( 𝑎0, 𝑎1, 𝑎2, 𝑎3 , 𝜔 = 𝑖)

𝑃 𝑥 = 𝑥2 − 𝑥 + 2

Exercises

1. Consider the polynomials 1 + 𝑥 − 2𝑥2 + 𝑥3 and −1 + 𝑥2:– Choose an appropriate power of two to execute the FFT for the

polynomial multiplication. Find the value of 𝜔.

– Give the result of the FFT for 𝑥2 − 1(no need to execute the FFT).

2. Consider the polynomials −1 + 2𝑥 + 𝑥2 and 1 + 2𝑥:– Choose an appropriate power of two to execute the FFT.

Find the value of 𝜔.

– Calculate their point-value representation using the FFT (execute the FFT algorithm manually).

– Calculate the product of the point-value representations.

– Execute the inverse FFT to obtain the coefficients of the product.


Fast Fourier Transformjordicf/Teaching/AP2/pdf/14_Fast... · 2020. 2. 7. · •Using the Master Theorem we conclude: RuntimeFFT𝑛=O(𝑛log𝑛) •Gilbert Strang (MIT, 1994):

Documents