NeuralArt 電腦作畫

Neural Art -‐-‐ 電腦作畫 by Mark Chang

A Neural Algorithm of Ar7s7c Style

•  作者： – Leon A. Gatys. – Alexander S. Ecker. – MaAhias Bethge

•  所屬單位： – Werner Reichardt Centre for Integra7ve Neuroscience and Ins7tute of Theore7cal Physics, University of Tubingen, Germany.

– Bernstein Center for Computa7onal Neuroscience, Tubingen, Germany.

如何作畫？

大腦畫家

景物畫風畫作

電腦類神經網路

大綱

•  人類視覺 •  電腦視覺 •  電腦作畫 •  作品展示

人類視覺

•  神經元 •  視覺傳遞途徑 •  錯覺

神經元

•  Neuron •  Ac7on Poten7al

Dendrite

Axon

Cell Body

Time

Voltage

Threshold

視覺傳遞途徑

Re7na

Visual Area V1

Visual Area V4 Inferior Temporal Gyrus (IT)

視覺傳遞途徑

Visual Area V1

Inferior Temporal Gyrus (IT)

Recep7ve Fields

Visual Area V4

錯覺

電腦視覺

•  Neural Networks •  Convolu7onal Neural Networks •  VGG 19

Neural Networks

n

W1

W2

x1

x2

b Wb

y

nin = w1x1 + w2x2 + wb

nout

=1

1 + e�nin

Sigmoid

Rec7fied Linear

nout

=

⇢nin

if nin

> 0

0 otherwise

Neural Networks

x

y

n11

n12

n21

n22

b b

z1

z2

Input Layer

Hidden Layer

Output Layer

W12,y

W12,x

W11,y

W11,b W12,b

W11,x W21,11

W22,12

W21,12

W22,11

W21,b W22,b

Convolu7onal Neural Networks

•  Convolu7onal Layer

depth

width width depth

weights weights

height

shared weight


•  Stride •  Padding

Stride = 1

Stride = 2

Padding = 0

Padding = 1


•  Pooling Layer 1 3 2 4

5 7 6 8

0 0 4 4

6 6 0 0

4 5

3 2

no overlap

no padding no weights

depth = 1

7 8

6 4

Maximum Pooling

Average Pooling


Convolu7onal Layer

Convolu7onal Layer Pooling

Layer

Pooling Layer

Recep7ve Fields Recep7ve Fields

Input Layer


Input Layer

Convolu7onal Layer with

Recep7ve Fields:

Max-‐pooling Layer with

Width =3, Height = 3

Filter Responses

Filter Responses

Input Image

VGG 19

•  Karen Simonyan & Andrew Zisserman. Very Deep Convolu7onal Networks for Large-‐scale Image Recogni7on.

•  ImageNet Challenge 2014 •  19 (+5) layers – 16 Convolu7onal layers (width=3, height=3) – 5 Max-‐pooling layers (width=2, height=2) – 3 Fully-‐connected layers

VGG 19

depth=64 conv1_1 conv1_2

maxpool

depth=128 conv2_1 conv2_2

maxpool

depth=256 conv3_1 conv3_2 conv3_3 conv3_4



maxpool maxpool maxpool

size=4096 FC1 FC2

size=1000 sogmax

電腦作畫

•  內容產生 •  畫風產生 •  作品產生

內容產生

大腦畫家景物

畫布計算兩者差異

神經反應

補上線條和顏色

內容產生

計算兩者差異

Filter Responses VGG19

修正差異

相片

畫布

修正後

Width*Height Depth

內容產生

Layer l’s Filter l Responses:

Lcontent

(P l, Cl) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2

Layer l’s Filter Responses: Lcontent

(P l, Cl) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2Input Photo: L

content

(p, c, l) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2

Lcontent

(p,x, l) =1

2

X

i,j

(X l

i,j

� P l

i,j

)2

@Lcontent

(p,x, l)

@X l

i,j

= X l

i,j

� P l

i,j

X l

X li,j

Input Canvas: x

Width*Height (j)

Depth (i)

Width*Height (j)

Depth (i)

內容產生

•  Backward Propaga7on Layer l’s Filter l Responses: X l

Input Canvas:

x

VGG19

@Lcontent

@x=

@Lcontent

@X l

@X l

x

x x� ⌘@L

content

@xUpdate Canvas

Learning Rate

內容產生

內容產生

VGG19

conv1_2 conv2_2 conv3_4 conv4_4 conv5_2 conv5_1

畫風產生

•  ”Style” is posi7on-‐independent

style extrac7on

畫風產生

VGG19 畫作

G

G

Filter Responses Gram Matrix

Width*Height

Depth

Depth

Depth

Posi7on-‐ dependent

Posi7on-‐ independent

畫風產生

1. .5

.5

.5

1.

1. .5 .25 1.

.5 .25 .5

.25 .25

1. .5 1.

Width*Height

Depth

k1 k2

k1

k2

Depth

Depth

Layer l’s Filter Responses Gram Matrix

F l1

F l2

F l3

F l4

F l1

F l2

F l3

F l4

G

Gli,j = F l

i · F lj

Gl4,1 = F l

4 · F l1

= 1⇥ 1 + 0⇥ 0.5 + 0⇥ 0 + ...

= 1

畫風產生

VGG19 Filter

Responses Gram Matrix

計算兩者差異

G

G

畫風

畫布

修正差異

修正後

畫風產生

Lstyle(a,x, l) =1

2

X

i,j

(X li,j �Al

i,j)2

@Lstyle(a,x, l)

@F li,j

= ((F l)T (X l �Al))j,i

Layer l’s Filter Responses

Layer l’s Gram Matrix

Layer l’s Gram Matrix

F li,j

Ali,j X l

i,j

Input Artwork:

Input Canvas: a

x

畫風產生

畫風產生

VGG19

Conv1_1 Conv1_1 Conv2_1


Conv1_1 Conv2_1 Conv3_1 Conv4_1

Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1

作品產生

Ltotal

= ↵Lcontent

+ �Lstyle

a

Lcontent

(p, c, l) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2

x

x x� ⌘@L

total

@xx

Filter Responses VGG19

Lcontent

(p,x)

Lstyle(a,x)

Gram Matrix

作品產生

VGG19 VGG19

Lcontent

(p,x) Lstyle(a,x)


Conv4_2

Ltotal

= ↵Lcontent

+ �Lstyle

作品產生

作品展示

•  內容 v.s. 畫風 •  不同起始狀態 •  不同VGG Layers •  素描、水彩 •  詩中有畫、畫中有詩

內容 v.s. 畫風

0.15 0.05

0.02 0.007

↵

�

不同起始狀態 noise 0.9 *noise + 0.1*photo photo

不同VGG Layers

Conv1_1 Conv2_1


Conv1_1 Conv2_1 Conv3_1 Conv4_1


↵

�= 0.002

素描、水彩

詩中有畫、畫中有詩

延伸閱讀 •  A Neural Algorithm of Ar7s7c Style –  hAp://arxiv.org/abs/1508.06576

•  Texture Synthesis Using Convolu7onal Neural Networks –  hAp://arxiv.org/abs/1505.07376

•  Convolu7onal Neural Network –  hAp://cs231n.github.io/convolu7onal-‐networks/

•  Neural Network Back Propaga7on –  hAp://cpmarkchang.logdown.com/posts/277349-‐neural-‐network-‐backward-‐propaga7on

•  電腦賦詩： –  hAp://www.slideshare.net/ckmarkohchang/computa7onal-‐poetry

程式碼

•  Python Tensorflow – hAps://github.com/ckmarkoh/neuralart_tensorflow

•  Python Theano – hAps://github.com/woonketwong/ar7fy

•  Python Theano (ipython notebook) – hAps://github.com/Lasagne/Recipes/blob/master/examples/styletransfer/Art%20Style%20Transfer.ipynb

•  Python deeppy – hAps://github.com/andersbll/neural_ar7s7c_style

圖片來源

•  hAp://www.taipei-‐101.com.tw/upload/news/201502/2015021711505431705145.JPG

•  hAps://github.com/andersbll/neural_ar7s7c_style/blob/master/images/starry_night.jpg?raw=true

特別感謝

•  臺大資工imlab

NeuralArt 電腦作畫

Technology