Convolutional Neural Networks on Graphs with Fast ... · regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks,

Convolutional Neural Networks on Graphswith Fast Localized Spectral Filtering

May 12 2017

GCNN

Content• 1. Introduction

• 2. Proposed Technique 2.1 Learning Fast Localized Spectral Filters 2.2 Graph Coarsening 2.3 Fast Pooling of Graph Signals

• 3. Numerical Experiments

2

GCNN

GCNN

3

GCNN

Abstract

• In this work, we generalize CNN from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words’ embedding, represented by graphs.

• We present a formulation of CNNs in the context of spectral graph theory. Importantly, the proposed technique offers the same linear computational complexity and constant learning complexity as classical CNNs

4

GCNN

Introduction

• The major bottleneck of generalizing CNNs to graphs, and one of the primary goals of this work, is the definition of localized graph filters which are efficient to evaluate and learn.

• CNNs extract the local stationarity property of the input data or signals by revealing local features that are shared across the data domain. These similar features are identified with localized convolutional filters or kernels.

• Localized kernels or compactly supported filters refer to filters that extract local features independently of the input data size, with a support size that can be much smaller than the input size.

5

GCNN

Contributions• Spectral formulation: GCNN built on established tools in graph

signal processing (GSP). • Strictly localized filters: Enhancing [4], the proposed spectral

filters are provable to be strictly localized in a ball of radius K.• Low computational complexity: The evaluation complexity of our

filters is linear.This method avoids the Fourier basis altogether, thus the expensive EVD necessary to compute it as well as the need to store the basis, a matrix of size n**2. Besides the data, our method only requires to store the Laplacian, a sparse matrix.

• Efficient pooling: We propose an efficient pooling strategy on graphs which, after a rearrangement of the vertices as a binary tree structure, is analog to pooling of 1D signals.

• Experimental results: superior both in accuracy and complexity to the pioneer spectral graph CNN[4]. And it performs similarly to a classical CNNs on MNIST.

6

GCNN

Proposed Technique

• Design of localized convolutional filters on graphs;

• Graph coarsening procedure that groups together similar vertices;

• Graph pooling operation that trades spatial resolution for higher filter resolution.

7

GCNN

Learning Fast Localized Spectral Filters

• There are two strategies to define convolutional filters:• 1. Spatial approach: challenge of matching local neighborhoods

• 2. Spectral approach: via convolutions with a Kronecker delta implemented in the spectral domain. The convolution theorem defines convolutions as linear operators that diagonalize in the Fourier basis.

- However, a filter defined in the spectral domain is not naturally localized and translations are costly due to the O(n**2) multiplication with the graph Fourier basis. But it can be overcome.

8

GCNN

Learning Fast Localized Spectral Filters

• 1. Graph Fourier Transform

• 2. Polynomial parametrization for localized filters

• 3. Recursive formulation for fast filtering

9

GCNN

Graph Fourier Transform

Too many maths symbols, I just conclude its tuition here:

1. Transform the undirected graph into graph Laplacian, which is Degree Matrix - Adjacency Matrix, then normalized it.

2. Find out the Fourier basis U (The Laplacian is indeed diagonalized by the Fourier basis)

3. The convolution operation become an element-wise Hadamard product in the Fourier domain.

10

GCNN

Polynomial parametrization for localized filters

1. Graph Fourier Transform's learning complexity is quite high and are not localized in space. These issue can be overcome with the use of a polynomial filter.

2. Considering a smoothing kernal, such as splines.

11

GCNN

Polynomial parametrization for localized filters

1. Graph Fourier Transform's learning complexity is quite high and are not localized in space. These issue can be overcome with the use of a polynomial filter.

2. Considering a smoothing kernal, such as splines.

12

GCNN

Recursive formulation for fast filteringUsing Polynomial parametrization is cool but the cost to filter a signal x is still high.(Actually these previous methods were proposed by [4])

So they come up with their methods: a polynomial function that can be computed recursively from L, as L is sparse, so the cost is much lower.

Approximation methods:

1. Chebyshev expansion(they use this one in their experiments)

2. Lanczos Algorithm

13

GCNN

Learning the filters

• Actually is the same as classical CNN, using Back Propagation over a mini-batch of S samples.

14

GCNN

Graph Coarsening

• After the Convolution operation, we need to think about the pooling operation. So we come across a problem: it requires meaningful neighborhoods on graphs, where similar vertices are clustered together. Doing this for multiple layers is equivalent to a multi-scale clustering of the graph that preserves local geometric structures.

• While this problem is NP-hard, so they use METIS, a greedy alogrithm to compute successive coarser versions of a given graph.

15

GCNN

Graph Coarsening

• After the Convolution operation, we need to think about the pooling operation. So we come across a problem: it requires meaningful neighborhoods on graphs, where similar vertices are clustered together. Doing this for multiple layers is equivalent to a multi-scale clustering of the graph that preserves local geometric structures.

• While this problem is NP-hard, so they use METIS, a greedy alogrithm to compute successive coarser versions of a given graph.

16

GCNN

Fast Pooling17

GCNN