Machine Learning for Computer Graphics An brief introduction By Dr. Zhang Hongxin State Key Lab of CAD&CG, ZJU.

Machine Learning for Machine Learning for

Computer GraphicsComputer Graphics An brief introductionAn brief introduction

By Dr. Zhang HongxinBy Dr. Zhang Hongxin

State Key Lab of CAD&CG, ZJU State Key Lab of CAD&CG, ZJU

OutlineOutline

BackgroundBackground What is Machine Learning?What is Machine Learning? Is it really useful for computer Is it really useful for computer

graphics?graphics? Our planOur plan

The largest challenge of The largest challenge of Today’s CG&AToday’s CG&A The tedious effort required to create digital The tedious effort required to create digital

worlds and digital life.worlds and digital life. ““Finding new ways to communicate and new kinds of Finding new ways to communicate and new kinds of

media to create.”media to create.”

Filmmakers, scientists, graphic designers, fine artists, Filmmakers, scientists, graphic designers, fine artists, and game designers.and game designers.

Pure procedural synthesis vs. Pure procedural synthesis vs. Pure dataPure data Creating motions for a character in a movieCreating motions for a character in a movie

Pure procedural synthesis.Pure procedural synthesis. compact, but very artificial, rarely use in compact, but very artificial, rarely use in

practice.practice. ““By hand” or “pure data”.By hand” or “pure data”.

higher quality but lower flexibility.higher quality but lower flexibility.

the best of both worlds: hybrid methods?!?the best of both worlds: hybrid methods?!?

Bayesian ReasoningBayesian Reasoning

Principle modeling of uncertainty.Principle modeling of uncertainty.

General purpose models for unstructured data.General purpose models for unstructured data.

Effective algorithm for data fitting and analysis Effective algorithm for data fitting and analysis under uncertainty.under uncertainty.

But currently it is always used as an black But currently it is always used as an black box.box.

Data driven modelingData driven modeling

What is machine learning?What is machine learning?

Data mining

(KDD)

Data-base

Computer Vision

Multi-media Bio-informatics

Artificial Intelligence

Computer Graphics

Machine LearningStatistics and

Bayesian methods

Control and information Theory

What is machine learning? What is machine learning? (Cont.)(Cont.) Definition by Mitchell, 1997

A program learns from experience E with respect to some class of tasks T and performance measure P, if its performance at task T, as measured by P, improves with experience E.

Learning systems are not directly programmed to solve a problem, instead develop own program based on:

examples of how they should behave from trial-and-error experience trying to solve the problem

Different than standard CS: want to implement unknown function, only have access to sample input-output pairs (training examples)

Hertzmann, 2003 For the purposes of computer graphics, machine learning should

really be viewed as a set of techniques for leveraging data(???).

Why Study Learning?Why Study Learning?

Develop enhanced computer systemsDevelop enhanced computer systems automatically adapt to user, customize often difficult to acquire necessary knowledge discover patterns offline in large databases

(data mining) Improve understanding of human, biological learning

computational analysis provides concrete theory, predictions explosion of methods to analyze brain activity during

learning Timing is good

growing amounts of data available cheap and powerful computers suite of algorithms, theory already developed

Main class of learning problemsMain class of learning problems

Learning scenarios differ according to the available information in training examples

Supervised: correct output available Classification: 1-of-N output (speech recognition,

object recognition,medical diagnosis) Regression: real-valued output (predicting market

prices, temperature) Unsupervised: no feedback, need to construct

measure of good output Clustering : Clustering refers to techniques to

segmenting data into coherent “clusters.” Reinforcement: scalar feedback, possibly

temporally delayed

And more …And more …

Time series analysis.Time series analysis. Dimension reduction.Dimension reduction. Model selection.Model selection. Generic methods.Generic methods. Graphical models.Graphical models.

Is it really useful for computer Is it really useful for computer graphics?graphics? Con: Everything is machine learning or Con: Everything is machine learning or

everything is human tuning?everything is human tuning? Sometimes, this may be true.Sometimes, this may be true.

Pro: more understanding of learning, but Pro: more understanding of learning, but yields much more powerful and effective yields much more powerful and effective algorithms. algorithms. Problem taxonomy.Problem taxonomy. General-purpose models.General-purpose models. Reasoning with probabilities.Reasoning with probabilities.

I believe the mathematic magic.I believe the mathematic magic.

Mesh Processing– Mesh Processing– clustering/segmentationclustering/segmentation

Hierarchical Mesh Decomposition using Fuzzy Clustering Hierarchical Mesh Decomposition using Fuzzy Clustering and Cuts. and Cuts.

By Sagi Katz and Ayellet Tal, SIGGRAPH 2003By Sagi Katz and Ayellet Tal, SIGGRAPH 2003

Texture synthesis and analysisTexture synthesis and analysis- Hidden Markov Model- Hidden Markov Model

""Texture Synthesis over Arbitrary Manifold Surfaces", by Li-Yi Wei and Texture Synthesis over Arbitrary Manifold Surfaces", by Li-Yi Wei and Marc Levoy. In Proceedings of SIGGRAPH 2001. Marc Levoy. In Proceedings of SIGGRAPH 2001.

"Fast Texture Synthesis using Tree-structured Vector Quantization", by Li-"Fast Texture Synthesis using Tree-structured Vector Quantization", by Li-Yi Wei and Marc Levoy. In Proceedings of SIGGRAPH 2000. Yi Wei and Marc Levoy. In Proceedings of SIGGRAPH 2000.

Image processing and synthesis Image processing and synthesis by graphical Modelby graphical Model

Image Quilting for Texture Synthesis and Transfer. Alexei A. Image Quilting for Texture Synthesis and Transfer. Alexei A. Efros and William T. Freeman. SIGGRAPH 2001.Efros and William T. Freeman. SIGGRAPH 2001.

Graphcut Textures: Image and Video Synthesis Using Graph Graphcut Textures: Image and Video Synthesis Using Graph Cuts. Vivek Kwatra, Irfan Essa, Arno Schödl, Greg Turk, Aaron Cuts. Vivek Kwatra, Irfan Essa, Arno Schödl, Greg Turk, Aaron Bobick. SIGGRAPH 2003.Bobick. SIGGRAPH 2003.

Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

BTF – reflectance texture BTF – reflectance texture synthesissynthesis

Synthesizing Bidirectional Texture Functions for Real-World Surfaces. Xinguo Liu, Yizhou Yu and Heung-Yeung Shum.

More recent papers…

Style machines - Time series Style machines - Time series analysisanalysis

By Matthew Brand (MERL) and By Matthew Brand (MERL) and Aaron Hertzmann. SIGGRAPH 2000Aaron Hertzmann. SIGGRAPH 2000

Motion texture - linear dynamic systemMotion texture - linear dynamic system

Yan Li, Tianshu Wang, and Heung-Yeung Shum. Motion Yan Li, Tianshu Wang, and Heung-Yeung Shum. Motion Texture: A Two-Level Statistical Model for Character Texture: A Two-Level Statistical Model for Character Motion Synthesis.Motion Synthesis.

Video Textures – Reinforce LearningVideo Textures – Reinforce Learning

Arno Schödl, Richard Szeliski, David H. Salesin, and Irfan Essa. Video textures. Proceedings of SIGGRAPH 2000, pages 489-498, July 2000.

Human shapes – Dimension Human shapes – Dimension ReductionReduction

The Space of Human Body Shapes: Reconstruction and The Space of Human Body Shapes: Reconstruction and Parameterization From Range Scans.Parameterization From Range Scans. Brett Allen, Brian Brett Allen, Brian Curless, Zoran Popović. SIGGRAPH 2003.Curless, Zoran Popović. SIGGRAPH 2003.

A Morphable Model for the Synthesis of 3D Faces.A Morphable Model for the Synthesis of 3D Faces. Volker Volker Blanz and Thomas Vetter. SIGGRAPH 1999.Blanz and Thomas Vetter. SIGGRAPH 1999.

Our PlanOur Plan

The SIG-ML4CGThe SIG-ML4CG http://www.cad.zju.edu.cn/home/zhttp://www.cad.zju.edu.cn/home/z

hx/ML4CG/hx/ML4CG/

ScheduleSchedule

TextbooksTextbooks

Information Theory, Inference, and Information Theory, Inference, and Learning Algorithms, Learning Algorithms, by David MacKay. by David MacKay.

Machine Learning Machine Learning by Tom Mitchell.by Tom Mitchell.

Data mining: Concepts and TechniquesData mining: Concepts and Techniques By Jiawei Han and Micheline KamberBy Jiawei Han and Micheline Kamber

Pattern Classification (2nd ed.) Pattern Classification (2nd ed.) by Richard O. Duda, Peter E. Hart and David G. by Richard O. Duda, Peter E. Hart and David G.

ReferenceReference

LinksLinks ML4CG by HertzmannML4CG by Hertzmann http://www.http://www.dgpdgp..torontotoronto..eduedu

/~/~hertzmanhertzman/mlcg2003//mlcg2003/hertzmannhertzmann-mlcg2003.-mlcg2003.pdfpdf

DDM conceptsDDM concepts http://http://dataminingdatamining..iheihe..nlnl/symposium/intro./symposium/intro.htmhtm

Q & AQ & A

ThanksThanks