MoSculp: Interactive Visualization of Shape and Timemosculp.csail.mit.edu/assets/slides_compressed.pdf · 2018. 10. 20. · MoSculp: Interactive Visualization of Shape and Time Xiuming

MoSculp: Interactive Visualization of Shape and Time

XiumingZhang1

William T. Freeman1,2

TaliDekel1,2

TianfanXue1,2

Stefanie Mueller1

QiuruiHe1,2

Andrew Owens1,3

JiajunWu1

1 MIT CSAIL 2 Google Research 3 UC Berkeley

1

2

3

4

5

6Video Courtesy of Tom Buehler (MIT CSAIL)

Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion

7

Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion

8

Motivation

9

Muybridge, The Human Figure in Motion, 1901

Motivation

10

Muybridge, The Human Figure in Motion, 1901 Edgerton, Back Dive, 1954

Motivation

11

Muybridge, The Human Figure in Motion, 1901 Edgerton, Back Dive, 1954Duchamp, Nude

Descending a Staircase, No. 2, 1912

Related Work

12

Edgerton, Stroboscopic Photography, 1927–1931

2D

Related Work

13

Edgerton, Stroboscopic Photography, 1927–1931

2D

Freeman & Zhang, Shape-Time Photography, CVPR ’03

Requires a depth camera

Related Work vs. Ours

14

Edgerton, Stroboscopic Photography, 1927–1931

2D

Freeman & Zhang, Shape-Time Photography, CVPR ’03

Requires a depth camera

MoSculp

3D w/ an RGB camera

Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion

15

System Walkthrough

16

17

18

19

20

21

22

23

24

25

Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion

26

User Studies: Design Choices

27

With Floor Reflections WithoutPreferred by 82%

User Studies: Efficacy in Conveying Motion

28

Baseline 1 (Stroboscopic)

Baseline 2(Shape-Time)

MoSculpPreferred by 75%

Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion

29

30

Input Video Motion Sculpture Generation

Depth-Preserving Compositing

3D Shape & Pose Estimation

… …

Overview

31

Input Video Motion Sculpture Generation

Depth-Preserving Compositing

3D Shape & Pose Estimation

… …

Overview

Approach: 2D Keypoint Detection

32

Time: t

[Cao et al., CVPR ’17]

Approach: 2D Keypoint Detection

33

[Cao et al., CVPR ’17]

Time: t + 1

34

Hidden Markov Model

Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip

35

Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip

• Small reprojection error

362D Image3D Model

[Loper et al., ToG ’15]

Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip

• Small reprojection error • Large probability of the poses

37

p( ) > p( )

Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip

• Small reprojection error • Large probability of the poses• Smooth evolution of poses

38

Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip

• Small reprojection error • Large probability of the poses• Smooth evolution of poses

39

Per-Frame Optimization[Bogo et al., ECCV ’16]

Original Camera View Novel View

Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip

• Small reprojection error • Large probability of the poses• Smooth evolution of poses

40

Original Camera View Novel View

Our Joint Optimization

41

Input Video Motion Sculpture Generation

Depth-Preserving Compositing

3D Shape & Pose Estimation

… …

Overview

Approach: Sculpture Generation

42

43

44

45

Input Video Motion Sculpture Generation

Depth-Preserving Compositing

3D Shape & Pose Estimation

… …

Overview

Approach: Compositing• Key challenge: how to “put together” 3D sculpture and 2D video?

46

Approach: Compositing• Naive Compositing: sculpture on top of the frames

47

Approach: Compositing• Full 3D Rendering: texturing the 3D models

48

Skirt Not Covered by 3D Model

Approach: Compositing• Solution: depth-preserving composite

49

Rendered Depth (Refined)

Approach: Compositing• Solution: depth-preserving composite

50

Approach: Before Refinement

51

Approach: After Refinement

52

Outline• Related Work• System Walkthrough• User Studies• Approach• Results• Conclusion

53

54

55

56

Single-Frame Shape and Pose Estimation

57

Our Joint Estimation

58

59

60

61

62

Texture from

Original Frames

63

64

65

Handling a Moving Camera

66

Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion

67

Limitation: Repeated, Localized Motion

68

69

Conclusion

70

Video Courtesy of Tom Buehler (MIT CSAIL)

Please come to our demo D-12 for more!

Thank you!71

http://mosculp.csail.mit.edu