Seminario Marco Agus, 4-10-2012

www.crs4.it/vic/

Technologies for improving real-time exploration of massive ((volumetricvolumetric))

models

Marco Agus[[email protected]]

CRS4Visual Computing

Cagliari, October 2012

M. Agus, Technologies for improving massive models understanding, October 2012

We focus on…….

• Advanced technologies for massive models rendering on light field displays

– Joint work with J. A. Iglesias Guitian, E. Gobbetti, F. Marton, F. Bettio, G. Pintore, R. Pintus, A. Zorcolo


Technology pillars

• Software: Scalable techniques for interactive rendering of massive models– State of the art methods able

to handle potentially infinite static surface and volume to handle potentially infinite static surface and volume models

• Hardware: Autostereoscopic light field displays– They are now able to render

the perceptual aura of real 3D objects

– They provide real compelling 3D experience without glasses


Research challenges• Light field display driving

– How to project geometry on such kind of systems?

• Image representation

– How to create effective light fields from 3D massive data?

– How to deal with volumes, surfaces, video streams?

• Evaluation

– Do light field displays improve perceptive cues and immersiveness?– Do light field displays improve perceptive cues and immersiveness?

– How to improve visual confort and depth discrimination?

• Interaction

– How to exploit light field displays to create natural interactionsystems?

– How to exploit display characteristics to provide meaningfulinformation?

• Applications

– How these kind of displays can be useful?


Main Contributions

José Antonio Iglesias Guitián, Enrico Gobbetti, and Fabio Marton. View-dependent Exploration of Massive Volumetric Mo dels on Large Scale Light Field Displays . The Visual Computer, 26(6--8): 1037-1047, 2010

Marco Agus, Fabio Bettio, Andrea Giachetti, Enrico Gobbetti, José Guitián, Fabio Marton, Jonas Nilsson, and Giovanni Pintore.An interactive 3D medical visualization system base d on a light field display. The Visual Computer, 25(9): 883-893, 2009

Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, Fabio Marton, and Giovanni Pintore. GPU Accelerated Direct Volume Rendering on an Inter active Light Field Display. Computer Graphics Forum, 27(3): 231-240, 2008. Proc. Eurographics 2008.

Fabio Marton, Marco Agus, Enrico Gobbetti, Giovanni Pintore, and Marcos Balsa. Natural exploration of 3D massive models on large-s cale light field displays using the FOX proximal na vigation techniqueComputers & Graphics, 2012.

Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, and Fabio Marton. Evaluating layout discrimination capabilities of co ntinuous and discrete automultiscopic displays.In Proc. Fourth International Symposium on 3D Data Processing, Visualization and Transmission, 2010.

Fabio Marton, Enrico Gobbetti, Fabio Bettio, José Antonio Iglesias Guitián, and Ruggero Pintus. A Real-time coarse-to-fine multiview capture system for all-in-focus rendering on a light-field displa y. In Proc. 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), 2010

Marco Agus, Giovanni Pintore, Fabio Marton, Enrico Gobbetti, and Antonio Zorcolo. Visual enhancements for improved interactive render ing on light field displays.In Eurographics Italian Chapter Conference, November 2011.


Main Contributions




VOLUMES

DATA




VIDEO STREAMS

SURFACES


Main Contributions




MEDICAL

APPLICATIONS




3D - TV

VIRTUAL

MUSEUMS


Main Contributions



USER PERFORMANCE

+

HUMAN FACTORS


Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, and Fabio Marton. Evaluating layout discrimination capabilities of co ntinuous and discrete automultiscopic displays.In Proc. Fourth International Symposium on 3D Data Processing, Visualization and Transmission, 2010.


PERCEPTION

INTERACTION


Light-field display: overview

• The key feature characterizing 3D light field displays is direction-selective light emission

• 3D display hardware based on commercially available commercially available technology developed by Holografikahttp://www.holografika.com– Specially arranged projector

array and a holographic screen

– Side mirrors increase the available light beams count

– Each projector emits light beams toward a subset of the points of the holographic screen


Physical behavior

• In the horizontal direction, selective light transmission

• Vertically, the screen scatters widely (diffuse behavior)

Projector

• Results in homogeneous light distribution and continuous 3D view

Screen

Light field


• In practice, at any moment in time, a given screen pixel has the same color when viewed from all vertical viewing angles

• In order to provide a full perspective effect, the vertical viewing angle must thus be shown.

Projection technique

viewing angle must thus be shown. We thus introduce a “virtual observer”, fixing the viewer height and distance from screen

• The resulting MCOP technique is exact for all viewers at the same distance from the screen and height as the virtual observer. It proves in practice to be a good approximation for all viewing positions in the display workspace.


Depth-dependent resolution

• The display design has consequences not only on the projection equation but also imposes limits on the spatial resolution that depends on depth

M. Agus, E. Gobbetti, J.A. Iglesias Guitián, F. Marton, and G. Pintore. GPU Accelerated Direct Volume Rendering on an Inter active Light Field Display. Computer Graphics Forum, 27(3), 2008. Proc. Eurographics 2008.

• In general, the size of the smallest feature that can be reproduced depends on the distance of its center from the screen and from the beam angular size


Rendering System overview

• Full-system architecture overview


Parallel rendering

• We employ a GPU cluster for rendering.

• Sort first parallel rendering approach

– Adaptive out-of-core GPU rendering vs Replicating data

– Static assignment: rendering process images.

• Good load balancing. Caused by the geometry of the display, • Good load balancing. Caused by the geometry of the display, with all projectors looking at the same portion of the model.


Light field displays considered

• Two light field displays are considered:– Large-scale model capable of visualizing

35Mpixels by composing images generated by 72 SVGA LED projectors.

• The screen has 160x90cm, 50◦ horiz. field-of view with 0.8◦ angular accuracy.

• Pixel size on the screen surface is 1.5mm.

• Rendering cluster: 18 Athlon64 3300 + Linux PCs equipped with two NVidia 8800GTS 640MB.PCs equipped with two NVidia 8800GTS 640MB.

• GPUs are several generations older. Slowdown of 2x−3x

– Small-scale model capable of visualizing 7Mpixels by composing images generated by 96 320x240 small CCDs

• The screen is 26 inches, 50 degrees horiz. field-of view with 0.8◦ angular accuracy.

• The display is driven by one linux PC equipped with one NVidia 8800GTS 640MB.


Massive volume rendering



Massive volume rendering


APPLICATION: APPLICATION: Massive volume

rendering



• Use CPU for …

– Creation & loading

– Octree refinement

– Encode current cut using an spatial index

• Use GPU for …

Technique overview

• Use GPU for …

– Stackless octree traversal

– Rendering

Enrico Gobbetti, Fabio Marton, and José Antonio Iglesias Guitián. A single-pass GPU ray casting framework for interactive out-of-core rendering of massive volumetric datasets.The Visual Computer, 24, 2008. Proc. CGI 2008.


Taking advantage of GPGPU

• Improved multi-resolution CUDA ray caster:

– Flexible ray traversal and compositing strategies

– Improved visibility feedback (e.g. scatter writes)scatter writes)

– Pre-integrated transfer-functions

– Integrate an adaptive frame reconstruction scheme


adaptive loaderpreprocessing

visibility

feedbackoctree refinement

[ creation and maintainance ] [ rendering ]

offli

ne

Method overview

volume

render

storage

octree node

database

has current working set enough accuracy?

yes

prepare to render

no

GPUCPU

offli

ne


Preview: massive volumes rendering


Massive surface rendering



Massive surface rendering


APPLICATION: APPLICATION: Virtual exploration

of massive surface models

Fabio Marton, Marco Agus, Enrico Gobbetti, Giovanni Pintore, and Marcos Balsa Rodriguez. Natural exploration of 3D massive models on large-s cale light field displays using the FOX proximal navigation te chnique. Computers & Graphics, 2012.


Handling massive models: overview

• Multiresolution structure based on a modification of Adaptive Tetra Puzzles

– Few Ktri Patches

– Optimized GPU/CPU communication

– Per-patch spatial index to organize patches in triangle strippatches in triangle strip

Giovanni Pintore, Enrico Gobbetti, Fabio Marton, Russell Turner, and Roberto Combet. An Application of Multiresolution Massive Surface Representations to the Simulation of Asteroid Missi ons . In Eurographics Italian Chapter Conference. Pages 9-16. Eurographics Association, November 2010.

Paolo Cignoni, Fabio Ganovelli, Enrico Gobbetti, Fabio Marton, Federico Ponchio, and Roberto Scopigno. Adaptive TetraPuzzles - Efficient Out-of-core Construc tion and Visualization of Gigantic Polygonal Models. ACM Transactions on Graphics, 23(3): 796-803, August 2004. Proc. SIGGRAPH 2004.


Handling massive models

• Offline construction

– spatial partition by longest edge bisection of tetrahedra

– fine-to-coarse parallel out-of-core simplification of the surface contained in diamondsdiamonds

• Run-time rendering

– selective refinement queries based on projected space error estimation on tetrahedron hierarchy

– adaptive loader rapidly produces LOD by combining precomputed patches


Results: massive models rendering


Multiview capture and rendering



Multiview capture and rendering


APPLICATION: Multiview capture Multiview capture

and rendering system

Fabio Marton, Enrico Gobbetti, Fabio Bettio, José Guitián, and Ruggero Pintus. A Real-time coarse-to-fine multiview capture system for all-in-focus rendering on a light-field display. In Proc. 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). 2011


Capture

• Simple front-end: a single PC acquires M-PEG video stream from low cost USB camera array.

• Each node need to receive all camera images since each projectors sees a images since each projectors sees a large portion of the display workspace

• Solution: front-end sends single-frame JPEG images through UDP multicast protocol to display cluster nodes

29


Upload to GPU

• Minimization of CPU work

– Moreover, old PCs not fast enough for JPG decoding.

• Pipelined CPU-GPU parallel decoding, interleaving:

– CPU:entropy decoding using libjpeg-turbo

– GPU: CUDA – GPU: CUDA

• dequantization,

• inverse-DCT,

• YCbCr to RGB conversion

– 6X speed-up wrt standard CPU solution

– Fast but still dominates the whole processingtime using old G80 GPUs!

30

CPUImg N-1

CPUImg 1

CPUImg 0

GPUImg 0

GPUImg N-2

GPUImg N-1


Depth estimation (CUDA)

• For each pixel of each projector

– Find depth and color

– identify a ray using equations which consider MCOP geometry of the display.

– identify the closest cameras to the ray using the camera viewing transforms.using the camera viewing transforms.

• Plane sweeping to find best depth

– Evaluate pixel similarities at different depths and keep the best one

31


Coarse to fine depth estimation & color sample

• Start at half depth with coarser resolution

• Upsample and refine

• Filter each level

• Up to final resolution

• Final color gathering• Final color gathering

32


Results: Multiview Capture System


Results: “First teleport experiment”


Illustrative methods



Illustrative methods


FOCUS: FOCUS: Improve volume understanding



Novel view-dependent illustrative tools

• The view-dependent characteristics of the display can be exploited to develop specialized interactive illustrative techniques designed to improve spatial understanding

• Simple head motions can reveal new aspects of the inspected data


Clip-plane with view-dependent context

• Traditional cut away visualization, when our view direction is orthogonal to the clip plane, while offering more helpful contextual information in other situations



• Compute distance from plane

• If distance is positive

• modify the opacity of samples by multiplying it by a view-dependent correction factor:

• Otherwise

• vary the opacity of the plane and shading parameters from the original ones at to full opacity and ambient plus emission shading at




Other view-dependent illustrative tools

• Context-preserving probe

• Band-picker• Band-picker

• More info in...

José Iglesias Guitián, Enrico Gobbetti and Fabio Marton.View-dependent exploration of massive volumetric models on large-scale light field displays.The Visual Computer, 26, 2010. Proc. CGI 2010.


System overview



System overview


FOCUS: FOCUS: Perceptual and performance

evaluation

Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, and Fabio Marton. Evaluating layout discrimination capabilities of continuous and discrete automultiscopic displays . In Proc. Fourth International Symposium on 3D Data Processing, Visualization and Transmission, 2010.


• The main goal of tests performed in the evaluation process is …

– … to elucidate if light field displays

Evaluation

– … to elucidate if light field displays could provide visual information not available with traditional volume rendering systems

• The main focus will be set on psychophysical tests.


• Stereopsis evaluation– Random dot spiral ramp– Depth discrimination

(overlapping disks)

• Spatial understanding

Evaluation

• Spatial understanding evaluation– Path tracing

performance evaluation


Evaluation results

Users rapidly recover all depth cues to instantaneously recognize complex structures

– Very useful for analysis of angiography datasets

• More details about the evaluation tests can be found in …M. Agus, F. Bettio, A. Giachetti, E. Gobbetti, J. A. Iglesias Guitián, F. Marton, J. Nilsson, and G. Pintore. An interactive 3D medical visualization system based on a light field display. The Visual Computer, Vol. 25, No. 9, pp. 883–893, 2009.

M. Agus, E. Gobbetti, J. A. Iglesias Guitián, F. Marton. Evaluating layout discrimination capabilities ofcontinuous and discrete automultiscopic displays. 3DPVT, 2010


System overview



System overview


FOCUS: Display adaptation

Marco Agus, Giovanni Pintore, Fabio Marton, Enrico Gobbetti, and Antonio Zorcolo. Visual enhancements for improved interactive render ing on light field displays.In Eurographics Italian Chapter Conference. Pages 1-7. Eurographics Association, November 2011.


Problem: visual discomfort

• Aliasing artifacts occur when objects are too far with respect to the screen

• Causes

– Geometry errors due to the discrete characteristics of projectors and displayprojectors and display

– Image-based calibration error

• subpixel errors on the display screen

• bigger errors as the distance from screen increases


Visual discomfort: our solution

• Non-linear depth remapping method

– constrains most of the scene to stay inside CVR

– drives users to focus on parts inside CVR

Depth-of-field simulation • Depth-of-field simulation method

– blurs parts of the scene in background

• Frame fade-out method

– reduces clipping artifacts due to the borders of the light field display


Non-linear depth remapping method

• Implemented on a vertex shader according to the equation

• B, F are the comfort thresholds

• DF, DB are the asymptotic output ranges


Depth of field simulation

• Depth-dependent blur to adapt the frequency content to display resolution

– Focused objects are sharp within CVR, around focal plane

– Background objects are instead blurred

– Objects close to the viewer cannot be blurred

• Implemented as image-based two-pass DOF simulation [Nguyen 2007]

– CoC proportional to display spatial resolution

– Post-processing pixel shader blending between hi-res and low-res images according to CoC and blurriness by stochastic Poisson sampling


Frame fade-out

• Limited angular workspace

– Objects can abruptly disappear while viewer moves horizontally

• Simple but effective solution

– Color blending which fades to background in boundary areas

– Objects smoothly fade out– Objects smoothly fade out

– Implemented in a fragment shader


Qualitative results

• Non-linear depth-mapping (front positions)

Without adaptation

Depth mapping DF = 1500 mm



Qualitative results

• Non-linear depth-mapping + DOF blur

Without adaptation

Depth mapping DB = 1000 mm

Depth mapping DB = 1000 mm + DOF


Qualitative results

Without adaptation



System overview



System overview


FOCUS: FOCUS: natural interaction

Fabio Marton, Marco Agus, Giovanni Pintore, and Enrico Gobbetti.FOX: The Focus Sliding Surface Metaphor for Natural Exploration of Massive Models on Large-scale Light FieldDisplays . In Proc. VRCAI. Pages 83-90, December 2011.


Natural interaction: requirements

• Light field display constraints

– Depth-dependent spatial resolution, calibration errors, angular boundsangular bounds

• Interaction metaphor should be simple

– Museum context

– Reduced number of DOFs

– Short learning time


FOX interface: key ideas

• Guided motion

– Scene is kept centered wrt display workspace

• DOF reduction

– Interaction consists of changing a contact point

Focus+

Visual confort

– Translation, rotation and zoom are coupled together

• Natural and intuitive

– Simple to learn: only one button

– It can be implemented with various devices

Ease of use+

Efficiency


FOX interface: components

• Translation and rotation

• Automatic zooming

Automatic hotspot • Automatic hotspot placement


Translation and rotation

• Surface slides on display hotspot

– (dx,dy) defines a displacement on current plane

– Closest point and normal is forced to display hotspot

– Up direction is kept

– Smoothing filters


Automatic zooming

• Speed-dependent automatic zooming

– Vector length dS = |dx,dy| defines rate of motion

– Pan-speed function with three states

– Steady (FOCUS) vs fast motion (EXPLORATION)


Hot-spot automatic placement

• Coarse sampling for computing average plane

• Hotspot depth corrected to put scene in focus


Resulting interface

• From device (2D pos + button state) to displacement vector dS

• Model matrix M function of dS and current state (model surface)

• Easy to implement with all pointing devices (2D mouse, 3D mouse, free-hand motion-tracking, touch-screens)

• Cursor glyphs indicating dS and pan-zoom state• Cursor glyphs indicating dS and pan-zoom state


Kinect implementation

• Hand tracking with gesture recognition (open/close)

– Covariance analysis in order to recognize singular points


Results

• Real-time exploration of David 0.25 mm model, composed of 970M triangles

• User evaluation:

– Metaphor comparison ( 5DOF ObjectInHand vs FOX)

– Device comparison (IS 9000 vs Kinect) – Device comparison (IS 9000 vs Kinect)


Interaction with IS device


Free-hand interaction (Kinect)


User evaluation

• 33 participants (26 novices, 7 experts)

• Quantitative evaluation

– FOX vs 5DOF, IS vs KINECT

– Guided and explorative tasks

– Task completion time and image quality– Task completion time and image quality

• Qualitative evaluation

– Metaphor comparison (ease of learning, ease of reaching positions, perceived 3D image quality, preferred one)

– Device comparison (ease of learning, ease of reaching positions, preferred one)


User evaluation: samples


Results: 3D image quality


Results: task completion time


Results: qualitative evaluation


Results: user preferences


Discussion

• Tasks are easier with FOX (especially for novice users)

• Overall image quality is sensibly better with FOX

• With respect to devices IS 900 performs better • With respect to devices IS 900 performs better (especially for novice users)

• FOX provides better comfort (even difference in ease of use is not significant)


Summary of contributions

• Interactive system for virtual exploration of massive models on light field displays

• Light field display adaptation techniques for improving visual perception

• Natural interaction metaphor for light field • Natural interaction metaphor for light field displays

• Perceptual and performance evaluation of light field displays for interactive exploration


• But there is still a lot of work to do ...– Improve light field representations

• Complex scenes (hybrid scenes, point-based representations, image-based representations)

• Model retargeting within the range of 3d displays

Current and future work

• Model retargeting within the range of 3d displays– Improve the interaction techniques

• e.g. natural interfaces for the manipulation of large models (gesture interfaces)

• Exploration of more complex virtual environments (jumps, fly-through, change of metaphors)


Take-home messages

• Data explosion �� Impossible to explore all data

you acquire

• Displays are evolving �� Many ways to improve

exploration of data and provide useful informationinformation

• Complexity reduction is needed

– Interface � Constrained but natural exploration

– Visual representation � Extraction of meaningful information+ Hints on exploratio + Integration and fusion


Take-home message

• Simple scene �� simple

exploration

– Interface can be simple

– User can explore all the scene to reach her target


Take-home message

• Huge scene �� Simple or complex interface?

– Where is the Pac-man food?

– How much time would Pac-Man need to complete the maze?


Marco Agus[[email protected]]

That’s all, folks…Thank you

CRS4CRS4Visual Computing

http://www.crs4.it/[email protected]

Seminario Marco Agus, 4-10-2012

Technology

enrico gobbetti

lightfield display

d massive models

largescale light field

surfaces fabio marton

fabio bettio

jos guitin

streams marco agus