Web3D 2013 Tutorialmetagrid2.sv.vt.edu/~npolys/Web3D2013/Med_Vol... · Web3D 2013 Tutorial: Medical and Volume Rendering with X3D Nicholas F. Polys, Ph.D. Andy Wood Virginia Tech

Web3D 2013 Tutorial:

Medical and Volume Rendering with X3D

Nicholas F. Polys, Ph.D. Andy Wood Virginia Tech

Peter Leskovsky, John Congote,

Esther Novo, Luis Kabongo, Ph.D. VicomTech

Topics

• Scenarios and Motivation

• Image Formats

• Processing Pipelines

• Presenting and Publishing (X3D, X3DOM)

Volume Rendering more than medical imaging

• Geology

• Meteorology

• FlowVisualization

• Microscopy

• Paleontology

• Non-invasive sensing:

– Transportation security

– Manufacturing QA

….

Web3D Medical Working Group

• Reproducible rendering for stakeholders throughout the healthcare enterprise

• An n-D Presentation must include: – Structured and interactive virtual environment

display (2D & 3D objects and time series)

– Platform-independent, royalty-free technology to enable vendor innovation

– Can be rendered with or without stereoscopy

– Openly-published

X3D Volume Rendering

• Necessary and Sufficient node set for industry’s Greatest Common Denominator:

– Volume Component: render styles

– X3D version 3.3

• Two independent implementations:

– www.h3d.org

– www.instantreality.org

Volume Processing and Presentation Tools

• Data – Sample xxxxx.dcm

– X3D Content Examples http://www.web3d.org/x3d/content/examples/Basic/VolumeRendering/index.html

– Volvis.org

– http://www.osirix-viewer.com/datasets/

• Warning: some are compressed w/ jpg2000 !

• Tools – ImageJ : http://rsbweb.nih.gov/ij/

• Plugins: DICOM reader, DICOM header inspector

– Seg3D.org

– Slicer.org; ITK-Snap

– X3D-Edit 3.3

• Viewers – H3D.org

– InstantReality.org

– MedX3DOM

X3D Volume Rendering

• Composable Render Styles covering the state of the art – Formalizes parameters and transfer functions for 3D rendering & blending

• BoundaryEnhancementVolumeStyle • CartoonVolumeStyle • ComposedVolumeStyle • EdgeEnhancementVolumeStyle • OpacityMapVolumeStyle • ProjectionVolumeStyle • ShadedVolumeStyle • SilhouetteEnhancementVolumeStyle • ToneMappedVolumeStyle

– Greatest Common Denominator

• Assign different RenderStyles to different segments, blend two volumes – BlendedVolumeStyle – SegmentedVolumeData – IsoSurfaceVolumeData

• Clipping Planes are already specified in X3D 3.2 Rendering Component

Silhouette Opacity Map Cartoon

Medical Interchange Profile

• X3D 3.3 • Minimal X3D node set (footprint) to meet DICOM requirements:

– Core – Time – Networking – Grouping – Rendering – Shape – Geometry3D – Geometry2D – Text – Lighting – Texturing – Interpolation – Navigation – Environmental effects – Event utilities – Texturing3D – Volume rendering

Includes polygon, line and point rendering; metadata on any node

ISO Process

• [ X3D 3.3 ] ISO/IEC DIS 1 19775-1:2008

–Clause 33 : Texturing3D Component

–Clause 41 : Volume Rendering Component

–Annex L: Medical Interchange Profile

• Final Draft International Standard (FDIS)

• Change document and unified spec drafts available to Web3D & DICOM members (* now published)

Online Videos & Examples

• Web Video summary: – Extensible 3D (X3D) Volume Rendering

• https://snoid.sv.vt.edu/medical/X3DVolumes/videos/VolumeVis-X3D-collected.mp4 (65 mb)

• X3D Examples – http://www.web3d.org/x3d/content/examples/Basic/

VolumeRendering/index.html

• For other other Videos, Images and Scenes using the VolumeData and VolumeRenderStyles of X3D 3.3 Clause 41, please visit: – https://snoid.sv.vt.edu/medical/X3DVolumes/

Volume Presentation

Many techniques:

• Volume rendering

– 3DSplatting, ray tracing, pixelshaders

– Established CPU and GPU algorithms

• Surfaces – actual meshes

• Segments – identifying voxels as groups

• ISOSurfaces – rendered at a threshold

I: Scenarios and Motivation

X3D Presentation Demo

• Nicholas Polys, Andy Wood Virginia Tech

Screenshots

Example Volume Rendering Styles

(Head MRI, XML encoding)

<Transform DEF='backdrop' >

<VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-vispart.nrrd"'/>

<OpacityMapVolumeStyle />

</VolumeData>

</Transform>

Example Volume Rendering Styles

(Head MRI, optic segment)

<ISOSurfaceVolumeData surfaceValues='.15' dimensions='.75 1 1' >

<ImageTexture3D containerField="voxels" url='"./Segments/masked-optic.nrrd"'/>

<CartoonVolumeStyle />

</ISOSurfaceVolumeData>

(Head MRI, cerebrum segment)

<VolumeData dimensions='.75 1 1' >

<ImageTexture3D containerField="voxels" url='"./Segments/masked-cerebrum.nrrd"'/>

<ComposedVolumeStyle>

<CartoonVolumeStyle />

<EdgeEnhancementVolumeStyle gradientThreshold='.8' edgeColor='0 0 .5' />

</ComposedVolumeStyle>

</VolumeData>

I: Scenarios and Motivation

Image Formats

• 2D image slices with order and spacing define the voxels

• Typical image stacks as:

– Tiff

– Png

– dcm

• Sizing, smoothing, resampling can happen in 2.5D space (image stack)

Image J

• Many useful (and programmable) functions to deal with image stacks:

• Invert, Blur, resample, resize reduce bit depth

• ImageJ : http://rsbweb.nih.gov/ij/ • Lots of Plugins: eg DICOM reader, DICOM header

inspector

III: Processing Pipelines

DICOM.dcm

• Medical.nema.org

• Ongoing work Item for n-Dimensional Presentation State

• Working Group 11 (Display)

• Extensive use cases documented

• MPR and CPR parameters of members surveyed

• Supplement draft in preparation

NRRD

• Nearly Raw Raster Data

• http://teem.sourceforge.net/nrrd/

• Nrrd and the teem library facilitates the pre-processing tasks (cropping, quantizing, resampling), as well as the basic analysis tasks which help to understand the structure of a dataset (histograms, maximum intensity projections, connected components)

II: Image Formats

Seg3D

Segment3D – a Gui tool With multiple methods for segmentation : - Neighborhood Growth - Thresholding

Export .nrrd - all-in-one - separate files Seg3d.org

III: Processing Pipelines

Slicer

• Slicer.org

ITKSnap

• http://www.itksnap.org

X3D Edit 3.3

• X3D-Edit is an open-source Extensible 3D (X3D) Graphics authoring tool for simple error-free editing, authoring and validation of X3D scenes.

• Lots of translation and QA features

• Stand-alone or Netbeans plug-in

• https://savage.nps.edu/X3D-Edit/

Volume acquisition

DICOM data

Surfacing

Viewer

workstation

Compose WCS Model (Scene) for Rendering;

assign appearances, views, etc.

Segmentation

nDPS

Processing Scheme for DICOM

Use Cases

X3D

Segmentation: Supp 111

Mesh: Supp 132

Other 3D data

(e.g. CAD, X3D)

Volume Processing Pipelines Source Data

X3D

<ImageTexture2D>

<VolumeData>

<ImageTexture3D> -voxels

<IsoSurfaceVolumeData>

<ImageTexture3D> -voxels

<SegmentedVolumeData>

<ImageTexture3D> -voxels

<ImageTexture3D> -segment identifiers

Image Processor (ImageJ)

Volume Data Processing

(TEEM)

Segmentation • Seg3D • Slicer • ITKSnap

DICOM Plugin

Image Stack

DICOM Stack

Raw Volume

NRRD Volume

Segment IDs

Individual Segments

Volume Image Data Source Data

X3D

<ImageTexture2D>

<VolumeData>

<ImageTexture3D> -voxels

<IsoSurfaceVolumeData>

<ImageTexture3D> -voxels

<SegmentedVolumeData>

<ImageTexture3D> -voxels

<ImageTexture3D> -segment identifiers

Image Processor (ImageJ)

Volume Data Processing

(TEEM)

Segmentation • Seg3D • Slicer • ITKSnap

DICOM Plugin

Image Stack

DICOM Stack

Raw Volume

NRRD Volume

Segment IDs

Individual Segments

Source Data

• Stacks of 2D images with order and spacing data generated by imaging machinery

• Some typical file formats for image stacks: – Tiff

– Png

– Dcm

• Image processing techniques can accomplish much of the work for rendering in 2.5D – Smoothing, (simple) segmentation, resampling,

contrast, etc.

Working with Images

• Resolution – number of pixels in x and y dimensions • Channels – color images are 3 or 4 channel (RGB or RGBA), typical

volume stack is single channel (greyscale) • Bit Depth – number of bits to encode value of a pixel, per channel • Volume data becomes large quickly:

– 512 512x512 8-bit greyscale images: 134 million voxels and 128MB (raw)

– 1024^3 volume, 16-bit RGB: 1.07 billion voxels and 48GB (raw)

• ImageJ – Image processor designed to work with stacks – Many useful (and scriptable) functions: invert, blur, resample, resize,

mask, etc. – Website: http://rsbweb.nih.gov/ij/

Volume Data Source Data

X3D

<ImageTexture2D>

<VolumeData>

<ImageTexture3D> -voxels

<IsoSurfaceVolumeData>

<ImageTexture3D> -voxels

<SegmentedVolumeData>

<ImageTexture3D> -voxels

<ImageTexture3D> -segment identifiers

Image Processor (ImageJ)

Volume Data Processing

(TEEM)

Segmentation • Seg3D • Slicer • ITKSnap

DICOM Plugin

Image Stack

DICOM Stack

Raw Volume

NRRD Volume

Segment IDs

Individual Segments

Volume Data Formats

• Raw voxel data (image stack) – Requires external metadata

• DICOM (.dcm) – Digital Imaging and Communication in Medicine:

http://medical.nema.org

– [MORE POINTS HERE]

• NRRD (.nrrd) – Nearly Raw Raster Data: http://teem.sourceforge.net/nrrd

– Wrapper for raw or compressed image stacks with plain text metadata

– TEEM library for pre-processing and analysis

Segmentation Source Data

X3D

<ImageTexture2D>

<VolumeData>

<ImageTexture3D> -voxels

<IsoSurfaceVolumeData>

<ImageTexture3D> -voxels

<SegmentedVolumeData>

<ImageTexture3D> -voxels

<ImageTexture3D> -segment identifiers

Image Processor (ImageJ)

Volume Data Processing

(TEEM)

Segmentation • Seg3D • Slicer • ITKSnap

DICOM Plugin

Image Stack

DICOM Stack

Raw Volume

NRRD Volume

Segment IDs

Individual Segments

Segmentation

• Transfer functions and rendering techniques are powerful, but not always sufficient

• Segmentation - mark features of interest for special rendering treatment

• Some techniques: – Hand segmentation (painting)

– Threshold (used in many approaches)

– Volume growing

– Connected components

Segmentation Tools

• Seg3D – Large selection of segmentation algorithms and tools – Layered segment masks with multi-layer operations and export

flexibility – http://www.sci.utah.edu/cibc-software/seg3d.html

• ITKSnap – Active contour segmentation (volume growing) and manual

tools – http://www.itksnap.org/pmwiki/pmwiki.php

• 3D Slicer – Volume manipulation and segmentation, with a focus on

registration (multiple volumes) and rendering – http://slicer.org/

Volume Rendering With X3D Source Data

X3D

<ImageTexture2D>

<VolumeData>

<ImageTexture3D> -voxels

<IsoSurfaceVolumeData>

<ImageTexture3D> -voxels

<SegmentedVolumeData>

<ImageTexture3D> -voxels

<ImageTexture3D> -segment identifiers

Image Processor (ImageJ)

Volume Data Processing

(TEEM)

Segmentation • Seg3D • Slicer • ITKSnap

DICOM Plugin

Image Stack

DICOM Stack

Raw Volume

NRRD Volume

Segment IDs

Individual Segments

Volume Rendering With X3D

• Volume rendering component added in X3D 3.3 – [ISO document info here]

• Examples found at: http://www.web3d.org/x3d/content/examples/Basic/VolumeRendering/

• Browsers: – H3D (full support): http://www.h3dapi.org/ – InstantReality (partial support):

http://instantreality.org/

• [not sure about officially supported file types, per browser]

Rendering a Volume: VolumeData Node

• Most basic volume rendering node in X3D

• Contains the voxels to be rendered and render styles to do so

<VolumeData dimensions='1.28 1.28 1.0' > <!-- VolumeRenderStyle node here (optional) --> <ImageTexture3D containerField='voxels' url=' "path_to_dataset" '/> </VolumeData>

Rendering a Volume: IsoSurfaceVolumeData

• Similar to the basic VolumeData node, but renders a surface across voxel gradients

<IsoSurfaceVolumeData dimensions='1.28 1.28 1.28‘ surfaceValues='.15' contourStepSize='0' surfaceTolerance='0' containerField='children'> <CartoonVolumeStyle colorSteps='32' /> <ImageTexture3D containerField='voxels' url=' "skull.nrrd" '/> </IsoSurfaceVolumeData>

Rendering a Volume: SegmentedVolumeData

• Allows different render styles for segments of interest

• More on this later!

Applying Distinctive Rendering Techniques

• VolumeStyle nodes allow different rendering techniques/filters – Of the form *VolumeStyle

– Example: <ToneMapVolumeStyle />

• Child to a volume data node – Exception: composing and blending

• Default style – OpacityMapVolumeStyle with a linear, alpha-only transfer function – Interpolates voxel opacity linearly by intensity

Opacity Map VolumeStyle

• The default style, the basis for all enhancement styles • Has only one field, “transferFunction”

– Two or three dimensional texture – One to four components

<OpacityMapVolumeStyle> <ImageTexture3D containerField='transferFunction' url='"engineTransferSchnitt.png"' /> </OpacityMapVolumeStyle>

Default With Transfer Function

Projection VolumeStyle

• A raycasting technique • Depending on the value of the “type” field, returns either the MAX,

MIN, or AVERAGE of the voxel values along the ray • If “intensityThreshold” is specified, returns the first local min/max

above/below the threshol

<ProjectionVolumeStyle type='MAX' enabled='true' intensityThreshold='0' containerField='renderStyle'/>

Max Projection Threshold Max Default Style

Per-Voxel VolumeStyles

• View-Dependent

– Opacity Map (default)

– Enhancement Styles

• Boundary, Edge, Silhouette

– Cartoon

• Lighting-Dependent

– Tone Map

– Shaded

Opacity Map

Edge Enhanced Cartoon

Shaded Tone Map

Boundary Enhancement Style

• Modifies voxels based on how quickly their surface normals are changing:

<BoundaryEnhancementVolumeStyle boundaryOpacity='0.9' opacityFactor='0.9' retainedOpacity='0.2'>

Default Boundary Enhanced

Edge Enhancement

• Voxels are colored based on how close to perpendicular their normal are to the view, outside of a threshold.

• Useful for surface features, not internal features.

<EdgeEnhancementVolumeStyle enabled='true' edgeColor=‘1 0 0 1‘ gradientThreshold='0.4' containerField='renderStyle'/>

Default Edge Enhanced

Silhouette Enhancement

• Modifies the color and opacity of voxels based on their normal values • Unlike edge enhancement, it can be used to reveal internal features

<SilhouetteEnhancementVolumeStyle silhouetteBoundaryOpacity='1' silhouetteRetainedOpacity='.5' silhouetteSharpness='10' enabled='true' containerField='renderStyle'/>

Default Silhouette Enhanced

Cartoon VolumeStyle

• Renders voxels based on the normal value as one of a specified number of color steps between an orthogonal (plane surface) color and parallel color:

<CartoonVolumeStyle enabled='true' colorSteps='4' orthogonalColor='1 1 1 1' parallelColor='0 0 0 1' containerField='renderStyle'/>

2 color steps 4 color steps 8 color steps

Shaded VolumeStyle

• Voxel appearance is controlled by a material node, similar to normal geometry (relative to light source)

• Can be computationally expensive

<ShadedVolumeStyle lighting='true' shadows='true' enabled='true' phaseFunction='Henyey-Greenstein' containerField='renderStyle'> <Material ambientIntensity='0.8' diffuseColor='0 .5 1‘ shininess='0.08' specularColor='1 1 1'/> </ShadedVolumeStyle>

Default Shaded

Tone Mapped VolumeStyle

• Uses the Gooch shading model to color voxels based on their orientations relative to a light source, between a warm (facing light) and cool (facing away) color

<ToneMappedVolumeStyle warmColor='0 0 1 0' coolColor='1 1 0 0' />

Default Tone Mapped

Combining Styles: ComposedVolumeStyle

• Apply multiple rendering styles to a volume

• Styles are applied strictly in the order they appear

• All per-voxel style nodes are composable

– ProjectionVolumeStyle is not composable

Combining Styles: ComposedVolumeStyle

<ComposedVolumeStyle enabled='true' ordered='false' containerField='renderStyle'> <SilhouetteEnhancementVolumeStyle silhouetteBoundaryOpacity='1' silhouetteRetainedOpacity='.1' silhouetteSharpness='10' enabled='true' containerField='renderStyle'/> <EdgeEnhancementVolumeStyle edgeColor='.5 0 0' gradientThreshold='.8' enabled='true' containerField='renderStyle'/> </ComposedVolumeStyle>

+ Style1 (Edge Enhance) Style2 (Silhouette) Composed Styles

Multipart Volumes: SegmentedVolumeData

• Requires two textures: voxels (as normal) and segment identifiers for each voxel

• Specify multiple render styles, in identifier/segment order (default is opacity map) – If more segments than styles, last style applies to

remaining segments

• Individual segments may be turned on or off using “segmentEnabled” field – Styles still required to maintain indexing

Multipart Volumes: SegmentedVolumeData

<SegmentedVolumeData dimensions='2.304 2.304 1.116' containerField='children'> <ImageTexture3D containerField='voxels' repeatS='false' repeatT='false' repeatR='false' url=' "mri_ventricles.nrrd" '/> <ImageTexture3D containerField='segmentIdentifiers' repeatS='false' repeatT='false' repeatR='false' url=' "mri_ventricles_segment.nrrd" '/> <OpacityMapVolumeStyle enabled='true' containerField='renderStyle'/> <ToneMappedVolumeStyle enabled='true' coolColor='0 0 1 0' warmColor='1 1 0 0' containerField='renderStyle'/> </SegmentedVolumeData>

Voxels

Segments

Styles

Default Segmented Volume

Multipart Volumes: Multiple Volumes

• Use different VolumeData nodes to represent each segment

• More computationally expensive than SegmentedVolumeData, but more flexible

– Can manipulate individual segments

– More fine-grained rendering control

• Requires masked volumes (more pre-processing)

Multipart Volumes: Multiple Volumes

Segment1

Segment2

Style 1

Style 2

<VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-halfhead.nrrd"'/> <OpacityMapVolumeStyle /> </VolumeData> <VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-cerebrum.nrrd"'/> <ShadedVolumeStyle lighting="TRUE" shadows="TRUE" > <Material diffuseColor='0 .5 1' specularColor='1 1 1' ambientIntensity='0.8' shininess='0.08' /> </ShadedVolumeStyle> </VolumeData>

1

2 3

4 5 Composited Volume

Multipart Volumes: Blended Render Style

• Blends multiple volumes/segments in one VolumeData node

• BlendedVolumeStyle node contains its own voxels and render style nodes – Contained voxels rendered according to styles,

then blended with parent

– Weight of blending is configurable

• BlendedVolumeStyle is composable and nestable!

Multipart Volumes: BlendedVolumeStyle

+

Volume1

Volume2

Style2

Default Style1

Volume1 (Default) Volume2 (Tone Map) Blended Volume

<VolumeData dimensions='512 512 452' containerField='children'> <BlendedVolumeStyle weightConstant1='0.51' enabled='true' weightConstant2='0.5' weightFunction1='CONSTANT' weightFunction2='CONSTANT' containerField='renderStyle'> <ToneMappedVolumeStyle enabled='true' coolColor='0 0 1 0' warmColor='1 1 0 0' containerField='renderStyle'/> <ImageTexture3D containerField='voxels' repeatS='false' repeatT='false' repeatR='false' url=' "internals.nrrd" '/> </BlendedVolumeStyle> <ImageTexture3D containerField='voxels' repeatS='false' repeatT='false' repeatR='false' url=' "body.nrrd" '/> </VolumeData>

Multichannel Microscopy

Using Channels as segments, each with its own style

• https://www.cellimagelibrary.org/

• Online Survey:

– Which of the renderings is most helpful to understand the…?

Preliminary Results Proximity Connectivity

Containment N= 50

Presenting & Publishing

X3D WebGL based Volume Rendering

John Congote, Luis Kabongo, Esther Novo,

Peter Leskovsky ([email protected])

Welcome to San Sebastian

WHERE WE CAME FROM

Volume Visualization (back in 2011)

• Pure WebGL implementation

<HTML><HEAD><META http-equiv="Content-Type" content="text/html;

charset=ISO-8859-1">

<TITLE>Real-Time Interactive Visualization of Volumetric Data with

WebGL</TITLE>

<SCRIPT type="text/javascript" src="sylvester.js"></SCRIPT>

<SCRIPT type="text/javascript" src="glUtils.js"></SCRIPT>

<SCRIPT type="text/javascript" src="glAux.js"></SCRIPT>

<SCRIPT type="text/javascript" src="volumerc.js"></SCRIPT>

</HEAD><BODY onload="volumerc_main('./raycast-color.frag','./aorta-

low.jpg','./tf.png');" bgcolor="white">

<CENTER>

<H1>Real-Time Interactive Visualization of Volumetric Data with

WebGL</H1>

<canvas id="canvas_win" width="512" height="512" style="border:1px solid

black"></canvas>

</br>

<button type="button" onclick="volumerc_main('./raycast-

bw.frag','./aorta-low.jpg','./tf.png');">Aorta LowRes B/W</button>

<button type="button" onclick="volumerc_main('./raycast-

color.frag','./aorta-low.jpg','./tf.png');">Aorta LowRes Color</button>

<button type="button" onclick="volumerc_main('./raycast-

bw.frag','./aorta-high.jpg','./tf.png');">Aorta HighRes B/W</button>

<button type="button" onclick="volumerc_main('./raycast-

color.frag','./aorta-high.jpg','./tf.png');">Aorta HighRes

Color</button>

<P>Rotation with the mouse, and Alt+Mouse to Zoom</P>

</BR>

…

WebGL Solution

• Achievements:

– volume rendering of 3D texture adapted into a 2D atlas

– Use of color/opacity transfer function

– Existing VR shaders adapted & simplified for performance and compatibility issues

From WebGL to …

• Drawbacks: – Hard to maintain as the code was specific to each site-

page-solution-dataset …

– Existing shaders were not necessarily and easily portable

– Hard to integrate in applications, limited interaction, …

• We needed something to: – Facilitate loading volume information

– Integrate existing basic interaction solutions

– Let us focus on rendering styles instead of rendering tricks

– Use a standardised structure to allow portability

MedX3DOM (2012)

• Implementation to support advanced medical visualization on the Web without plugins

• MEDX3D standard implemented into the X3DOM framework

– MEDX3D: extension of the X3D ISO standard to

support advanced medical visualization functionality

– X3DOM: framework for integrating and manipulating X3D scenes as HTML5/DOM elements

Methodology for MedX3DOM Node generation for two components

• Texturing3D • X3DTexture3DNode

• ComposedTexture3D

• ImageTexture3D

• PixelTexture3D

• TextureCoordinate3D

• TextureCoordinate4D

• TextureTransformMatrix3D

• TextureTransform3D

• ImageTextureAtlas

• VolumeRendering • X3DComposableVolumeRenderStyleNode

• X3DVolumeDataNode

• X3DVolumeRenderStyleNode

• BlendedVolumeStyle

• BoundaryEnhancementVolumeStyle

• CartoonVolumeStyle

• ComposedVolumeStyle

• EdgeEnhancementVolumeStyle

• IsoSurfaceVolumeData

• OpacityMapVolumeStyle

• MPRVolumeStyle

• ProjectionVolumeStyle

• SegmentedVolumeData

• ShadedVolumeStyle

• SilhouetteEnhancementVolumeStyle

• ToneMappedVolumeStyle

• VolumeData

Implemented/In progress Defined

Not implemented Abstract

Atlas Image Type

• Atlas

– Composed image

– Parameters • Number of slices

• Number of rows

• Number of columns

Ex: Brain Atlas Image

The texture atlas is the mosaic of all the slices of a volume in one image, the order of the images is given by the rows and columns.

New Node ImageTextureAtlas

ImageTextureAtlas node defines a texture map by specifying a single image file that contains slides of complete 3D data. Parameters indicate the information necessary for mapping voxels to the geometry.

ImageTextureAtlas: ImageTexture { SFInt32 [in] numberOfSlices 0

SFInt32 [in] slicesOverX 0

SFInt32 [in] slicesOverY 0

}

Modified OpacityMapVolumeStyle

OpacityFactor: Global contrast value of the volume data LightFactor: Global radiance of light value of the volume data

OpacityMapVolumeStyle : X3DComposableVolumeRenderStyleNode { SFBool [in,out] enabled TRUE SFNode [in,out] metadata NULL [X3DMetadataObject] SFNode [in,out] transferFunction NULL [X3DTexture2DNode,X3DTexture3DNode] SFFloat [in,out] opacityFactor 0.01 [0,1] SFFloat [in,out] lightFactor 0.3 [0,1]

}

X3DOM (Simple Example) <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-

strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="X-UA-Compatible" content="chrome=1" />

<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />

<title>Hello World</title>

<link rel="stylesheet" type="text/css" href="x3dom.css" />

<script type="text/javascript" src="x3dom.js"></script>

</head>

<body>

<h1>XHTML Hello World</h1>

<p>With X3D-namespace and case senstive element/node names. Works with self-closing tags (e.g.

Viewpoint and Material)</p>

<X3D xmlns="http://www.web3d.org/specifications/x3d-namespace" showStat="false" showLog="false"

x="0px" y="0px" width="400px" height="400px“ altImg="helloX3D-alt.png">

<Scene>

<Viewpoint position='0 0 10' />

<Shape>

<Appearance>

<Material diffuseColor='0.603 0.894 0.909' />

</Appearance>

<Box DEF='box'/>

</Shape>

</Scene>

</X3D>

</body>

</html>

X3DOM (Simple Example)

MEDX3D <?xml version="1.0" encoding="utf-8"?>

<X3D version="3.2" profile="MedicalInterchange">

<Scene>

<Group>

<NavigationInfo type='"EXAMINE" "WALK"'/>

<Viewpoint description="first" position="0 0 0.6"/>

<Background skyColor="1 1 1"/>

<Transform scale=".001 .001 .001">

<!-- The volume -->

<VolumeData DEF="volume” rayStep="0.01“ useSlicing="false“

dimensions="250 250 156">

<Image3DTexture containerField="voxels“ url='"data/S76280/IM00010“>

</VolumeData>

</Transform>

</Group>

</Scene>

</X3D>

MedX3DOM <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN“ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-

strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="X-UA-Compatible" content="chrome=1" />

<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />

<title>Hello World</title>

<link rel="stylesheet" type="text/css" href="x3dom.css" />

<script type="text/javascript" src="x3dom.js"></script>

</head>

<body>

<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'

showStat='true' showLog='true' width='500px' height='500px'>

<Scene>

<Background skyColor='0.0 0.3 0.65'/>

<Viewpoint description='Default' zNear='0.0001' zFar='100'/>

<Transform>

<VolumeData id='volume' dimensions='4.0 4.0 4.0'>

<ImageTextureAtlas containerField='voxels' url='media/volume/aorta4096.png'

numberOfSlices='96' slicesOverX='10' slicesOverY='10'/>

<OpacityMapVolumeStyle lightFactor='0.01'>

<ImageTexture containerField='transferFunction' url='transfer.png'/>

</OpacityMapVolumeStyle>

</VolumeData>

</Transform>

</Scene>

</X3D>

</body>

</html>

Volume Visualization 2012

Aorta Example

<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace’ showStat='true' showLog='true' width='500px' height='500px'>

<Scene>

<Background skyColor='0.0 0.0 0.0'/>

<Viewpoint description='Default' zNear='0.0001' zFar='100'/>

<Transform>

<VolumeData id='volume' dimensions='4.0 4.0 4.0'>

<ImageTextureAtlas containerField='voxels' url='media/volume/aorta4096.png' numberOfSlices='96' slicesOverX='10' slicesOverY='10'/>

<OpacityMapVolumeStyle lightFactor='0.02' opacityFactor='0.4'>

<ImageTexture containerField='transferFunction' url='media/volume/transfer/transfer.png'/>

</OpacityMapVolumeStyle>

</VolumeData>

</Transform>

</Scene>

</X3D>

http://x3dom.org/x3dom/test/functional/volrenOpacityTestTF_aorta.xhtml

Skull Example

<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'

showStat='true' showLog='true' width='500px' height='500px'>

<Scene> <Background skyColor='0.0 0.0 0.0'/> <Viewpoint description='Default' zNear='0.0001'

zFar='100'/> <Transform> <VolumeData id='volume' dimensions='4.0 4.0

4.0'> <ImageTextureAtlas containerField='voxels'

url='media/volume/skull-at.png' numberOfSlices='256' slicesOverX='16' slicesOverY='16'/>

<OpacityMapVolumeStyle> </OpacityMapVolumeStyle> </VolumeData> </Transform> </Scene> </X3D>

Brain Example

<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'

showStat='true' showLog='true' width='500px' height='500px'>

<Scene>

<Background skyColor='0.0 0.0 0.0'/>

<Viewpoint description='Default' zNear='0.0001' zFar='100'/>

<Transform>

<VolumeData id='volume' dimensions='4.0 4.0 4.0'>

<ImageTextureAtlas containerField='voxels' url='media/volume/brain-at_4096.jpg' numberOfSlices='226' slicesOverX='16' slicesOverY='16'/>

<OpacityMapVolumeStyle>

</OpacityMapVolumeStyle>

</VolumeData>

</Transform>

</Scene>

</X3D>

Aorta Example

<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'

showStat='true' showLog='true' width='500px' height='500px'>

<Scene> <Background skyColor='0.0 0.0 0.0'/> <Viewpoint description='Default' zNear='0.0001'

zFar='100'/> <Transform> <VolumeData id='volume' dimensions='4.0 4.0

4.0'> <ImageTextureAtlas containerField='voxels'

url='media/volume/aorta4096.png' numberOfSlices=‘96' slicesOverX='10' slicesOverY=‘10'/>

<OpacityMapVolumeStyle> </OpacityMapVolumeStyle> </VolumeData> </Transform> </Scene> </X3D>

MPR Style (2012)

<X3D xmlns='http://www.web3d.org/specifications/x3d-namespace’ showStat='true' showLog='true' width='500px' height='500px'>

<Scene>

<Background skyColor='0.0 0.0 0.0'/>

<Viewpoint description='Default' zNear='0.0001' zFar='100’ position="0.85713 1.49578 9.85028" orientation="-0.85521 0.50269 0.12614 0.17466" />

<Transform>

<VolumeData id='volume' dimensions='4.0 4.0 4.0'>

<ImageTextureAtlas containerField='voxels' url='media/volume/aorta4096.png’ numberOfSlices='96' slicesOverX='10' slicesOverY='10'/>

<MPRVolumeStyle id='style' positionLine='0.5'>

</MPRVolumeStyle>

</VolumeData>

</Transform>

</Scene>

</X3D>

http://x3dom.org/x3dom/test/functional/volrenMPR_aorta.xhtml

WORK IN PROGRESS

How to get an Atlas? …

• Do it by hand with an image editor – Time consuming

• Use an image viewer that can create a thumbnail atlas for you – A bit tricky to use, little flexibility

• Write your own C++/… application – A bit long, requires effort and little glory

• Use a python script like the one we made for you

NEW Python Script for Atlas

• Get it here: http://www.volumerc.org/documentation.html

• Requires Python, PIL • Use it as is:

– Parses all slices in a folder and merges then into a PNG atlas at different resolutions (full, 81922, 40962, 20482, 10242)

– Automatic data rescale – DICOM compatibility, uses

Window/Level values (requires pydicom and numpy)

• Personalize it for your own data format: – Implement loadMyData function

Not optimised!! (memory)

Usage: command <InputFolder> <OutputFilename>

- <InputFolder> must contain only one set of

PNG files to be processed

- <OutputFilename> must contain the path and

base name of the desired output, extension

will be added automatically

How to extend it …

• Follow X3DOM extension guide:

– http://x3dom.org/docs/dev/components.html

• Example:

– Implement the iso surface volume raycasting

• Process:

– Check the supporting nodes within X3D specification

• Found! -> ISOSurfaceVolumeData

– Take the X3DOM source code from github

– Extend functionality

VolumeRendering.js BEFORE

/* ### ISOSurfaceVolumeData ### */ x3dom.registerNodeType( "ISOSurfaceVolumeData", "VolumeRendering", defineClass(x3dom.nodeTypes.X3DVolumeDataNode, function (ctx) { x3dom.nodeTypes.ISOSurfaceVolumeData.superClass.call(this, ctx); this.addField_MFNode('renderStyle', x3dom.nodeTypes.X3DVolumeRenderStyleNode); this.addField_SFNode('gradients', x3dom.nodeTypes.X3DTexture3DNode); this.addField_MFFloat(ctx, 'surfaceValues', []); this.addField_SFFloat(ctx, 'contourStepSize', 0); this.addField_SFFloat(ctx, 'surfaceTolerance', 0); }, { nodeChanged: function() {}, fieldChanged: function(fieldName) {} } ) );

VolumeRendering.js AFTER /* ### ISOSurfaceVolumeData ### */

x3dom.registerNodeType(

"ISOSurfaceVolumeData",

"VolumeRendering",

defineClass(x3dom.nodeTypes.X3DComposableVolumeRenderStyleNode,

function (ctx) {

//Some initialization code… },

{

nodeChanged: function() {},

fieldChanged: function(fieldName) {},

uniforms: function() {

//Define the uniforms for the shaders },

textures: function() {

//Prepare the textures },

vertexShaderText : function() {

//Generate vertex shader },

fragmentShaderText : function (numberOfSlices, slicesOverX, slicesOverY) {

//Generate fragment shader }

}

)

);

Usage <X3D xmlns='http://www.web3d.org/specifications/x3d-namespace'

showStat='false' showLog='true' width='500px' height='500px'>

<Scene>

<Background skyColor='0.0 0.0 0.0'/>

<Viewpoint description='Default' zNear='0.0001' zFar='100'/>

<Transform>

<VolumeData id='volume' dimensions='4.0 4.0 4.0'>

<ImageTextureAtlas containerField='voxels'

url='media/volume/aorta4096.png' numberOfSlices='96'

slicesOverX='10' slicesOverY='10'/>

<ISOSurfaceVolumeData surfaceValues='0.2'

contourStepSize='0.2' lightFactor='0.02' opacityFactor='0.4'>

<ImageTexture containerField='transferFunction'

url='media/volume/transfer/transfer.png'/>

</ISOSurfaceVolumeData>

</VolumeData>

</Transform>

</Scene>

</X3D>

IN PROGRESS Result

ISO=20%

ISO=40%

ISO=60%

ISO=80%

ISO=80% Lighting OFF

FUTURE

DICOM, NRRD, RAW Format??

• Javascript is very slow to process binary data

• The loaders of this kind of volume data are based in C and the conversion or reimplementation is difficult

• Other types of volume formats exists, like MHD, NetCDF, …., imposible to implement all.

• MEDX3DOM is for the web, using web formats

MEDICAL IMAGING

• DICOM format is ´´the´´ standard for medical imaging, but internally is a container of several kind of data, sound, 2D images, 3D images, metadata,etc.

• WADO: It is a webservice which provides medical images through the web stored in a PACS server.

• MEDX3DOM: support WADO (Web Access to DICOM Objects)

MEDX3DOM Future …

• Next improvements – Integration Combination with mesh models

– Implementation of different styles (MIP, X-Ray, Composed …)

– Lighting (Phong, Global illumination …)

– Animation (4D timesteps, video, flow animation …)

– Data transfer optimization (streaming, compression,…) … collaborative visualization

Volume Rendering beyond medical imaging

• Geovisualization

• Volume Attention Maps

• Meteorology

• FlowVisualization

• Confocal microscopy

• ….

Keep updated

• Links:

– http://demos.vicomtech.org

– http://www.volumerc.org

– http://www.web3d.org/x3d/specifications/x3d

– http://www.x3dom.org

Thank you!

• Acknowledgements: – Web3D Consortium (X3D / MEDX3D)

– Fraunhofer IGD (X3DOM)

– John Edgar CONGOTE CALLE (main architect and developer of this initiative) ... now Developer Relations Engineer @Unity3D