Chris Oat ATI Research, Inc. Advanced Character Rendering.

Chris OatATI Research, Inc.

Advanced Character Rendering

2AGDC 2004 - Advanced Character RenderingATI Research, Inc.ATI Research, Inc.

Outline

• Skin Rendering– Algorithm Overview– Texture-space lighting– Blur and dilation– Soft shadows– Translucent Shadows– Acceleration technique: Early-Z culling optimization

• Hair Rendering– Modeling/Texturing– Shading– Depth sorting: Early-Z culling optimization


Skin Rendering

• The human eye is very sensitive to the perception of skin, particularly with respect to facial recognition

• Most of the lighting from skin comes from subsurface scattering

• Skin color is mainly from the epidermis• Pink/Red color mainly from blood in the dermis• Traditional Lambertian lighting model assumes

hard surfaces with no subsurface scattering so it doesn’t work well for skin


Approximate Skin Cross Section

Air

Epidermis

Dermis

Bone, muscle, guts, etc.


Our Objective

• Develop a simple, efficient skin rendering algorithm for real-time applications

• Must be very fast as it is only one of several rendering techniques that will be used concurrently to render realistic characters

• Results should approximate subsurface scattering


Basis for Our Approach

• SIGGRAPH 2003 sketch Realistic Human Face Rendering for “The Matrix Reloaded”

• Rendered a 2D light map• Simulate subsurface diffusion in image

domain (different for each color component)

• Used traditional ray tracing for areas where light can pass all the way through (e.g.. Ears)


Texture-Space Lighting

• Realistic Human Face Rendering for “The Matrix Reloaded” (SIGGRAPH’03):

• Our results:

Matrix: Reloaded

Ruby: The Double Cross


Real Time Texture-Space Lighting

• Render diffuse lighting into an off-screen texture using texture coordinates as positions

• Blur the off-screen diffuse lighting• Read the texture back and add specular

lighting in subsequent pass• Bump map is used only for specular

lighting pass (high frequency detail is lost due to diffusion)


Texture Coordinate as Position• Light as a 3D model but draw into texture• Vertex shader outputs texture coordinates as projected

“position” then the rasterizer does the unwrap• Vertex shader computes light vectors based on 3D

position and interpolates


Basic Approach

Geometry

Light in Texture Space

Blur

Back Buffer

Sample texture space light


VsOutput main (VsInput i){ // Compute output texel position in screen space VsOutput o; o.vPos.xy = i.texCoord*2.0-1.0;// move from [0,1] to [-1,1] o.vPos.zw = 1.0;

// Pass along texture coordinates o.texCoord = i.texCoord;

// Skin (Transform vertex from object space to world space) float4x4 mSkinning = SiComputeSkinningMatrix(i.weights, i.indices); float4 vPos = mul(i.vPos, mSkinning); o.vNormal = mul(i.vNormal, mSkinning);

// Perspective divide: needed for shadow mapping vPos = vPos/vPos.w;

// Compute light vectors using world space vertex // position . . .

Texture-Space Lighting: Vertex Shader


float4 main (PsInput i) : COLOR{ // Compute Diffuse for Light 0 float3 vNormal = normalize(i.vNormal); float3 cLightColor = SiGetLightColor(0); float3 vLight = normalize(i.vLightVec0); float3 cDiffuse = dot(vNormal, vLight) * cLightColor; // Compute Diffuse for Light 1 & 2 . . .

// Look up blur size and save in alpha float sBlurSize = tex2D(tBlurSize, i.vTexCoord);

float4 o = float4(cDiffuse, sBlurSize); return o;}

Texture-Space Lighting: Pixel Shader


Spatially Varying Blur

• Blur to simulate the subsurface scattering component of skin lighting

• Use a grow-able Poisson disc filter• Read the kernel size from a texture• Allows varying the subsurface effect

– Higher for places like ears/nose

– Lower for places like cheeks


Filter Kernel

• Stochastic sampling• Poisson distribution• Samples stored as 2D

offsets from center

Center Sample

Outer Samples


Blur Kernel Size Map

Blur Kernel Blur Kernel Size MapSize Map

Texture Space Texture Space LightingLighting

ResultResult


Dilation• Texture seams can be a problem

(unused texels, bilinear blending artifacts)• During the blur pass we need to dilate• Use alpha channel of off-screen texture• If any sample has 1.0 alpha, just copy the

sample with the lowest alpha


Dilation Results

Without Dilation With Dilation


Shadows

• Shadow mapping– Apply shadows during texture lighting pass

– Get “free” blur• Soft Shadows• Simulates subsurface interactions• Lower precision/size requirements• Reduces shadow map artifacts

– If using multiple lights, it’s still the same number of blur passes


Shadow Maps• Create projection matrix to

generate depth map from the light’s point of view

– Use bounding sphere of head/character to ensure texture space is used efficiently

• Shadow map rendering pass: Write depth (distance from light) into off-screen texture

• On texture lighting pass: Test depth values in texture-space lighting pixel shader


Basic Approach + Shadows

Geometry Light in Texture Space

Blur

Back Buffer


Generate Shadow Map


Shadow Map: Vertex Shader

float4x4 mSiLightProjection; // Light projection matrixVsOutput main (VsInput i){ VsOutput o;

// Compose skinning matrix float4x4 mSkinning = SiComputeSkinningMatrix(i.weights, i.indices);

// Skin position/normal and multiply by light matrix float4 vPos = mul(i.vPos, mSkinning); o.vPos = mul(vPos, mSiLightProjection);

// Compute depth (Pixel Shader is just pass through) float dv = o.vPos.z/o.vPos.w; o.depth = float4(dv, dv, dv, 1); return o;}


Texture Lighting with Shadows: Vertex Shader

VsOutput main (VsInput i){ // Same lead in code as before . . .

// Compute texture coordintates for shadow map o.vLightPos = mul(vPos, mSiShadowLight); o.vLightPos /= o.vLightPos.w; o.vLightPos.xy = (o.vLightPos.xy + 1.0f)/2.0f; o.vLightPos.y = 1.0f-o.vLightPos.y; o.vLightPos.z -= 0.01f; return o;}


Texture Lighting with Shadows: Pixel Shader

sampler tShadowMap;float sFaceShadowFactor;float4 main (PsInput i) : COLOR{ // Same lead in code . . .

// Compute Light 0 float3 cLightColor = SiGetLightColor(0); float3 vLight = normalize(i.vLightVec0); float NdotL = SiDot3Clamp(vNormal, vLight); float VdotL = SiDot3Clamp(-vLight, vView); float4 t = tex2D(tShadowMap, i.vLightPos.xy); float lfac = sFaceShadowFactor; if (i.vLightPos.z < t.z) lfac = 1.0f; float3 cDiffuse = lfac * saturate((fresnel*VdotL+NdotL)*lightColor);

// The rest of the shader is the same as before . . .}


Shadow Map & Shadowed/Lit Texture

Shadow Map Shadow Map (depth)(depth)

Shadows in Texture Shadows in Texture SpaceSpace


Result with ShadowsResult with Shadows


Shadow from Translucent Objects

• Algorithm– Draw depth of opaque shadow geometry to

shadow buffer’s alpha channel

– Additively blend RGB of translucent shadow geometry into shadow buffer RGB channels

• Depth test is ON• Depth write is OFF

– Texture-space lighting pixel shader must now blend non-shadowed pixels by the translucent RGB value


Basic Approach + Translucent Shadows

Opaque Geometry

Light in Texture Space

Blur

Back Buffer


Generate Shadow Map

Transparent Geometry

Blend translucency into shadow map


Shadow Map for Transparent Shadows

Translucent Translucent shadow mapshadow map

Opaque Opaque shadow mapshadow map RGBRGB


Translucent Shadow Results

Opaque ShadowsOpaque Shadows Translucent ShadowsTranslucent Shadows


Specular Lighting• Use a bump map for specular lighting• Per-pixel exponent• Need to shadow specular light

– Bright specular light in heavily shadowed regions wouldn’t make sense!

– Too expensive to do another blur pass shadows• Use luminance of blurred diffuse texture

– Modulate specular from shadowing light by luminance of texture space light (very low frequency)

– Darkens specular in shadowed areas but preserves specular in unshadowed areas

• Not a limitation, just an optimization– You could do a separate blur pass for shadows

(rather than bake them into diffuse light texture)


Normal Map Compression

• Exposed in Direct3D using FOURCC: ATI2 • Two channels (x and y)

– Each channel is compressed separately– Each uses a block encoding like DXT5 alpha

– Derive z in pixel shader

• Useful for tangent space normal maps (where the sign of the z component is known)

• Can be used for any two channel texture

221 yx


Final Pixel Shader (with Specular)

sampler tBase;sampler tBump;sampler tTextureLit;float4 vBumpScale;float4 main (PsInput i) : COLOR{ // Get base and bump map float4 cBase = tex2D(tBase, i.texCoord.xy); float3 cBump = tex2D(tBump, i.texCoord.xy);

// Get bumped normal float3 vNormal = SiConvertColorToVector(cBump); vNormal.z = vNormal.z * vBumpScale.x; vNormal = normalize(vNormal);


Final Pixel Shader (with Specular): Continued

// View, reflection, and specular exponent float3 vView = normalize(i.vView); float3 vReflect = SiReflect(vView, vNormal); float exponent = cBase.a*vBumpScale.z + vBumpScale.w;

// Get "subsurface" light from lit texture. float2 iTx = i.texCoord.xy; iTx.y = 1-i.texCoord.y; float4 cLight = tex2D(tTextureLit, iTx); float3 cDiffuse = cLight*cBase;


Final Pixel Shader (with Specular): Continued

// Compute Light 0 float3 cLightColor = SiGetLightColor(0); float3 vLight = normalize(i.vLight0); float RdotL = saturate(dot(vReflect, vLight)); float shadow = SiGetLuminance(cLight.rgb); shadow = pow(shadow,2); float3 cSpecular = saturate(pow(RdotL,exponent)*lightColor); cSpecular *= shadow;

// Compute Light 1 & 2 (same as above but no shadow term) . . .

// Final color float4 o; o.rgb = cDiffuse + cSpecular; o.a = 1.0; return o;}


Specular Shadow Dimming Results

Specular Without ShadowsSpecular Without Shadows Specular With ShadowsSpecular With Shadows


Using Early-Z for Culling

• Testing z-buffer prior to pixel shader execution– Can cull expensive pixel shaders– Only applicable when pixel shader does not output depth

• This texture-space operation doesn’t need the z-buffer• We could store data other than depth in z-buffer• Use Early-Z to cull useless computations

– Back face culling– Distance and frustum culling

• Set z-buffer on lighting pass according to frustum, distance from viewer and facing-ness of polygons

• Set z-test such that non-visible polygons fail z-test• Reduces cost of image-space blur in regions that don’t

need it (because they aren’t visible)


Back Face Culling

Over the shoulder view of RubyOver the shoulder view of Ruby

Back facing pixels culled using early-zBack facing pixels culled using early-z


Skin Rendering: Summary

• Simple and fast skin rendering algorithm• Texture-space blur with dilation• Soft shadows “for free”• Translucent shadows• Early-Z culling acceleration techniques


Demo


Skin Rendering References

• [Borshukov03] Borshukov and Lewis. Realistic Human Face Rendering for “The Matrix Reloaded”. Technical Sketches, SIGGRAPH 2003.

• [Mertens03] Mertens et al. Efficient Rendering of Local Subsurface Scattering. Pacific Graphics 2003.


Hair Rendering Overview

• Hair rendering technique using polygonal hair model

• Shading: Mix of– Kajiya-Kay hair shading model

– Marschner’s model presented at SIGGRAPH 2003

• Simple, approximate depth-sorting scheme

– Early-Z culling optimizations


Hair Rendering

• Hair is important visually– Most humans have some hair on their heads

• Hair is challenging to render:– There is a lot of it! (100k – 150k strands on a

human head)

– Many different hair styles

– ~25% of the total render time of Final Fantasy – The Spirits Within was spent on the main character’s hair


Hair Model - Geometry

• Several layers of patches to approximate volumetric qualities of hair

• Per-vertex ambient occlusion term to approximate self-shadowing


Hair Model – Geometry

Reasons for using a polygonal hair model:• Lower geometric complexity than line

rendering– Makes depth sorting easier/faster

– Pretty much a necessity for real time use on current graphics hardware

• Integrates well into current art pipelines


Hair Model - Textures

• Base texture– Stretch noise

– Hair color set in a shader constant

• Alpha texture– Should have fully

opaque regions

• Specular shift texture• Specular noise texture

Base Texture

Alpha Texture


Hair Lighting: Kajiya-Kay

• Anisotropic strand lighting model• Use hair strand tangent T instead of

normal N in lighting equations• Assumes hair normal to lie in plane

spanned by T and view vector V

• Example: Specular = N·H term

),dot( HNyspecularit

),dot(1

),sin(

2HT

HTyspecularit

yspecularit

T

N

V

L

H


Hair Lighting: Marschner

• Based on measurements of hair scattering properties

• Observations– Primary specular highlight

shifted towards hair tip

– Secondary specular highlight • Colored• Shifted towards hair root• Sparkling appearance

• For simplicity we’re trying to match these observations phenomenologically Hair strand

interior

Primaryhighlight Secondary

highlight


Hair Shader Implementation

• Vertex shader– Just passes down tangent, normal, view

vector, light vector and ambient occlusion term

• Pixel shader– Diffuse lighting

– Two shifted specular highlights

– Combines lighting terms


• Kajiya-Kay diffuse term sin(T, L) looks too bright without proper self-shadowing

• Instead, use scaled and biased N·L term:

• Brightens up areas facing away from the light when compared to plain N·L term

– Simple subsurface scattering approximation

• Softer look

Diffuse Lighting Term


Shifting Specular Highlights• Move tangent along patch normal

direction to shift specular highlight along hair strand

• Assuming T is pointing from root to tip:

– Positive shift moves highlight toward tip– Negative shift moves highlight toward

root• Look up shift value from texture to

break up uniform look over hair patches

T T’T”

N

float3 ShiftTangent (float3 T, float3 N, float shift){ float3 shiftedT = T + shift * N; return normalize(shiftedT);}


Specular Strand Lighting• Specular strand lighting using half-angle vector

– Using reflection vector and view vector would make the shader a little more complicated

• Two highlights with different colors, specular exponents and differently shifted tangents

• Modulate secondary highlightwith noise texture

float StrandSpecular (float3 T, float3 V, float3 L, float exponent){ float3 H = normalize(L + V); float dotTH = dot(T, H); float sinTH = sqrt(1.0 - dotTH*dotTH); float dirAtten = smoothstep(-1.0, 0.0, dot(T, H)); return dirAtten * pow(sinTH, exponent);}


Putting it All Together

float4 HairLighting (float3 tangent, float3 normal, float3 lightVec, float3 viewVec, float2 uv, float ambOcc){ // shift tangents float shiftTex = tex2D(tSpecShift, uv) – 0.5; float3 t1 = ShiftTangent(tangent, normal, primaryShift + shiftTex); float3 t2 = ShiftTangent(tangent, normal, secondaryShift + shiftTex);

// diffuse lighting: the lerp shifts the shadow boundary for a softer look float3 diffuse = saturate(lerp(0.25, 1.0, dot(normal, lightVec)); diffuse *= diffuseColor;

// specular lighting float3 specular = specularColor1 * StrandSpecular(t1, viewVec, lightVec, specExp1); // add 2nd specular term, modulated with noise texture float specMask = tex2D(tSpecMask, uv); // approximate sparkles using texture specular += specularColor2 * specMask * StrandSpecular(t2, vieVec, lightVec, specExp2); // final color assembly float4 o; o.rgb = (diffuse + specular) * tex2D(tBase, uv) * lightColor; o.rgb *= ambOcc; // modulate color by ambient occlusion term o.a = tex2D(tAlpha, uv); // read alpha texture return o;}

(Note: external constants are orange)


Combination of Lighting Terms

lusionambientOccebaseTextur

specularspeculardiffusefinalColor

)( 21

Ambient Occlusion Diffuse Term Specular Terms Combined


Comparison

Kajiya-Kay Marschner et al. Photograph Our shader

(Left three pictures from [Marschner03])


Approximate Depth Sorting• Back-to-Front rendering order

necessary for correct alpha-blending

• For a head with hair this is very similar to rendering from inside to outside

• Use static index buffer with inside to outside draw order

– Computed a preprocess time

– Sort connected components (hair strand patches) instead of individual triangles


Sorted Hair Rendering Scheme

• Pass 1 – opaque parts– Enable alpha test to only pass opaque pixels– Disable backface culling– Enable Z writes, set Z test to Less

• Pass 2 – transparent back-facing parts– Enable alpha test to pass all non-opaque pixels– Cull front-facing polygons– Disable Z writes, set Z test to Less

• Pass 3 – transparent front-facing parts– Enable alpha test to pass all non-opaque pixels– Cull back-facing polygons– Enable Z writes, set Z test to Less


Performance Tuning

• Use early Z culling extensively to save us from running expensive pixel shader

• Usually half the hair is hidden behind the head

– Draw head first• Early Z culling can’t be used when alpha

test is enabled!– Solution: Prime Z buffer with a very simple

shader that uses alpha test– Use Z testing instead of alpha testing in

subsequent passes for same effect• Early Z culling saves considerable fill

overhead!


Optimized Scheme: Part 1

Prime Z-Buffer with depth of opaque hair regions • Enable alpha test to only pass opaque pixels• Disable backface culling • Enable Z writes , set Z test to LESS• Disable color buffer writes• Use simple pixel shader that only returns alpha

• No benefits from early-Z culling in this pass, but shader is very cheap anyway



Render opaque regions• Start using full hair pixel shader• Disable backface culling• Disable Z writes• Set Z test to EQUAL

– Z test passes only for fragments that wrote to Z in pass 1, which are the opaque regions

• This and subsequent passes don’t require alpha testing and thus benefit from early-Z culling



Render transparent back-facing parts

• Cull front-facing polygons• Disable Z writes

– Z order isn’t necessarily correct

• Set Z test to LESS


Optimizing Scheme: Pass 4

Render transparent front-facing parts

• Cull back-facing polygons• Enable Z writes• Set Z test to LESS

• Enabling Z writes prevents incorrect depth order at expense of possibly culling too much

• Covers up potential depth order artifacts of previous pass


Demo


Pros and Cons Of This Approach

Pros:• Low geometric complexity

– Reduced load on vertex engine

– Makes depth sorting faster

• Easy fall-back for lower-end hardware

Cons:• Sorting scheme assumes little to no animation

in hair model– Pony tails need to be handled seperately

– You could sort at runtime to overcome this

• Not suitable for all hair styles


Hair Rendering Summary

• Polygonal hair model• Hair lighting• Simple approximate depth-sorting

scheme• Optimization Tips


Hair Rendering References

• J. Kajiya and T. Kay. Rendering fur with three dimensional textures. In SIGGRAPH 89 Conference Proceedings, pp. 271-280, 1989.

• Stephen R. Marschner, Henrik Wann Jensen, Mike Cammarano, Steve Worley, and Pat Hanrahan, Light Scattering from Human Hair Fibers. In Proceedings of SIGGRAPH 2003.

• SIGGRAPH 2003 Hair Rendering Course Notes


Putting it All TogetherRuby: The Double Cross


Summary

• Efficient and Scalable techniques for rendering characters

• Texture-space lighting for simulating subsurface scattering in skin

• Polygonal hair strand modeling and rendering

• Early-Z optimizations


Thank you!

• Dave Gosselin• Thorsten Scheuermann• Pedro Sander• Chris Brennan


Questions?

Chris Oat [email protected]

Chris Oat ATI Research, Inc. Advanced Character Rendering.

Documents

skin slide

rendering techniques

texture space blur

offscreen texture

diffusion slide

texture coordinates

chris oat ati research

texcoord skin