Subodh Kumar Dept of Computer Sc. & Engg. IIT Delhisubodh/courses/CSL781/pdf...Subodh Kumar Dept of Computer Sc. & Engg. IIT Delhi Geometry Shader Graphics Pipeline 2 Vertex Vertex

Computer Graphics

Subodh Kumar

Dept of Computer Sc. & Engg. IIT Delhi

Geometry Shader

Graphics Pipeline

2

Vertex

Vertex Shader

Framebuffer

Clip & Setup

Rasterize Fragment Shader

Raster OPs

Primitive Assembly

Texture

Blend

Picture

Connectivity Textures

Tessellation

✔ ✔✔

✔ ✔

We have talked about the basic components of the graphics pipeline. Tessellation, Primitive assembly, and Geometry shader are left for you to explore. We will skip their details in this course.

We will discuss how to incorporate some of the strengths of ray-tracing into this pipeline: shadows, reflections, transparency.

We will need some additional functionality to support these. Let’s talk about the stage between the frame-buffer memory and the fragment shader (FS). We know FS produces color and depth per sample for the frame-buffer for display or for the next pass of rendering. (The depth/color values may later be filtered in case of multisampling. Find out how to set up multi-sampling for off-screen frame-buffer for multi-pass rendering.)

However, the depth produced by shader is not necessarily written directly to the frame-buffer. Additional operations are defined on the output, we will see. One question to ponder: why not perform that operation in the FS itself?

In addition, we will see FS can produce more data (stencil and multiple colors for blending or multiple frame buffers like stereo). These post-FS operations, also called ‘raster operations’ allow us to implement effects like shadows and transparency, which the basic pipeline does not support. Recall shadow and transparency require some ‘global illumination.’ They cannot be computed only with properties of the fragment (color, normal) and globals (light, camera). We will talk about shadows in this unit and transparency in the next one.

Raster Operations

! Output is always frame-buffer ! Input may include frame-buffer data

! Fixed function hardware close to the frame buffer memory can access it efficiently

! Bringing that into the shader induces large latency ! Input include FS output

! Z ! R, G, B, A ! Stencil: a constant number of bits per pixel

! Inter-operate: ! Effect Color write based on A value ! Effect Color write based on stencil ! Effect Z write based on stencil

Raster operation are controlled by the OpenGL state. The programmer simply says: Apply function number, say “21,” on color values. (OpenGL has constants with more meaningful names than number 21, of course.) We will talk about some of these function. We will refer to the frame-buffer data for a pixel as FB and the fragment shader output as FS.

Z/Color-operations

! Conditional writes ! Write masks: write if mask is 0 (or vice versa)

! e.g., glDepthMask(GL_TRUE); glColorMask(.); glStencilMask(.) ! Masks can be per component: R, G, B, Z, ..

! Stencil is mainly used for complex per-pixel masks ! Also see: discard statement in glsl

! Window “Clip” ! glScissor(x, y, width, height)

! Color modify supported ! Logic Ops

! glEnable(GL_COLOR_LOGIC_OP); glLogicOp(GL_XOR); ! FB.color = FB.color XOR FS.color

! Blend ! FB.color = a*FB.color + b*FS.color

Look up the functions mentioned on this slide.

Depth

! Z is just another variable to control FB writes

! Usually has the depth value of the fragment ! Perspective-correct interpolation of (1/Z) ! Normalized to range 0..1 ! Other applications possible

glEnable(GL_Depth_TEST); // Use depth test glDepthFunc(GL_LEQUAL); // Write if FS.Z≤FB.Z

5

Color and Alpha

! Arbitrary fourth component per pixel ! Carry-over from days when there was a single

frame-buffer ! RGB + A ! A is ‘just a parameter’ for per-pixel operation

! Or, FB.G " FB.G*FS.A + FS.G

! Applies to each of <R, G, B> normally ! Any component can be masked out

! Alpha can be used as fragment ‘write mask’ ! glEnable(GL_ALPHA_TEST); // Use alpha test ! glAlphaFunc(GL_LEQUAL, 0.5); // write if FS.A≤0.5

Alpha value is often used to implement transparency, but it is a more general parameter. Only R,G,B is finally displayed on the screen.

Color and Alpha





! Alpha can be used as fragment ‘write mask’ ! glEnable(GL_ALPHA_TEST); // Use alpha test ! glAlphaFunc(GL_LEQUAL, 0.5); // write if FS.A≤0.5 But deprecated in favor of discard, since no FB read isn’t required

Color and Alpha





! Alpha can be used as fragment ‘write mask’ ! glEnable(GL_ALPHA_TEST); // Use alpha test ! glAlphaFunc(GL_LEQUAL, 0.5); // write if FS.A≤0.5 But deprecated in favor of discard, since no FB read isn’t required

Also see GL_SAMPLE_ALPHA_TO_COVERAGE

Stencil Buffer: S

! More general per pixel write enable

glEnable(GL_STENCIL_TEST);

// Stencil test:

glStencilFunc(GL_LESS, 1, 0x1);

glStencilOp(GL_REPLACE, GL_INCR, GL_INVERT);

Function, Ref, Mask

sfail, zfail, zpass

// Write fragment if FB.S&0x1 < 1

// FB.S = FB.S fails? Ref : FS.Z passes? (~FB.S) : (FB.S + 1) // FB.S passes ⇒ stencil test passes, // FS.Z passes ⇒ depth test passes for fragment

We will use the variable S to refer to FB stencil value. Although technically FS does produce a stencil value, it is not generated per fragment. It is the ref value in the most recent glStencilFunc/glStencilFuncSeparate call. That makes stencil different from Z, which is indeed per fragment. The number of bits in stencil is determined when the frame-buffer is created. Usually, an initial stencil pattern is written to FB before rendering to control, for example, which pixels to actually render. We will see an example that renders a mirror later. Look up the functions mentioned on the slide (and the next).

Stencil Buffer: S



// Stencil test:



Function, Ref, Mask

sfail, zfail, zpass



Stencil Buffer: S



// Stencil test:



Function, Ref, Mask

sfail, zfail, zpass



Also see: glStencilFuncSeparate

Stencil Buffer

! Like color/z buffer; one entry per pixel ! Usually a few bits commandeered from Z bits

! Stencil value also masks whether to render ! Render to Stencil

! Stencil operation does both void glStencilFunc (enum func, int ref, uint mask ) func: NEVER, ALWAYS, LESS, LEQUAL, EQUAL,

GEQUAL, GREATER, or NOTEQUAL void glStencilOp (enum sfail, enum zfail, enum zpass)

KEEP, ZERO, REPLACE, INCR, DECR, INVERT

glEnable(GL_STENCIL_TEST)

Shadow Volumes

We see an application of stencil to render shadow. Remember, a point (fragment) is in shadow if some other geometry lies between the fragment and the light source. There is another way to frame this. Each object creates a shadow region. If a fragment lies in this region, it is in shadow. This is a useful technique for situations when a few large occluders cast all the (important) shadows. The shadow region of a triangle, or an occluder that may comprise multiple triangles, is a volume in space, which we call a shadow volume. These volumes are constructed once for each occluder as long as the light position remains fixed. There is a separate volume for each occluder, for each light source.

Shadow Volumes

The shadow volume is an intersection of half-spaces. This animation shows how to construct a shadow volume — rather the polygons that bound that shadow volume — of an occluding triangle. One polygon is, of course, the triangle itself. It bounds the shadow volume. Three other polygons are formed by the edges of the triangle. The plane spanning the light-point and the edge bounds the shadow volume. Actually, not the entire plane but the part of it bounded by the triangle-edge it spans and the rays joining the light-point to the two ends of the edge, except these rays originate at the the edge’s end-points and not the light-point. It’s a half-plane (and infinite). We would want to render the half-plane, so we clip the half-plane by another plane, far away from the light-point, and turn the half-plane into a finite shadow-polygon. The three shadow-polygons so formed by the three edges, the occluding triangle itself, and the far-away cap (formed by the intersection of the rays with the far-away plane) together form a closed volume. Simply speaking, the vertices of the triangle and the cap form the shadow volume. As long as the fragment is not further away from the light than the cap, it’s containment within the volume is evidence of its being in shadow with respect to that light.

If the selected occluder is an object made of several triangles, rays need to be formed for vertices on the silhouette. Think about how to tell if an edge is on a silhouette.

Shadow Volumes

Shadow Volumes

Shadow Volumes

Stencil-based Shadow Volume Algorithm

1. Render the scene in shadow color ! Maybe black; we want the Z-buffer

2. For each light source ! Using the depth from step 1, and

shadow polygons, build stencil buffer, disabling scene-pixels in shadow ! Use pre-built shadow polygons

! Find silhouettes of occluders ! Find far away points on the rays to silhouettes ! Form the polygons, possibly triangulate

3. Render the scene again, with lighting ! Stencil buffer masks out shadowed areas

The shadow question is: Given the original scene and the shadow polygons, how do you determine — in the fragment shader — if the fragment is in shadow. This algorithm tells you how. Step 2 is key, and is described in a subsequent slide. It determines if the fragment at pixel coordinate (x, y), whose depth is now in the Z-buffer, is sandwiched between shadow polygons (rather its depth is between that of two shadow polygons). The third step is the real rendering of the scene, except it now has a stencil buffer to guide it if a fragment is in shadow or not. Shadowed fragments are simply discarded and not drawn. (The color values at those locations are left over from step 1.)

Front and Back Facing

12

E

Pf

Pb

P3

>90°

<90°

=90°

>90°

EP·nP > 0 ⇒ back-facing< 0 ⇒ front-facing< 0 ⇒ Silhouette

Outward-normal’s direction is not the only way to determine faced-ness. Sometime, normals are not provided, or they may be artificially created.

A triangle’s vertices can instead be oriented consistently. For example, they may be clock-wise (cw) when viewed from exterior. In that case, if they appear cw on screen, the eye is in the exterior.

Using outward normal’s orientation to determine faced-ness of points on a solid object (yellow figure)

nPb

A slight detour related to containment in closed solids.

We can tell something about the bounding surface of a solid from its orientation towards us. Suppose the normals of a surface that bounds a volume are consistently oriented towards the exterior of the volume. We can classify a point as front facing if the normal is pointing towards the eye and the eye is in the exterior. For points on a surface fully enclosing a volume, a back-facing point would be occluded by some front-facing one. This is a quick test to determine if a fragment, or an entire triangle, need not be drawn at all. Note that the decision is only local: P3 is front-facing and still hidden by another back-facing and front-facing pieces. No back-facing part is ever visible though.

Front and Back Facing

12

E

Pf

Pb

P3

>90°

<90°

=90°

>90°

EP·nP > 0 ⇒ back-facing< 0 ⇒ front-facing< 0 ⇒ Silhouette

Outward-normal’s direction is not the only way to determine faced-ness. Sometime, normals are not provided, or they may be artificially created.

A triangle’s vertices can instead be oriented consistently. For example, they may be clock-wise (cw) when viewed from exterior. In that case, if they appear cw on screen, the eye is in the exterior.

Using outward normal’s orientation to determine faced-ness of points on a solid object (yellow figure)

OpenGL allows automatic discarding of back-facing primitives — It is called culling.

These tests are customizable — See glFrontFace(..), glCullFace(..), gl_FrontFacing (in FS)

nPb

Stencil Generation

! Disable depth and color writes ! Set the stencil operation to increment on

depth pass (counting shadow boundaries in front of the object)

! Draw the shadow volumes front faces ! Enable back-face culling (i.e., front drawing)

! Set the stencil operation to decrement on depth pass

! Draw the shadow volume back faces ! Enable front-face culling

Back to step 2 of the shadow volume rendering. A fragment with depth=z is in shadow not if is sandwiched between any two shadow polygons but between a front-facing polygons of a volume and a back-facing one. Indeed, we may have multiple occluders and hence multiple shadow volumes. Only if the fragment is outside all volumes, is it to be lit.

The algorithm on this slide accomplishes this by counting in the stencil buffer. The Z-buffer is first initialized with the object’s depths. The shadow polygons are drawn next. Every time a front-facing shadow polygon is in front of z at any pixel, the stencil value increments at that pixel. Every time a back-facing shadow polygons is in front of z at a pixel, the stencil value decrements at that pixel. At the end, that leaves the number of front-facing (from the eye) shadow polygons between the eye and the scene fragment’s z. These belong to the shadow volumes that began but did not end before the object. Note that every time a ray enters a shadow volume at a front-facing polygon, it will exit it at a back facing polygon. If the ray encounters a scene object before that exit, that point of intersection is inside the volume.

Since stencil update function cannot be changed inside the fragment shader, two separate rendering passes are required.

This process is demonstrated in the next animation.

Stencil Shadow Volumes

p

p

f

p

Part of scene (in shadow)

Part of scene (not in shadow)

Occluder

Shadow Polygons are in red

Intersections along a ray correspond to the same

pixel coordinate

Consider pixels corresponding to rays a and b and see which fragments Z-pass (p) and which fail (f)

⇒ increment stencil⇒ decrement stencil⇒ neither, Z-fail

a b

Can something go wrong, though? Recall clipping. In particular, Z-clipping. Only the parts of geometry between the camera’s ‘Near’ and ‘Far’ planes are actually retained. What if a shadow polygons is clipped and as a result it does not update the stencil operation? That will lead to the wrong count, and possibly the wrong result. And what if Eye is itself in the shadow volume? That will also lead to the wrong result. That is fixed by a slight adjustment to the algorithm, in the next slide.


p

p

f

p

What if Eye is in shadow?



Occluder



pixel coordinate



a b


p

p

f

p




Occluder



pixel coordinate



a b


p

p

f

p

p




Occluder



pixel coordinate



a b


p

p

f

p

p

f




Occluder



pixel coordinate


⇒ decrement


a b

Reverse Stencil

! Disable depth and color writes ! Set the stencil operation to increment on

depth fail (counting shadow boundaries behind the object)

! Render the shadow volumes back faces ! Set the stencil operation to decrement on

depth fail ! Render the shadow volumes front faces

What has changed? We now draw the back-faces of the shadow volumes first, but increment the stencil on failing, meaning count back-faces behind the scene-object at each pixel. In the next pass, we reduce this count by the number of front-faces behind the object. If there were more back-faces, it means there is a shadow volume that the object is inside. It does not matter whether Eye is in the shadow volume, as we do not care about shadow polygons between the eye and the object.

In addition, we do not care much about near clipping causing a problem, only far clipping. Usually, far-clipping is less of a problem than near-clipping. Near-clipping would always be a problem if Eye is inside a shadow volume (although stencil can be initialized differently in the previous algorithm to handle that case; think about that).

Animation for this algorithm in the next slide.


a b

p

p

f

p⇒ increment stencil⇒ decrement stencil⇒ neither, Z-pass




Does this solve the problems of shadow-polygons being clipped? How about not clipping these polygons by ensuring that the shadow polygons do not veer outside the view-frustum, they are inside or straddle the boundaries. This means the near and far planes become parts of the shadow volume. Clipping must ensure that these just at the boundary polygons are retained. (One way is keep the shadow volume strictly inside the frustum.)

A related question is why Z-clip in the first place? Recall this has to do with the precision of Z-buffer. Given a fixed number of bits for Z in the frame-buffer, only a fixed number of depths can be represented, no matter how large the range. This means the representable values will be further apart if the range is large, i.e., Z-precision (i.e., Z-value resolution) will be lower. This causes fragments right behind each other to not resolve correctly and sometime switch order. (This is referred to as Z-fighting.)

Shadow volumes are reasonable, but require significant computation. By default, the algorithm must be repeated for each light. And there are many passes of rendering. That is expensive. There is a more efficient algorithm (but it has different problems) based on shadow maps that we will discuss next. They use the texture mechanism.


a b

p

p

f

p

p

⇒ increment stencil⇒ decrement stencil⇒ neither, Z-pass





a b

p

p

f

p

p

f






a b

p

p

f

p

p

f





Shadow Map

Scene

pixel

The shadow map algorithm more directly answers, is there another object between the light and the fragment point (for each light source). Recall shadow rays from Ray tracing, which answer that exact question. How do you incorporate the shadow ray in rasterization pipeline? We have done this before.

After all, a rendering pass of this pipeline amounts to casting one ray per pixel (or per sample). However, we would not render from the point of view of each fragment to find just one pixel corresponding to the shadow ray.

On the other hand, all shadow rays (from different fragments) converge on the light. If a ray from light-source in the direction of the fragment hits that fragment first, there is no shadow from that light. Shadow-map accomplishes this.

Shadow map is simply the rendering of the scene from the point of view of the light source (one per light source). This is precomputed every time the light moves with respect to the objects, meaning shadow shape/position changes. If only the viewer camera (Eye) moves, no re-computation is necessary. The remaining questions to answer are:1. What are rendering parameters for shadow map generation?2. From which shadow map pixel to look up depth value (and how)?

Shadow Map

Scene

pixel

fragment

Shadow Map

Scene

pixel

fragment

Shadow Map

Scene

pixel

fragment

Some other object

Shadow Map

Scene

pixel

fragment

Some other object

Rendering

Shadow Map

Scene

pixel

fragment

Some other object

Z-only

Rendering

Shadow Map

Scene

pixel

fragment

Some other object

Z-only

Rendering

Z-stored

Shadow Map

Scene

pixel

fragment

Some other object

QueryZ-only

Rendering

Z-stored

Shadow Map

Scene

pixel

fragment

Some other object

Query

in shadow

Z-only

Rendering

Z-stored

Shadow Map

Scene

pixel

fragment

Some other object

Query

Image = shadow map

in shadow

Z-only

Rendering

Z-stored

Shadow Map Generation

! Render to texture ! Depth only (see glDrawBuffer())

! Light-camera position is that of light ! Direction:

! Image should cover rays in all directions ! Cube map: use six directions towards faces of a cube

centered on the camera ! This also suggests the field of view: 90°

! Near and far? ! Clipping could be a problem, choose so occluders are

not clipped ! View-port/Framebuffer size?

! ‘Large enough’18

What would you change for directional light?









For render to texture, review glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, ..









For render to texture, review glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, ..

Also: geometry shader has a feature that can render to all faces in the same pass:

See variable gl_layer

Shadow Map Look-up

! Find x,y to look up in shadow map ! Unproject fragment (i, j) to Light space

! Fragment depth in camera space is known ! Project into light’s image space

! Let the projected values be (x, y, z)

! x and y may not be integers ! If shadow map is in texture, use texture

interpolation ! Fetch Depth buffer/texture (SM.z)

! Compare projected z with stored z ! SM.z < z ⇒ fragment in shadow

The algorithm is simple enough. How do you un-project a fragment? If the fragment is in clip space, multiplication with (P.MV)-1 will fetch world space, which can then be transformed to light’s pseudo-camera. Or, instead, the pseudo-camera parameters can be passed as Uniforms. Now, the world-space coordinates can be transformed to real camera as well as pseudo-camera in vertex shader and interpolated in fragment shader. This directly provides light-camera’s (x,y,z) in the fragment shader. (See textureProj for perspective divide in texture look up.)

(x, y, z) can also be seen as a direction from the light position (which is 0,0,0 in pseudo-camera coordinate system). It can be used directly in cube-map look up.

What can go wrong?1. The shadowing object could be clipped out2. The SM.z == z for a lit fragment. Precision could be a problem3. How many texels in shadow-map are enough? Is shadow map aliased?

Shadow Map Aliasing

The easiest solution for occluder clipping is to ensure near = nearest and far = furthest occluder.

A common solution for Z precision problems is to offset the fragment’s Z value used to compare with SM.Z. Moving it closer to light would give the right result for objects not in shadow, as well as objects well behind its shadow-causing occluder (with respect to the light). How much to offset? Even a small offset could, for example, be a problem where the shadow of an object is sometimes touching the object (like in the image above), or near the silhouette of object (from the light’s camera). Sometimes using n.l to increase bias where the angle is high helps. Another solution may be to use more Z precision in harder areas, but this remains an open problem.

Increasing the shadow-buffer size should reduce aliasing artifacts, as shown in this picture. However, there can always be objects in the shadow map that are far from the light, leading to a small number of projected texels in the shadow-map. If these objects are near the viewer camera, these texels get magnified. Texture interpolation helps only somewhat.

One solution is known as cascading shadow maps. Where occluders (i.e, scene geometry) are divided into 4-5 sets, and a different shadow map is created for each set, with parameters tuned for that set. But the resolution mismatch is not just about the distance of the object from the light but also its orientation.

A more complex method known as perspective shadow maps tries to do it all within the same map, but the resolution is not constant everywhere. The intuition behind the idea is that objects that may be perspectively small in the light’s camera, be scaled up during the shadow-map rendering pass. This allows them to occupy more texels in the shadow-map texture.

Shadow Map Aliasing

single texel i.e., single shadow map pixel

Shadow Map Aliasing

Filtering

Perspective Scaling in Shadow Map

21

αβ

V

L

dp ~ ds L cos β/(V cos α)

ds

dp

Pixel size

Object fragment

n

View vector

Shadow-map projection size

(Light vector)

dp / ds = V.n |L| / (n.L |V|)

The details of the algorithm are skipped here but the main idea is demonstrated in this slide. if dp/ds > 1, shadow-map resolution may be wasted. If dp/ds < 1, resolution is insufficient. Of course, this arithmetic is not independent of the viewer’s camera. (Alternatively, one can try to scale objects in the shadow map independent of the view-space, but that could lead to unnecessary scale-down of some objects.) Can you think of some ways to improve shadow maps?

Subodh Kumar Dept of Computer Sc. & Engg. IIT Delhisubodh/courses/CSL781/pdf...Subodh Kumar Dept of Computer Sc. & Engg. IIT Delhi Geometry Shader Graphics Pipeline 2 Vertex Vertex

Documents