Underlying Technologies Part Two: Software

Underlying TechnologiesUnderlying TechnologiesPart Two: SoftwarePart Two: Software

Mark GreenMark Green

School of Creative MediaSchool of Creative Media

IntroductionIntroduction

Software not as easy as hardware:Software not as easy as hardware: wide range of software techniques, hard to wide range of software techniques, hard to

classify like hardwareclassify like hardware several components that need to work several components that need to work

together, hard to know where to starttogether, hard to know where to start wide range of hardware configurations, not as wide range of hardware configurations, not as

simple as 2D softwaresimple as 2D software

Hardware ConfigurationsHardware Configurations

In 2D have a standard hardware In 2D have a standard hardware configuration:configuration: input: keyboard and mouseinput: keyboard and mouse output: single 2D displayoutput: single 2D display

with 3D can have many configurations:with 3D can have many configurations: HMDHMD projectionprojection single screensingle screen

Hardware ConfigurationsHardware Configurations

Want to produce an application once, not Want to produce an application once, not once for every possible hardware once for every possible hardware configurationconfiguration

software needs to be more adaptable, software needs to be more adaptable, change based on hardware configurationchange based on hardware configuration

complicates the development of support complicates the development of support softwaresoftware

Range of Software TechniquesRange of Software Techniques

Want our software to be very efficient: Want our software to be very efficient: reduce latency, high update ratesreduce latency, high update rates

some applications can be quite large, some applications can be quite large, need to efficiently organize dataneed to efficiently organize data

all of this complicates VR software, too all of this complicates VR software, too many things to consider, hard to know many things to consider, hard to know where to startwhere to start

ComponentsComponents

What are the main components of a VR What are the main components of a VR application?application? 3D Objects: geometry and appearance, but 3D Objects: geometry and appearance, but

may also want sound and forcemay also want sound and force Behavior: the objects need to be able to do Behavior: the objects need to be able to do

things, move and reactthings, move and react Interaction: users want to interact with the Interaction: users want to interact with the

application, manipulate the objectsapplication, manipulate the objects

3D Objects3D Objects

Need object geometry, object’s shape, Need object geometry, object’s shape, basis for everything else, called modelbasis for everything else, called model

polygons used for geometry, sometimes polygons used for geometry, sometimes restricted to trianglesrestricted to triangles

different from animation, free form different from animation, free form surfaces based on sophisticated mathsurfaces based on sophisticated math

need speed, so restricted to polygonsneed speed, so restricted to polygons

3D Objects3D Objects

Where does geometry come from?Where does geometry come from? Really depends on the applicationReally depends on the application Could use a text editor to enter all the Could use a text editor to enter all the

polygon vertices, some people actually do polygon vertices, some people actually do this!this!

Could use a program, for example Could use a program, for example OpenGL, works for small modelsOpenGL, works for small models

3D Objects3D Objects

Use a 3D modeling or animation programUse a 3D modeling or animation program for non-programmers this is the easiest for non-programmers this is the easiest

way, but it takes time to develop modeling way, but it takes time to develop modeling skillsskills

also many different program and file also many different program and file formatsformats

want a modeler that does a good job of want a modeler that does a good job of polygons, not all modelers are good at thispolygons, not all modelers are good at this

3D Objects3D Objects

Another source of objects is scientific and Another source of objects is scientific and engineering computationsengineering computations

can be easy to convert to polygons, can be easy to convert to polygons, already have position dataalready have position data

other types of data can also be converted other types of data can also be converted into geometry, but this can be more into geometry, but this can be more difficultdifficult

3D Objects3D Objects

Also need to consider appearance:Also need to consider appearance: colour of the objectcolour of the object how it reflects lighthow it reflects light transparencytransparency texturetexture

can be done with modeler, or later in the can be done with modeler, or later in the VR programVR program

BehaviorBehavior

How should objects behave?How should objects behave? What happens when the user hits an object?What happens when the user hits an object? What happens when an object hits another What happens when an object hits another

object?object? Can objects move around the environment?Can objects move around the environment?

Each object could have a range of Each object could have a range of behaviors, react differently to different behaviors, react differently to different events in the environmentevents in the environment

BehaviorBehavior

Behavior is harder than modelingBehavior is harder than modeling animation programs can be useful, but not animation programs can be useful, but not

alwaysalways animation is quite different:animation is quite different:

animator is in complete control, knows what’s animator is in complete control, knows what’s happening all of the timehappening all of the time

in VR the user is in control, can interrupt or in VR the user is in control, can interrupt or mess up any animationmess up any animation

BehaviorBehavior

Short animations (less than 5 seconds) Short animations (less than 5 seconds) can be useful, basic motion unitscan be useful, basic motion units

other types of behaviors must be other types of behaviors must be programmed or scriptedprogrammed or scripted

more flexible, can respond to the events more flexible, can respond to the events that occur in the environmentthat occur in the environment

easier to combine, objects can do two easier to combine, objects can do two things at same timethings at same time

InteractionInteraction

Users want to interact with the environmentUsers want to interact with the environment pick up objects and move them aroundpick up objects and move them around very different from 2D interactionvery different from 2D interaction much more freedom, more direct interactionmuch more freedom, more direct interaction still exploring the design space, not stable still exploring the design space, not stable

like 2D interactionlike 2D interaction still working on standard techniquesstill working on standard techniques

Application StructureApplication Structure

look at application structurelook at application structure provides a framework for discussing provides a framework for discussing

various software technologiesvarious software technologies divide an application into various divide an application into various

components, and then look at the components, and then look at the components individuallycomponents individually

Application StructureApplication Structure

Model

ApplicationProcessing

Output Devices

InputDevices

Model Traversal

InputProcessing

Application StructureApplication Structure

Model: representation of objects in the Model: representation of objects in the environment, geometry and behaviorenvironment, geometry and behavior

Traversal: convert the model into Traversal: convert the model into graphical, sound, force, etc outputgraphical, sound, force, etc output

Input Processing: determine user’s Input Processing: determine user’s intentions, act on other part of applicationintentions, act on other part of application

application processing: non-VR parts of application processing: non-VR parts of the applicationthe application

Interaction LoopInteraction Loop

Logically the program consists of a loop Logically the program consists of a loop that samples the user, performs that samples the user, performs computations and traverses the modelcomputations and traverses the model

Input processing

Computation

Model Traversal

ModelModel

Contains the information required to Contains the information required to display the environment:display the environment: geometry, sound, forcegeometry, sound, force behaviorbehavior

the graphical part is the most developed, the graphical part is the most developed, so concentrate on itso concentrate on it

try to position sound and force within this try to position sound and force within this modelmodel

GeometryGeometry

This is what we know the bestThis is what we know the best need to have a graphical representation of need to have a graphical representation of

objects in the environment:objects in the environment: accurate shape representationaccurate shape representation ease of modelingease of modeling efficient displayefficient display integrates with behaviorintegrates with behavior

Scene GraphScene Graph

Main technique for structuring the modelMain technique for structuring the model based on hierarchical structure, divide the based on hierarchical structure, divide the

object into parts or componentsobject into parts or components simplifies the modeling task, work on one simplifies the modeling task, work on one

part at a timepart at a time easy to modify the individual partseasy to modify the individual parts add behaviors, sound, force, etc to the add behaviors, sound, force, etc to the

modelmodel

Scene GraphScene Graph

car

Wheel Wheel Wheel Wheel Body

Scene GraphScene Graph

Individual units are called nodes:Individual units are called nodes: shapes: polygons, meshes, cubes, etcshapes: polygons, meshes, cubes, etc transformations: position the nodes in spacetransformations: position the nodes in space material: colour and texture of objectsmaterial: colour and texture of objects grouping: collecting nodes together as a grouping: collecting nodes together as a

single objectsingle object soundssounds behaviorbehavior

Scene GraphScene Graph

Many different scene graph architectures, Many different scene graph architectures, will look at one in more detail laterwill look at one in more detail later

differences:differences: scene graph for whole VE Vs. one per objectscene graph for whole VE Vs. one per object types of nodes in the scene graphtypes of nodes in the scene graph ease of modification, static Vs dynamicease of modification, static Vs dynamic

BehaviorBehavior

Harder to deal with than geometryHarder to deal with than geometry simple motions aren’t too bad, but much simple motions aren’t too bad, but much

harder to get sophisticated behaviorharder to get sophisticated behavior the general solution now is to write code, the general solution now is to write code,

okay for programmersokay for programmers would like to have a higher level approach would like to have a higher level approach

for non-programmersfor non-programmers

BehaviorBehavior

Problem: want objects to respond to Problem: want objects to respond to events in the environmentevents in the environment

can have some motions that are simple can have some motions that are simple animations, but most of the motions need animations, but most of the motions need some knowledge of the environmentsome knowledge of the environment

example: an object moving through the example: an object moving through the environment must be aware of other environment must be aware of other objects so it doesn’t walk through themobjects so it doesn’t walk through them

BehaviorBehavior

Some simple motions produced by Some simple motions produced by animating transformation nodesanimating transformation nodes

animation variables used to control animation variables used to control transformation parameters, example: transformation parameters, example: rotation or translationrotation or translation

could import animations, use some form of could import animations, use some form of keyframing package to produce the motionkeyframing package to produce the motion

BehaviorBehavior

Simple motions could be triggered by Simple motions could be triggered by events in the environmentevents in the environment

example: collision detection, if an object is example: collision detection, if an object is moving through the environment and a moving through the environment and a collision detected it changes directioncollision detected it changes direction

hard to come up with good trigger hard to come up with good trigger conditions, a few obvious ones, but not conditions, a few obvious ones, but not much after thatmuch after that

BehaviorBehavior

Another approach is to use a general Another approach is to use a general motion modelmotion model

best example of this is physics, try to best example of this is physics, try to simulate real physics in the environmentsimulate real physics in the environment

this gives a number of natural motions, this gives a number of natural motions, and objects respond to the environmentand objects respond to the environment

works well in some environment, but has works well in some environment, but has some problemssome problems

BehaviorBehavior

One problem is the complexity of the One problem is the complexity of the mathematics, often need to simplifymathematics, often need to simplify

computations can be a problem, computations can be a problem, particularly for complex objectsparticularly for complex objects

hard to control, need to know forces and hard to control, need to know forces and torque's that produce the desired motions, torque's that produce the desired motions, can be very hard to determinecan be very hard to determine

BehaviorBehavior

Some attempts to produce general motion Some attempts to produce general motion controllerscontrollers

maybe the eventual solution, but nothing maybe the eventual solution, but nothing much nowmuch now

use of scripting languages, can add some use of scripting languages, can add some program control to the scene graph, but program control to the scene graph, but not full programmingnot full programming

Model TraversalModel Traversal

The process of going through the model The process of going through the model and generating the information to be and generating the information to be displayeddisplayed

this is part software and part hardware, this is part software and part hardware, look through the entire processlook through the entire process

hardware parts have implications for how hardware parts have implications for how we build models and the graphics we build models and the graphics techniques usedtechniques used

A Simple ModelA Simple Model

A simplified model of the display process, A simplified model of the display process, explains hardware performanceexplains hardware performance

traverse geometry Pixel

Model Screen

TraverseTraverse

Traverse the model, determine objects to Traverse the model, determine objects to be drawn, send to graphics hardwarebe drawn, send to graphics hardware

usually combination software/hardware, usually combination software/hardware, depends on CPU and bus speeddepends on CPU and bus speed

early systems were hardware, didn’t scale early systems were hardware, didn’t scale wellwell

many software techniques for culling many software techniques for culling modelsmodels

GeometryGeometry

Geometrical computations on polygons: Geometrical computations on polygons: transformations and lightingtransformations and lighting

floating point intensivefloating point intensive divide polygons into fragments, screen divide polygons into fragments, screen

aligned trapezoidaligned trapezoid time proportional to number of polygons time proportional to number of polygons

and verticesand vertices

PixelPixel

Fill fragments, colour interpolation, texture Fill fragments, colour interpolation, texture mapping, transparency, hidden surfacemapping, transparency, hidden surface

all the per pixel computations, time all the per pixel computations, time depends on number of pixels, also colour depends on number of pixels, also colour depth on low end displaysdepth on low end displays

scalable operations, can add more scalable operations, can add more processors for more speedprocessors for more speed

Design ConsiderationsDesign Considerations

Any of the stages could block, depend on Any of the stages could block, depend on display mixdisplay mix

lots of small polygons cause traversal and lots of small polygons cause traversal and geometry stages to blockgeometry stages to block

large polygons cause pixel stage to blocklarge polygons cause pixel stage to block can use buffers to reduce blockingcan use buffers to reduce blocking a careful balancing processa careful balancing process

Design ConsiderationsDesign Considerations

CPU/Image Generator trade-offCPU/Image Generator trade-off cheap boards just do pixel stage, use CPU cheap boards just do pixel stage, use CPU

for everything else:for everything else: scales with CPU speedscales with CPU speed large polygons and texture mappinglarge polygons and texture mapping

moving geometry onto board increases moving geometry onto board increases performance, trend in low cost displaysperformance, trend in low cost displays

PC Hardware EvolutionPC Hardware Evolution

Start with CPU doing most of the workStart with CPU doing most of the work Graphics board:Graphics board:

image memoryimage memory fill and hidden surfacefill and hidden surface texture mappingtexture mapping

graphics speed determined by CPU, graphics speed determined by CPU, limited assistance from graphics cardlimited assistance from graphics card

Graphics Card MemoryGraphics Card Memory

Memory used for three things:Memory used for three things: image storeimage store hidden surface (z buffer)hidden surface (z buffer) texture mapstexture maps

texture can be stored in main memory with texture can be stored in main memory with AGP, but this isn’t most efficientAGP, but this isn’t most efficient

better to have texture memory on boardbetter to have texture memory on board

Image MemoryImage Memory

Amount depends on image sizeAmount depends on image size double buffer, two copies of image double buffer, two copies of image

memorymemory front buffer: image displayed on screenfront buffer: image displayed on screen back buffer: where the next image is back buffer: where the next image is

constructedconstructed can construct next image while the current can construct next image while the current

image is displayed, better image quality image is displayed, better image quality and faster displayand faster display

Z BufferZ Buffer

Used for hidden surface removalUsed for hidden surface removal z buffer: one value for each pixel, distance z buffer: one value for each pixel, distance

from eye to object drawn at that pixelfrom eye to object drawn at that pixel when drawing a pixel, compare depth of when drawing a pixel, compare depth of

pixel to z bufferpixel to z buffer if closer draw pixel and update z bufferif closer draw pixel and update z buffer otherwise, ignore the pixelotherwise, ignore the pixel

Graphics AccelerationGraphics Acceleration

Next step: move pixel operations to Next step: move pixel operations to graphics cardgraphics card

fill and z buffer 3D trianglesfill and z buffer 3D triangles add smooth shading and texture mappingadd smooth shading and texture mapping CPU does traversal and geometry CPU does traversal and geometry

processingprocessing

Graphics AccelerationGraphics Acceleration

Next step: move geometry processing to Next step: move geometry processing to graphics cardgraphics card

CPU traverses model, send graphics CPU traverses model, send graphics primitives to display cardprimitives to display card

all transformations and lighting done on all transformations and lighting done on graphics cardgraphics card

less dependence on CPUless dependence on CPU

Current TrendsCurrent Trends

Pixel processing (Geforce 2): a program Pixel processing (Geforce 2): a program that processes each pixel, control lighting that processes each pixel, control lighting and other effectsand other effects

support for multiple textures, etcsupport for multiple textures, etc Vertex processing (Geforce 3): a program Vertex processing (Geforce 3): a program

processes each vertex, can change processes each vertex, can change geometry at display timegeometry at display time

real-time deformations and IKAreal-time deformations and IKA

Current TrendsCurrent Trends

Move to programming all aspects of the Move to programming all aspects of the graphics card (3DLabs VP series)graphics card (3DLabs VP series)

Also making programming more Also making programming more sophisticated, closer to CPUsophisticated, closer to CPU

Floating point textures and image memory Floating point textures and image memory (ATI and 3DLabs VP series)(ATI and 3DLabs VP series)

Higher dynamic range -> better image Higher dynamic range -> better image quality, better for programmingquality, better for programming

Input ProcessingInput Processing

Users need to interact with the Users need to interact with the environmentenvironment

they have a set of input devices, have they have a set of input devices, have position and orientation informationposition and orientation information

need to translate this into their intentionsneed to translate this into their intentions Interaction Technique (IT): basic unit of Interaction Technique (IT): basic unit of

interaction, converts user input into interaction, converts user input into something the application understandssomething the application understands

Input ProcessingInput Processing

Each IT address a particular interaction Each IT address a particular interaction task, something that the user wants to dotask, something that the user wants to do

look at interaction tasks first, then talk a look at interaction tasks first, then talk a little bit about ITs for themlittle bit about ITs for them

interaction tasks divide into two groups:interaction tasks divide into two groups: application independent: required by many application independent: required by many

different applicationsdifferent applications application dependentapplication dependent

Interaction TasksInteraction Tasks

Mainly look at application independent Mainly look at application independent interaction tasksinteraction tasks

the main ones are:the main ones are: navigationnavigation selectionselection manipulationmanipulation combinationcombination

NavigationNavigation

Need to get from one part of the Need to get from one part of the environment to anotherenvironment to another

two types:two types: locallocal globalglobal

with local navigation the destination is with local navigation the destination is within view, move on continuous path from within view, move on continuous path from current location to destinationcurrent location to destination

NavigationNavigation

In global navigation the destination is In global navigation the destination is remote, can’t move directly to itremote, can’t move directly to it

need some way of locating destination, need some way of locating destination, and then some way of jumping to itand then some way of jumping to it

variation: browsing / exploration don’t have variation: browsing / exploration don’t have a destination, exploring the environment or a destination, exploring the environment or looking for particular objectslooking for particular objects

SelectionSelection

The selection tasks involves selecting The selection tasks involves selecting somethingsomething

there are several variations, depending there are several variations, depending upon what’s being selected:upon what’s being selected: list or command selectionlist or command selection object selectionobject selection location selectionlocation selection

SelectionSelection

List selection: a pre-defined list of things to List selection: a pre-defined list of things to select fromselect from

example: the commands on a menuexample: the commands on a menu need to present the list, and the user need to present the list, and the user

selects one item from the listselects one item from the list object selection: number of objects not object selection: number of objects not

pre-defined, created by the user, changes pre-defined, created by the user, changes in size as the program runsin size as the program runs

SelectionSelection

For object selection can’t just present a list For object selection can’t just present a list of objects to be selected fromof objects to be selected from

location selection: selecting a point in location selection: selecting a point in space, may be used as location of object, space, may be used as location of object, or as part of an object’s shapeor as part of an object’s shape

can’t see a point in empty space, so this is can’t see a point in empty space, so this is harder than the previous twoharder than the previous two

ManipulationManipulation

Standard set of object manipulations, Standard set of object manipulations, change position, size and orientationchange position, size and orientation

grab the object and move itgrab the object and move it could also have deformations that change could also have deformations that change

the object’s shapethe object’s shape hard to get general techniques beyond the hard to get general techniques beyond the

few standard onesfew standard ones

CombinationCombination

Take two or more objects and put them Take two or more objects and put them together to form a new objecttogether to form a new object

need to match up the shapes exactly, so need to match up the shapes exactly, so they join in the right waythey join in the right way

difficult to do unaided, usually need some difficult to do unaided, usually need some form of constraint to simplify the processform of constraint to simplify the process

Application Dependent TasksApplication Dependent Tasks

Usually involve the application dataUsually involve the application data ways of controlling the view of the dataways of controlling the view of the data ways of manipulating the dataways of manipulating the data example: a CAD or animation program will example: a CAD or animation program will

have a different set of manipulations than have a different set of manipulations than a network visualization programa network visualization program

Interaction TechniquesInteraction Techniques

Not a well established set of techniques, Not a well established set of techniques, yetyet

depend on input devices and styledepend on input devices and style example: a fixed range device (tracker) example: a fixed range device (tracker)

sometimes works best with pointing at sometimes works best with pointing at objects, while a puck or joystick might objects, while a puck or joystick might work better with grabbingwork better with grabbing

need to try different combinationsneed to try different combinations

Interaction TechniquesInteraction Techniques

Some problems encountered:Some problems encountered: distance: objects too far away to grabdistance: objects too far away to grab feedback: how do you highlight the object that feedback: how do you highlight the object that

has been selected?has been selected? Object to be selected may be hidden by other Object to be selected may be hidden by other

objectsobjects object density may make selection and object density may make selection and

manipulation difficultmanipulation difficult

Application ProcessingApplication Processing

Not much to say hereNot much to say here some applications have a considerable some applications have a considerable

amount of processing, computation based amount of processing, computation based on user inputon user input

don’t want this to effect application latencydon’t want this to effect application latency need to control resources devoted to need to control resources devoted to

computation, use other processorscomputation, use other processors

Making it run rightMaking it run right

Now that we have an idea of what’s Now that we have an idea of what’s involved, how do we put it all togetherinvolved, how do we put it all together

want to have low system latency, get want to have low system latency, get images on the screen as fast as possibleimages on the screen as fast as possible

don’t want to wait for anythingdon’t want to wait for anything divide the application into components that divide the application into components that

execute separatelyexecute separately

Decoupled Simulation ModelDecoupled Simulation Model

Separate process for application Separate process for application computations, this is easycomputations, this is easy

separate process for expensive input separate process for expensive input devices, trackers that need lots of devices, trackers that need lots of computation or have latencycomputation or have latency

a separate process for input processing a separate process for input processing and displayand display

maybe a separate process for modelmaybe a separate process for model

Application StructureApplication Structure

Model

ApplicationProcessing

Output Devices

InputDevices

Model Traversal

InputProcessing

Decoupled Simulation ModelDecoupled Simulation Model

Each process can run at its own rateEach process can run at its own rate display process runs as fast as possible, display process runs as fast as possible,

doesn’t wait for other processesdoesn’t wait for other processes uses most recent value from input devices uses most recent value from input devices

and application computationand application computation reduces system latencyreduces system latency