LiDAR Data Processing and Classification

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

Automated Image and Point Cloud Interpretation

Techniques and Applications

Translating Human Visual Interpretations into Algorithms in Spectral, Morphological, and Contextual

Feature Extraction

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

DSM (Digital Surface Model) nDSM (Normalized Digital Surface Model)

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Slope (Perspective) Slope (Aerial)

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Curvature (Perspective)

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Curvature (Perspective)

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Segmentation using nDSM, Curvature

Vector Segmentation (Forest Patchabove removed by Permieter and Area)

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

ERDAS - ObjectveRoad Classification Process

Planar Fitting

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Context Based Island Detection (eCognition)

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.

Mike Bularz

Prof. Robert J. Hasenstab

GEOG – Independent Study

CONTEXT – POINT CLOUDS AND BIG DATA TRENDS

The role of large datasets in Information Technology

The last few years in the world of information technology have seen an explosion in the amount

of data being collected about the world around us, with limited realization of the full potenti One

manifestation of demand for being able to process large datasets lies in the world of remote sensing,

image and point cloud interpretation, and computer vision. This is particularly true in many industries or

sectors interested in monitoring and modeling precise aspects of the natural environment that require

very particular processes to make sense of complex sets of sensed information. . The focus of this

document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data

and the fusion of these measurements with imagery. The goal here is to examine a few processing

algorithm types in the abstract, and discuss potential applications of these processes for feature

extraction in a geospatially enabled environment.

Point Clouds and LiDAR processing Demand

Government, Intelligence, Natural Resources (extraction or management), as well as marketing

and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the

computer algorithms to process them to make technology more interactive or information more

exploitable. Government agencies increasingly are seeking out finer scale data about the built

envirionment and ways to process this data into actionable information products. For example, LiDAR is

being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as

more abstract interpretations such as determining rooftop areas suitable for solar panel installation,

modeling of noise in urban environments, and security applications such as crowd evacuations or

line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser

range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling

legislature in California.5 These vehicles present one example of the private sector pushing for rapid and

automated processing of large point clouds. The robotics community has been working out a standard of

point cloud processing to enable piloting of robots in environments. Another potential application of

point cloud processing in the private sector includes drones, which are expected to soon populate our

airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6

HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION

LiDAR Processing – Popular Approaches

The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has

prompted a search to streamline feature extraction while maintaining accuracy. A few common

methodologies to feature extraction from LiDAR have culminated from these efforts, and are the

underlying processes in popular tools and software extensions claiming to automate or semi-automate

the feature extraction process.

It is worth noting, LiDAR

data in the geospatial industry and

related professions is currently

categorized, in terms of processing

approach and application based on

its collection method: Airborne and

Terrestrial.7 Airborne LiDAR can be

points sensed from overhead in a

fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from

helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed

from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and

remote-control sensing robots. Each data collection method is dependent on the necessary precision of

the application, and has a different common processing approach.

Height-based segmentation

The most common approach, and most easily implementable, is a height-based segmentation of

features. This is most commonly applied for data collected from fixed-wing and helicopter aerial

flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The

classification is based on interpolated surface models from the points.The bare earth surface model,

which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is

referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points

collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,

vegetation, structures, and other features captured.

The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to

obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the

height values representing relative distance from ground. It is useful in classifying trees, buildings, power

lines, and other infrastructure, but does not segment the different types alone. The process is essentially

fully automated.

Shape – Fitting

Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular

Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures

from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but

related variation of this is a semi-automated process of drawing lines by hand into point clouds, using

snapping or fitting user interfaces.89

Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is

explained further in the context of many processing approaches in the following section.

Spectral Fusion / Intensity and RGB Values

More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the

vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from

imagery into the points or vice versa, burning in intensity into an image band. These are spectral

classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image

collection (passive remote sensing) or point intensity (active sensing). Common application for this is

seen in biology and forestry, where it is useful to use spectral information to discern various tree and

shrub species.

Point Cloud and Image Processing in a “Computer Vision” Environment

Recently, there have been developments in much more advanced and effective ways to process point

cloud information and imagery. Some of the methods stem from LiDAR processing software, while

others are being transplanted from the general image processing community. The goal of these software

tools and packages is to provide a means by which to translate “human vision”, or how we segment

parts of an image of point cloud that we perceive, into computer algorithms to replicate the

segmentation power of the human mind. There are several general categories of how we perceive our

environment through visual cues.

Image-based visual clues

The first types of visual clues are from actual value statements about objects: color / tone,

morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially

auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are

near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around

the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a

residential area, and the largest structure is probably a school).

Vector and Second-order visual clues

The second type of visual interpretations are based off of defined shapes. Defined shapes are

objects we have recognized as true, such as vectors delineating what may be houses, or our mental

outline of the various perceived features. Theoretically, vector-based visual clues are typically

second-order visual clues in interpretation, as we are defining an object based on first order clues of

morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s

properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of

angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,

symmetry, and contextual analyses based on other vectors.

ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING

AND SOFTWARE PACKAGES

Image Based Algorithms

Trying to translate visual cues from image based-perceptions relies on manipulating data in a format

representative of its color, texture, brightness, etc. This can be done in vectors representing a the

average values of these at an area, but the primary method for interpreting these characteristics is

raster, or gridded image based.

Spectral

Many of the concepts in image-based interpretation stem from classification methods used on

remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral

parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels

as water that meet this requirement) to supervised, where samples of features such as grass, trees,

roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping

based off of the modeled samples is performed.

Slope

There are some particularly unique image-based processing techniques that apply to point clouds,

though, and these are all based off of interpolated elevation models’ characteristics. One example of

this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud

elevation grid, and selecting the highest slopes. This is a translation of how the human visual

interpretation segments buildings and structures out of an image – by determining where the slope is

the highest we define the rough location of walls, and tall trees.

Curvature

A further classification method is based off of texture – often referred to as “curvature” of the elevation

model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a

surface is we can classify out certain features much easier: Trees and shrubbery have much higher

curvatures that mandmade structures such as houses.

Aspect

Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to

segment features based on general orientation. For instance, house walls will be typically oriented

towards roads, and combining this contextual information can help place seeds (house walls facing

roads) by which to grow into features (houses).

Vector based Algorithms

By deriving vectors representing either

features, or the rough area of features using

image segmentation algorithms, we can further

attempt to locate certain features by definite

characteristics such as feature size (area),

shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other

features, density, etc.).

Size, Perimeter

When looking at images, we segment objects by size as well: homes will be larger than trees, shopping

centers larger than homes, and forest stands will be larger than individual patches of foliage and tree

plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further

discern between these features. Perimeter length of the vectors plays a role too, a although a house and

a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and

curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from

two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

volume, which takes into account the height of derived features.

Shape – Characteristics

To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is

also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of

sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree

vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be

smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on

buildings, and limited acute angles. Using calculations to highlight continuous lines such as building

edges can help to refine the vector during classification, or to ouline features such as roads or power

lines.

Shape, Type

A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point

clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting

to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be

done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.

Contextual and Growing Algorithms

Contextual Segmentation

A combination of characteristics derived from vectors and images can be used to translate common

human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major

intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and

roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These

types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual

analysis can be applied to search for non-static environmental objects, based off of basic buffer and

spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken

X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial

analysis, rather than mapping of the physical environment.

Growing Algorithms

Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or

expand into pixels around a designated “seed” starting point based on characteristics of these points

helps to determine boundaries of features as well. This process is less human-vision oriented as it is to

be definitive for computers. Determining locations of centers of homes in a residential area, and region

growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can

delineate homes very well. This information can be contextual as well – region growing around a known

fire-starting point into a forest patch can delineate burn paths.

WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT

NEED

The workflow or sequence of algorithms (and likely software packages) you use will vary based on the

type, and precision of classification you are seeking. The simplest scenario: You are attempting to just

classify ground points in your cloud to create a surface model for flood and stormwater mapping

purposes, all you will need is to calculate height extremities to single out the features, although it is

likely the data vendor has already done this for you. More often than not, this is not the case, and has

led you to exploring the world of point and image processing.

Examples:

Workflow for classifying buildings and vegetation in a suburban residential area

Classifying buildings is a relatively challenging, but feasible process which depends on the degree of

accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of

buildings are easy to determine. If your project is trying to model buildings, and requires definitions for

footprints, and roof overhangs, among other building components like facades, then you will be

spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected

for this purpose yet.

nDSM Calculation

The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the

building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the

DEM from the DSM are used to outline all structures. The next steps deal with shape-based

classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with

vector-based fine-tuning of the classification.

Curvatures and Slope

First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and

highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature

derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave

behind the other structures, which will be mostly buildings but may be other structures like water

towers or power lines.

Vector Calculations: Shape Area and Perimeter Length

Next, the vector-based fine-tuning of our features continues. The remaining building shapes are

converted into vectors, and a range representing the high and low value for shape area, or the square

footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors

representing other features). A perimeter threshold is applied as well to remove a large forest patch to

the north.

Potential steps to fine tune the building shapes:

Simplification of Tree and Building Vectors

At this point, rough vectors are generated for our

buildings, and trees. We can either simplify the shapes,

applying a smoothing to the vectors for trees, and a

orthogonal / rectangular simplification for the buildings.

This may not be appropriate for all building shapes, if

they are not rectangular.

Region Growing from Seeds

An alternate approach at this step can be to calculate

polygon centroid for each vector, and use these points as

seeds for region growing across a color, color-infrared, or

fusion raster (raster with LiDAR intensity or heights burnt

in as a band). This can be done to get pixel shapes for

trees and houses which can be converted to vectors.

Region growing can be useful for non-linear and circular

building shapes.

Segmentations in dense urban areas

Dense Urban areas present a unique challenge because

they are compacted, layered in mixed-use developments,

and often of unique shapes dreamed up by architects.

Extremes in High / low building heights can have affects

on calculations and interpolations as well. Using region

growing, or spectral fusion of multi-band rasters with

interpolated point cloud metrics as bands can be

helpful.

Delineating roads with line fitting

Road delineation is only semi-complex, and can be

done without using point clouds on just imagery. A

point cloud could be used to interpolate heights in

the end, or can be used to create an “intensity”

band from the LiDAR metrics with which to aid

spectral segmentation of the roads.

First, an image (potentially including intensity as

one of the bands) is segmented using a classification

method from remote sensing techniques. An

appropriate segmentation can be a multi-resolution

segmentation which computes statistics at different

scales between neighboring cells, or other

supervised classifications.

Roads are identified by defining spectral thresholds

which determine a road in Red, Green, Blue,

Infrared, and Intensity space. The pixels are

coverted to a raster, and a centerline is calculated.

Classifying a vegetation and natural features with

spectral information

Vegetation can be much more precisely classified by

incorporating LiDAR metrics with spectral

information. Instead of merging derivative elevation

and intensity rasters into a BGR, IR image, the

image is burned onto the points if it is of

appropriate resolution. Precise definition of various species of trees can be defined and put in as

thresholds by which to classify images.

Classifying lakes and islands with contextual clues

Contextual information is often the most neglected in common remote-sensing and point cloud

segmentation techniques, but it is arguably the closest to human vision and perception. When we look

at an image, we examine individual objects first, but also examine the rest of the image to understand

context. These interpretations are based on our conceived notions of typical distributions in

environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)

would not fool us to be a large tree considering factors in how it is situated in the context of other

houses, and house orientation which we assume by the context of where the house is next to a road,

and where the driveway is. Advanced contextual clues underlie some of the principles of

photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in

software for image processing.

CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND

DATA

Synthesis / expansion of remote sensing

Advances in image processing spurred by collection of high-resolution imagery and point cloud

information are furthering the development of technologies in geospatial applications and remote

sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)

community including Integraph, ESRI, and Overwatch Industries have released point cloud processing

applications or amended the functionality of their software to support the .las (American Society of

Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging

Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a

rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing

demand for techniques and processes to segment point clouds into useful datasets and derivatives.

Cross-pollination with robotics

The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,

along with intense research funding from DARPA for autonomous navigation of robots, and autonomous

navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the

environment and obstacles.11 Google is also pushing the initiative with their experimentation with

autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which

could benefit greatly from being able to collect and process high-resolution imagery and range-finding

data from sensors.

Improving human-interface devices and experiences

From the consumer marketing perspective – application of sensing technologies is going through major

revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning

technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and

body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human

posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their

walking pattern and posture.13

ROLE OF POINT CLOUD PROCESSES IN BIG DATA

Uses of Precise Data

Derivatives from point cloud data and large datasets have huge potential applications and public

management, environmental management, resource extraction, intelligence and defense technologies,

consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models

have many applications as mentioned previously and in some of the case processing examples.

Having large datasets available is one asset, having precise deliverables and information about these

datasets is another. Being able to process these large datasets into actionable information and

informative research will harness the amount of information currently being collected.

Works Cited

1 Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.

2 "The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.

3 "2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.

4 "Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.

5 "Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec. 2012.

6 "Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d. Web. 14 Dec. 2012.

7 "Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.

8 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

9 Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State University - Department of Civil and Material Engineering (n.d.): n. pag. Web.

10 Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.

11 "DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.

12 Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.

13 Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.