Schematic Surface Recosnstruction-an Overview

Schematic Surface Reconstruction

Abhilash Chandran

Abstract In this paper we discuss an algorithm whichspecializes in reconstructing an architectural scene, from asparse 3D point cloud generated using structure from motionscanning technique. This algorithm takes advantage of specificconditions like profile curves and transport curves that arecommon in most of the architectural scenes. Incorporatingseveral mathematical models and approaches along with theinformation obtained from structure from motion point clouds,it extracts the salient features of a building and reconstructs aschematic model using only the basic elements like transportcurves, profile curves and generates a floor plan of the buildingwithout any prior knowledge about the layout of the structureor the scene itself. Later in the evaluation some ideas have beenproposed which might add value to current methodology.

I. INTRODUCTION

In the advent of various technical innovations in the 3Dtechnology, the necessity of capturing and extracting valuableinformation has gained an acute importance from a consumerdomain as well as information domains domain. A clearshift of focus from a 2D perspective to 3D perspective ofinformation could be easily noticed in the recent years.This focus shift from 2D to 3D is necessary, because inan environment where automated agents interact, the actuallocation of objects are important. For e.g. in an industrialscenario, actual location of objects is necessary for objectgrasping or at times evaluation of the build quality of theend products. In common scenarios for robot, it is necessaryfor the robot to know the path and depth of its location tonavigation and perform certain tasks.

Following this line of innovations the authors of thepaper[1] have introduced a technique to automate theschematic model extraction from a sparse 3D point cloud.The primary focus of the authors who introduced this al-gorithm, make use of the input from the SfM, taking intoaccount, its nature of higher sparsity compared to otherscanning devices like Kinect or laser scanners[1]. Fitting aswept surface on SfM is very much of a challenging taskdue to its nature according to the main authors.

In this paper we will have brief a introduction to the3D point clouds to acquaint the readers with the relevantbackground details to have a better understanding of theschematic surface reconstruction algorithm which will beexplained further in this paper. This paper also describesthe basic terminologies like transport curves, profile curveswhich are necessary to form a solid basis in understandingthe algorithm.

The schematic representation of point clouds is necessarydue to several reasons, namely compactness, easy and intu-itive rendering of structures from sparseness. Depicting pointcloud in terms of shapes and curves is necessary for domain

specific applications like construction of floor plans, validityof building structures. Another way of looking at it is that weneed architectural scenes not as point clouds which have nomeaning but in a rather simplified manner as floors, rooms,domes, cones. This method of geometric parsing can alsobe extended to representations of architectures as rooms,living area, number of floors. This makes sense intuitivelybecause this makes addition of 3D models also easy likeadding furniture to rooms or changing the structure of room.

II. RELATED WORKA. Structure From Motion

A 3D point cloud as the name suggests, is a sparsecollection of points in a three dimensional space. Thisis generated with the help of 3D scanners like Structurefrom Motion(SfM) technique, Kinect, Time of Flight(TOF)cameras, laser scanners and images. Because we abundantimages available in the internet namely Flickr, Street viewetc, it is easier and economical to use these images forconstructing 3D models of the most prominent architecturalscenes as done in the modeling[2] as shown in Fig.1 forColosseum.

Of these techniques lets discuss briefly how the SfMtechnique is implemented. At first the images of a monumentare collected in one place for the algorithm to scan through.Next, feature detection is performed on these images usingSIFT[3] algorithm. Then the process of feature mapping isperformed to find correspondences among these set of im-ages. These correspondences will coincide with the originalartifacts of the structure/monument in the 3D world. Usingcorrespondences, the algorithm sorts the features accordinglyin tracks. Tracks is essentially a connectivity graph repre-sented in the form of a matrix. A row of a track representsthe features that are calculated from an image in the originalset of images. A column of the track matrix represents aparticular feature that occurs in all the cameras.

Once this track matrix is constructed, a pair of cameraswhich have the most number of features are selected first forcalculation of the camera parameters and reconstruction ofthe 3D points. Then cameras are added one by one to theSfM pipeline according to a specific criteria. This criteriaenforces that the next image added has a certain minimumnumber of features which correspond to the previously addedset of cameras and they must not completely overlap oneof the previous images added earlier. The advantage of theformer criteria is that repetitive and non-informative imagesare clearly excluded from being considered. In addition wecan infer that this criteria helps in avoiding the local minimain the optimization as much as possible. This is called Sparse

Fig. 1: This is a point cloud of a Colosseum constructedfrom images collected over internet.[1]

bundle adjustment, which minimizes the re-projection errorof the reconstructed 3D points.

B. Schematic Representation

Schematic representation is a compact representation of acomplete three dimensional structures like buildings. Froman architects perspective it is often referred as a blueprint.One of the basic example for this would be the floor planof a building with some rooms and doors as shown in Fig.2.A schematic representation reduces the complexity of a 3Dstructure with the help of simple lines and curves. Theserepresentations are easily grasped by humans. Despite itssimplicity in layout, it is very much capable of expressingthe details of the complex structure.

III. ELEMENTS OF SCHEMATIC SURFACE MODEL

The schematic surface model focused in this paper iscomposed of two different types of planes and curves aslisted below. Transport plane Transport curve Profile plane Profile curve

It is also important to take a note that multiple profilecurves could share the same transport plane yet followdifferent transport curves. This will be discussed further inthe upcoming sections.

A. Transport Planes and Curves

Transport planes are identical to the ground plane andusually parallel to the ground plane. This plane holds thetransport curves which drives the profile curves along thetransport curves to form a structure. Transport curves t(u)as shown in Fig.2 is parallel to transport planes and lie on the

Fig. 2: A simple floor plan of a home.

Fig. 3: Transport plane, with a transport curve t(u) where btis the ground plane normal.

transport plane. bt is the transport plane normal and this is thecommon normal for all the profile curves in an architecturalscene.

B. Profile Planes and Curves

Profile Curves define the shape of an architectural struc-ture. For e.g. imagine an arch inside a church. The shapeof the arch is the profile curve. So the profile curves areorthogonal to the transport curves and thereby to the transportplane itself. This is one of the common feature of manyarchitectures and used by the author[1]. The height of theprofile curve along Zp axis, thus defines the vertical extrusionof the surface while reconstructing. As shown in the Fig.4 thecircle, square and the polygon could be identified as a simpleprofile curve. The plane which is covered by the profile curveis referred as the profile plane along Yp.

Fig. 4: Transport plane, with various Profile curves likecircle, square and polygon. Xp is the transport direction axis.

C. Swept Surface

A swept surface is generated by sweeping a profile curvep(v) along a transport curve t(u) on the transport plane.While sweeping profile curve, the orthogonal nature of p(v)and t(u) will be maintained. Swept surface is formulated as

S(u, v) = t(u) +R(u)p(v) (1)

where

R(u) = [t(u), bt t(u), bt] (2)R(u) is the Rotation applied on the profile curve.t(u) is the transport direction.

Fig. 5: A circular profile curve p(v) swept along a transportcurve t(u) to form a torus.[1]

IV. SCHEMATIC SURFACE RECONSTRUCTION

A. Ground Plane Normal

The authors [1] begin their approach by pre-processingthe point cloud obtained from the SfM by performing aPrincipal Component Analysis. Considering xi as any pointin the given point cloud, the approach tries to identify twoprincipal directions ci1 and ci2 with a distance threshold ofTR and ni which is the point normal[2]. The TR distance ischosen based on a statistical approach considering the firstquartile of number of neighboring points is 100 for all theinput points in the point cloud. ni the point normal, directedtowards the camera[2].

After the initial processing, the authors imply a techniqueto identify the ground plane normal. That is, within a givenpoint cloud, majority of the points will have one of its prin-cipal directions perpendicular to the ground plane normal bt.For a given swept surface S(u), the two principal directionsare R(u)p(v) and t(u)[4]. Based on this proof the groundplane normal bt could be deduced by arg maxb

i(c1i

b) (c2i b).This approach is optimized further by the authors, by

taking the point ni into the deduction rule of bt as shownbelow.

bt = arg maxbi

((c1i b)(c2i b)(ni b))(ni b)(3)

Additionally the transport direction of each point is givenas follows, by identifying the unit direction vector for eachpoint.

ti =

{btni|btni| ni btUndefined otherwise

B. Extracting the Transport Curve Points

After deducing the ground plane normal bt which isa crucial factor of the schematic surface reconstructionapproach, the algorithm moves ahead to choose the transportcurve points. This section discusses how several transportplanes are identified from the point cloud and the statisticsapplied to choose an optimal transport plane. The authorsuse the strategy of choosing transport curve points of thoseplanes with minimal curvature and minimal noise.

By definition of schematic surface model for architecturalscenes for any given point xi, the (bt, ni) angle is alwaysconstant within a transport curve[1]. This acts as the firstselection criteria i.e. given a transport plane pit, the noise

Fig. 6: A point cloud structure with a spanned cav-ity/hole[left] and the corresponding partial profile curvesgenerated [1].

level of the plane is measured by calculating the variance ofthis angle as

(pit) = stddev{(bt, ni)|xi pit}The second selection criteria begins by projecting the

point normal ni onto the transport plane pit, and estimatingthe curvature ki of transport curves. Then RMS method isapplied to approximate the curvature of the plane pit as

c(pit) = rms{ki|xi pit}This is carried out over several iterations and list of

transport planes are generated. Then a cost of selecting aplane pit is calculated as (pit) + c(pit) and the planewith the least cost is chosen. This transport plane, alsoreferred as transport slice, is then intersected with the pointcloud to extract the transport curve points. By analyzing theconnectivity of the points generated from this intersection, adraft transport curve could be generated.

C. Profile Curve Reconstruction

Each of the transport curve points extracted earlier actsas a seed for the next step in the algorithm. If we imaginem transport curve points for a simple square shape building,then each of these points define m different profile slicespiip where 1 i m. Then each of these profile slice pipcontains a large number of points. The authors introduce amechanism to filter out specific point from each profile sliceapplying the following selection criteria. Select those points whose point normal ni is orthogonal

to the transport direction. This reduces the angled slicesfrom being considered.

Choose points based on the non-self intersection as-sumption which is a general rule in any architecturalscene[1].

Considering a structure with a hole spanning across itssurface, it is easy to imagine that several partial profile slicesare created by the above filtering as shown in the Fig.6.Once the set of these profile curves are extracted from a pointcloud, a merging mechanism is applied by transforming theseslice onto a canonical plane. Considering a profile slice piippoint xi, any point xj on the other profile slice is transformedonto the canonical profile plane coordinate yij by[1]

yij = Ri1(xjxi) (4)

Fig. 7: An accumulated profile curve from the profile slicein Fig.6[1].

and the corresponding point normal nij is transformed as

nij = Ri1nj (5)

This way the profile slices are merged to form an accu-mulated profile curve as shown in Fig.7.

1) Profile Slice Clustering: Sometimes multiple profilecurves share the same Transport curve. In addition to themechanism discussed above for profile curve reconstruction,a clustering methodology is used to group the profile curves.

Clustering the profile slices helps improve the accuracy ofreconstructed profile curve. A seed slice is chosen repeatedlyto cluster the curve points.It also improves connectivitywithin the swept surface.

D. Transport Curve Reconstruction

The next major step in this algorithm is to reconstruct theTransport curves. Rearranging the swept surface equation.

t(u) = S(u, v)R(u)p(v) (6)

where S(u, v) is the position of the point.R(u) is the rotation at the point.p(v) is the reconstructed profile curve.

Now an interpolation technique is used to robustly extractthe transport curve. This is achieved in a two step process.For a point xj its corresponding point pij on the profile curvep(v) is estimated by intersecting the line (y yij).Zp = 0with the curve[1]. Once the point pij is estimated, the nextstep is to extract the corresponding transport point applyingthe Eq.6. That is each point xj on a profile slice piip istransformed to [1]

zij = xj Rjpij (7)

Each point xi on the profile slice is transformed into

zij = xi Rjpij (8)

Once these points are accumulated several other profileslices are chosen to interpolate the points between twocontinuous profile slices to reconstruct the transport curveas shown in Fig.8.

Fig. 8: An accumulated transport curve and profile curve.[1]

E. SweepingOnce the transport curves and the profile curves are

extracted from the 3D point cloud the next step is to formswept surfaces which are generated by sweeping a meshof the profile curve along the transport curves. The finalschematic representation of a scene is defined by multiplenumber of swept surfaces, wherein the connectivity betweenthese surfaces are defined according to the extracted transportcurves[1].

Various conditions occur while doing this sweeping mech-anism depending on the architectural identity and uniquenessof the design of the buildings. For e.g. intersections mayoccur between profile curves. The authors use the techniqueof marking the points which are already swept using anotherprofile curve. So points which are marked as swept will notbe considered while sweeping a second profile curve, in thescenario of multiple profile curves which is common in mostof the architectural scenes. The marking of points is carriedout using the distance threshold TR.

Also multiple transport planes are possible in a biggerstructure. In this case the transport planes are chosen in adecreasing order of their sizes and the sweeping is carriedout.

F. Floor Plan GenerationThe transport planes extracted from the point clouds could

help in extracting another crucial aspect of the structurewhich is the floor/surface plan of the building itself. Thisinformation is extracted by connecting various transportcurves which are already extracted. the transport planeswhich intersect most surfaces are identified and the transportcurves on these planes are used to extract the floor plans forthe scene.

A sample floor plan is shown here in Fig.9 based onthe experiments carried out by the authors. Additionally thealgorithm color codes the information if the transport curvesare obtained form different transport planes as in Fig.9.

Fig. 9: The reconstructed floor plans of the Allen Center andUris Library from the experiments by the authors[1].

G. Optimization

By reconstructing the surfaces from the profile curvesextracted earlier many details of the structure are lost, asthe sweeping is applied directly without considering any ofthe depth information from the 3D point cloud.

To overcome this issue, the authors apply a technique tominimize the energy function including some optimizationparameters as follows.

Esweep = Edata + nEtangent + sEsmooth (9)

Edata is a minimizing function which tries to reduce thedistance between the reconstructed surface points and theactual 3D points[1].

Edata =|(xi Sd(ui, vi)).Ns(ui, vi)|2

Here Sd is the optimized surface reconstructed from theoriginally reconstructed surface and Ns(u, v) is the normaldirection for the error estimation.[1].Etanget is minimization of tangent fitting costs to both the

profile and transport curves. This is based on the expectationthat the derivatives of the curves are perpendicular to theircorresponding normal fields[1].

Etangent =

(|pd(v).Np(v)|)2 + |td(u).Nt(u)|2)Further the smoothness of the swept surfaces are optimizedby using the second order derivatives of the transport andprofile curves[1].

Esmooth =

(||pd)||2 + ||td(u)||2)In order to obtain some additional details of the depth of

the objects like windows and holes within the architecturalstructure, the authors apply a minimized displacement mapon top of the reconstructed swept surface as shown below[1].

Edisp = Edata + dEmesh (10)

andEmesh =

(|Su.Ns|2 + |Sv.Ns|2) (11)

Here Su and Sv are the two partial derivatives of the dis-placed swept surface[1].The above approach tries to penalizethe big jumps within the normal directions along which thesurfaces are swept.

V. EXPERIMENTS

The authors implemented and conducted experiments onthe point clouds of various architectural scenes. This in-volved some prominent monuments like Colosseum1. Theexperiments were performed on different amounts/density ofpoint clouds. The results of this experiment is shown inFig.10 which is table with details like the no of points inthe SfM, amount of time spent in minutes for each sector ofthe algorithm like sweeping and optimization. The results ofthese experiments are provided in the authors website[5].

Based on the results of the experiments conducted, it wasidentified that the algorithm performs better even with lesseramounts, i.e. sometimes even upto 10% of the actual 3D

Fig. 10: The statistics of reconstruction and the timings ofeach step in minutes. The experiments were conducted on aPC with Intel Xeon X5680 3.33Ghz CPU and 12GB Ram.[1]

Fig. 11: Reconstruction of Colosseum. The upper part ofthe image shows the curves extraction and lower part showsreconstruction with optimization.[1]

points obtained from the SfM. Especially for Colosseum thealgorithm was able to reconstruct with only 1% of the actual3D points as in Fig.11. It could also be noticed that manyholes exists within the reconstructed Colosseum due to thelack of details from the corresponding SfM. Despite this lackof details, the algorithm is successful in extracting the surfaceplan of the same.

As the authors noted, overarching these efficiency, theuse of the threshold TR in the reconstruction could limitthe outcome, for sparse clouds of complex surfaces wheresuch surfaces would be broken into small pieces[1]. Thisthreshold is chosen automatically based on the density of thecloud during the pre-processing of point clouds using certainstatistics. Since this could vary according to the input pointcloud it has some proportional effect.

An example result with intermediate images for the recon-

Fig. 12: Extracted Profile and Transport curves for the St.Peters Basilica..[1]

Fig. 13: Reconstructed 3D Structure of St. Peters Basil-ica. The right side of the image shows an optimizedreconstruction.[1]

struction of the St. Peters Basilica is shown in the Fig.12and 13 respectively.The experiment results furnished(also thevideos in the authors website[5]) demonstrates the ability ofthe algorithm to detect the Ground normal consistently andextract the curves with greater accuracy without loosing anydetails of the structure being reconstructed.

VI. EVALUATION AND FUTURE WORK

The extraction of profile and transport curves from thepoint cloud is relatively cumbersome and is prone to missimportant details of the scene. Though the approach intro-duced by the authors[1], handles the effective details of thestructure, as the authors noted for complex structures couldevoke incomplete reconstruction. Especially if the details ofthe point cloud is sparse for such structure. Additionally thecalibrations of the cameras, angle of the image, distortion ofthe images used etc. could impact the point cloud generatedby SfM techniques leading to lack of details like pointnormal.

This could be overcome by introducing prior curves to thealgorithm. Prior curves in this context means that a genericidea about the profile or transport curves of the architecturalscene being considered for reconstruction. This will help inverifying the correctness of the profile and transport curvesextracted by matching it against prior curves. It could helpimprove the details and correctness of the reconstructedstructures. For e.g. if we are reconstructing the Leaning towerof Pisa Fig.14 then the inclination of the entire structurecould be considered as a prior for the profile curve extraction.

Fig. 14: The leaning tower of Pisa[6].

One of the bottlenecks of this algorithm is the pre-processing of 3D point clouds to obtain the point normals.

Since the algorithm depends on the point normal detailsfrom the SfM output which is used ground plane normal, theoutput of SfM is important which depends on other factorsas mentioned earlier. This part of the algorithm could beadapted to work with Point normal calculation using Patchbased Multi View Stereo(PMVS)[7]. What PMVS does isto work on the point cloud and the corresponding imagessimultaneously to prune for denser point clouds and to extractthe patches of planes which will have a more precise andaccurate information about the normals. This helps skippingerroneous way of calculating the point normals from thedirection of the camera, because this algorithm tries to takevoting mechanism of normal calculations from the multiplepoints rather than a single point.

Apart from this, the amount of detail available in thereconstructed 3D structure could be enhanced and usedfor indoor mapping of building. The experiments could beextended to indoor rooms in such a way that new/existingprefabricated 3D objects like books, chairs, couches can beadded on top of the existing artifacts such as shelves, tableetc, if extracted accordingly.

VII. CONCLUSION

The approach introduced by the authors[1] is one of theideal techniques, to represent 3D structures from sparse aswell as dense point clouds. One of the highlights of thealgorithme is the optimization techniques used in extractingspecific details like depth of artifacts from the scene. Theidea of using simple planar curves for the reconstruction,is a clear method which is applicable in generic man madearchitectures. Also this algorithm is found to be workingefficiently of denser point clouds like MVS as noted by theauthors[1].

Certainly there are some limitations to this algorithmdespite its efficiency and correctness of reconstruction. Thisalgorithm, for instance would perform poorly if the structurehas more curved layouts or details like multiple pillars withinthe structure as in Fig.[6]. This technique lags behind forpoints of poorly textured regions of the scene. As suggestedin the future works section, a prior knowledge of these planarcurves would imply that the results will be much morecleaner and precise in nature. Though this might involveadditional computation, it might prove to be effective. Alsoto be noted is that, giving prior curves is fruitful only if thestructure is wide as in Fig.12 and has multiple number ofprofiles. In other cases like Colosseum the basic algorithmwould suffice.

This whole approach as of now cannot be implementedas a real time application due to the complexity of the cal-culations involved, yet could eventually become one, giventhe advancements in the processing technology like GPUs.If a parallel processing enhancement could be embeddedalongside, it could help well in real time reconstruction andrepresentation of the scenes. Especially the generation ofSfM which is a precursor for this algorithm, consumes alot of time for processing millions of images and proves tobe a challenging task for a real time scenario. This technique

in its current form could only be used as a pre-processed addon into knowledge base of robots/agents, exploring such kindof environments.

REFERENCES[1] C. Wu, S. Agarwal, B. Curless, and S. M. Seitz, Schematic surface

reconstruction, in Computer Vision and Pattern Recognition (CVPR),2012 IEEE Conference on. IEEE, Jun. 2012, pp. 14981505. [Online].Available: http://dx.doi.org/10.1109/cvpr.2012.6247839

[2] N. Snavely, S. M. Seitz, and R. Szeliski, Modeling the world frominternet photo collections, International Journal of Computer Vision,vol. 80, no. 2, pp. 189210, 2008.

[3] M. Brown and D. G. Lowe, Automatic panoramic image stitching usinginvariant features, International journal of computer vision, vol. 74,no. 1, pp. 5973, 2007.

[4] H. Rom and G. Medioni, Part decomposition and description of 3dshapes, in Pattern Recognition, 1994. Vol. 1-Conference A: ComputerVision & Image Processing., Proceedings of the 12th IAPR Inter-national Conference on, vol. 1. IEEE, 1994, pp. 629632.

[5] C. Wu, S. Agarwal, B. Curless, and S. M. Seitz. (2012) Schematicsurface reconstruction. [Online]. Available: http://grail.cs.washington.edu/projects/schematic/

[6] A. M. g. Wikimedia Commons. (2015) Creative commons attribution-share alike 3.0. Online;accessed 25-January-2015. [Online]. Avail-able: http://en.wikipedia.org/wiki/Leaning Tower of Pisa#mediaviewer/File:Leaning tower of pisa 2.jpg

[7] Y. Furukawa and J. Ponce, Accurate, dense, and robust multiview stere-opsis, Pattern Analysis and Machine Intelligence, IEEE Transactionson, vol. 32, no. 8, pp. 13621376, 2010.

INTRODUCTIONRELATED WORKStructure From MotionSchematic Representation

ELEMENTS OF SCHEMATIC SURFACE MODELTransport Planes and CurvesProfile Planes and CurvesSwept Surface

SCHEMATIC SURFACE RECONSTRUCTIONGround Plane NormalExtracting the Transport Curve PointsProfile Curve ReconstructionProfile Slice Clustering

Transport Curve ReconstructionSweepingFloor Plan GenerationOptimization

EXPERIMENTSEVALUATION AND FUTURE WORKCONCLUSIONReferences