Applying Security and Privacy Enhancements to a Visual Localization and Tracking Systemcs.uccs.edu/~gsc/pub/master/falrashe/doc/report.pdf · 2015-04-14 · ii Abstract -- Sending

Applying Security and Privacy Enhancements to a Visual

Localization and Tracking System

By

Fahad Alrasheedi

B.S in ITC Arab Open University, 2010

A project

submitted to the Graduate Faculty of the University of Colorado at Colorado Springs

in partial fulfillment of the requirements for the degree of

Master of Engineering in Information Assurance

Department of Computer Science

2015

This project for the Master of Engineering degree by Fahad Alrasheedi has been approved for the Department of Computer Science by

_______________________________________________________ Dr.  Jonathan Ventura, Advisor

_______________________________________________________ Dr. Dr. Jia Rao, Committee member

_______________________________________________________ Al Brouillette, Committee member

_______________________________ Date

ii

Abstract -- Sending data over the Internet without using security measures to keep the data

protected from modification and even from eavesdropping is a dangerous practice and not

recommended at all. The situation is even worse if the purpose of the communication is to send

sensitive data that demands a high level of security such as bank transactions, military operations,

localization and tracking systems, or even private communications with family members and friends.

Localization and tracking systems that use the Internet as a communication medium between the

person ( client ) and a localizer (server ) should take into an account the security part of the process

otherwise it will be easy for an attacker to read the message content including the query from the

client and the response from the server . The advancement of technology and the rapid growth of

solutions including commercial or open source endeavors, are intended to tackle such security-

related problems and network vulnerabilities to help us keep our communications over the Internet

more secure and reliable. In this paper, we will review the common network attacks which may

happen to the system proposed by Ventura and T. Höllerer [1], which uses a normal TCP

connection, and describe methods an attacker could use to exploit such a kind of unencrypted

connection . Then , we will implement and evaluate different levels of security to enhance the

system and make it more secure.

iii

ACKNOWLEDGMENTS

I would like to express my special appreciation and thanks to my advisor Professor Dr. Ventura, you

have been a tremendous mentor for me. I would like to thank you for encouraging my research and

for allowing me to grow as a research scientist.

I would also like to thank my committee members, professor Dr.Rao, and Mr.Brouillette for serving

as my committee members even at hardship. I also want to thank you for letting my defense be an

enjoyable moment, and for your brilliant comments and suggestions, thanks to you.

A special thanks to my family. Words cannot express how grateful I am to my mother , father for all

of the sacrifices that you’ve made on my behalf. Your prayer for me was what sustained me thus far.

At the end I would like express appreciation to my beloved wife Hanadi who spent sleepless nights

with and was always my support in the moments when there was no one to answer my queries.

iv

LIST OF TABLES

Table 1 : AES parameters . source [9] ……………………………………………………………………………………………17

Table 2: Comparison average total time between DB1 and DB2………………………………………………………………..22

Table 3 : Comparison between CP and GPU when dealing with DB1………………………………………………………….23

Table 4: Comparison between CP and GPU when dealing with DB2…………………………………………………………..24

v

LIST OF FIGURES

Figure 1 : Overview …………………………………………………………………………………………………………………1

Figure 2 : Camera Matrix , source [10] ……………………………………………………………………………………………5

Figure 3 Flowshart to build 3D from VRLT………………………………………………………………………………………8

Figure 4 : Feature Extraction………………………………………………………………………………………………………8

Figure 5 : Matching features linearly………………………………………………………………………………………………9

Figure 6 : Making tracks……………………………………………………………………………………………………………9

Figure 7 Bootstrapping two images…………………………………………………………………………………………………9

Figure 8 : Adding the remaining images ……………………………………………………………………………………….….9

Figure 9 : Reconstruction the 3 D…………………………………………………………………………………………………..9

Figure 10 : Certifacet Signing………………………………………………………………………………………………………10

Figure 11 : Image bytes in Hexadecimal……………………………………………………………………………………………12

Figure 12 : Wireshark catpture for normal TCP…………………………………………………………………………………13

Figure 13 : Wireshark catpture for TCP/SSL…………………………………………………………………………………….14

Figure 14 : a square matrix of plaintext……………………………………………………………………………………………16

Figure 15 : Overview of AES algorithm source [9] ………………………………………………………………………………20

vi

TABLE OF CONTENTS

CHAPTER PAGE ABSTRACT ................................................................................................................................... ii ACKNOWLEDGMENTS ............................................................................................................ iii LIST OF TABLES ........................................................................................................................ iv LIST OF FIGURES .........................................................................................................................v CHAPTERS CHAPTER 1 – Introduction .............................................................................................................1

1.1 Overview ...............................................................................................................1 1.2 Why to Secure VRLT and Research question ......................................................2 1.3 Common Possible TCP Attacks ............................................................................3

CHAPTER 2 – Related Work Related Work and Literature Review 4 ..........................................4 CHAPTER 3 – Creating 3D Model Reconstruction ........................................................................5 3.1 Camera Calibration ...............................................................................................5 3.3 Taking Photos for the Environment ......................................................................7 3.3 Building 3D Model Reconstruction ......................................................................7 CHAPTER 4 – Applying Different Security Levels ......................................................................10 4.1 Security First Level .............................................................................................10 4.1.1 User Authentication ..................................................................................10 4.1.2 Using X.509 Certificates ...........................................................................10 4.2 Security Second Level ........................................................................................11 4.2.1 Encryption of Connection Stream .............................................................11 4.3 Security Third Level ...........................................................................................14 4.3.1 Database Encryption .................................................................................14 4.3.2 Using AES Algorithm ...............................................................................15 CHAPTER 5 – Evaluation and Results .......................................................................................21 5.1 Mutual User Authentication ................................................................................21 5.2 Securing TCP Connection with SSL ...................................................................21 5.3 AES Encryption of Database ..............................................................................22 CHAPTER 6 – Discussion and Future Work ..............................................................................24 REFERENCES .............................................................................................................................27 APPENDICES Appendix A ...................................................................................................................................28 Appendix B ...................................................................................................................................30 Appendix C ...................................................................................................................................32 Appendix D ...................................................................................................................................33 Appendix E ...................................................................................................................................34

vii

Chapter 1 : Introduction

1- Problem Overview

Figure 15 : Overview

The system proposed by Ventura and T. Höllerer [1] is used to localize the client in a real time

when he takes a picture and sends it to a remote server which in turn extracts the features from the

sent picture and matches these features with a 3D reconstruction of a previously stored pictures of

the same environment. Next, the server estimates the best pose and sends the result back to the

client.

The main disadvantage of this system is that it uses an unsecure TCP connection between the

client and the sever. In other words, there is no secure connection area applied above the TCP layer

like SSL or TSL. If this system is used for police , military purposes or any sensitive work to

localize and track team members, it would be easy to exploit the system by the attacker to eavesdrop

2

on the communication between the two parties and possibly modify it for certain unauthorized

reasons .

Such systems should function in a more secure environment even if the system performance is

downgraded by the security and privacy enhancements integrated into the system. Moreover,

security and privacy should be addressed and taken into an account when designing localization and

tracking systems for military work or any environment which demands secure and private operation.

The system could be enhanced with different levels of security to prevent the possible threats that

may happen when using an unsecured TCP connection between the client and the remote server. The

connection stream between the two parties can be encrypted so the attacker cannot detect what is

being sent, even if he can eavesdrop on the connection. The system also could be modified in such a

way it could authenticate the two parties mutually before even starting to exchange data. There are

different ways to apply the user authentication, such as using a username and password to

authenticate the client by the server. Also, certificates for both client and server could be used to

mutually authenticate one another. The latter is our option to achieve user authentication in our

modified system.

2- Why to Secure VRLT , and Project Research Question :

• Localization and tracking system itself should be one of security products , for that reason it

should work in a very secure environment .

• One of the beigest reason as a motivation to enhance VRLT with more security levels is

terrorism . If such system is adopted by military or police forces in its current status , it will

be an easier for terrorists to capture the request and the response being transmitted , and

possibly alter the data .

• In general , it is good practice to secure the connection if a public channel is used as medium

of communication .

Research question : What are the overheads , in term of time consumption , of security

enhancements to a Visual Localization and Tracking systems .

3

3- Common Possible TCP Attacks : There are two types of attack ( active and passive) :

3.1 Active Attacks :

3.1.1 TCP Sequence Number Prediction Attack: When Client C and Server S creates a new

connections, an initial sequence number (ISN) generator is employed which selects a new 32 bit

ISN. This sequence number will be use for counting and tracking TCP segments being transmitted

during the session . These ISN’s will be agreed on upon the successful 3 Handshaking involved in

TCP connection creation Pan,Xiao [12] . It is easy to predict the TCP sequence number by the

attacker due to the natural of TCP/IP protocol , and with a help of sniffer software Pan,Xiao [12] ,

this attack could lead to connection hijacking attack .

3.1.2 TCP Session Hijacking Attack :It is possible for this attack to happen to TCP connection ,

attacker A could sniff the TCP connection between the client C and the server S , then A can inject

the connection with a segment to the server S spoofing the source address as the client C .

Eventually , the server S will accept the segment and update acknowledgment numbers . Also , the

attacker could apply Denial of service attack to the client to increase the chance of injecting the

connection with his segment . TCP session hijacking could also happen without sniffing the

connection between the client C and the server S , the attacker simply will guess the sequence

number in the current TCP segment and the port number , such hijacking is called a blind hijacking

Pan,Xiao [12] .

3.1.3 IP Spoofing : is the act of concealing the source IP address by forging the packet header with

different IP address , so it appears to the recipient of the packet that the sender is the latter address

not from the true location so the reply coming from the recipient will be routed to that IP address

instead of going to the real sender [13].

3.2 Passive Attacks :

3.2.1 Eavesdropping : is one of the basic attacks in order to track an individual user of his online

behavior. This involves the tracker capturing packets sent by the user in a particular network using a

packet sniffer software like Wireshark. Modio, TCPDump, Snort, Packet Filter (PF) etc. The tracker

can analyze the sender’s IP address and the destination IP address, the packet payload, which might

contain sensitive information like names, addresses and even phone numbers (Risso, F., &

Degioanni, L. 2001).

4

Chapter 2 : Related Work and Literature Review

Localization and tracking systems can be: indoors or outdoors. There have been a number of

researches conducted in this area. H. Kawaji , K. Hatada , T. Yamasaki , K. Aizawa [6] is an

example of indoor localization and Ventura and T. Höllerer [1] is an example of outdoor

localization.

Moreover, the tracking systems can be done through GPS signal like the system proposed by S.

Panzieri, F. Pascucci, and G. Ulivi, [8], Wi-Fi access points or using computer vision techniques for

example [ 2 , 3 , 4 , 5 ]. In our literature review we will consider existing computer vision-based

tracking systems to see if any previous papers addressed security and privacy in the system design.

To narrow down the research domain, we will not focus on all previosu works that apply to the

computer vision techniques ,since not all of them use a client-server concept to do tracking and

localization task. For example, the system of C. Arth, D. Wagner, M. Klopschitz, A. Irschara, and

D. Schmalstieg [7] performs image capture and localization on the same device. Instead, we will

focus on the systems which have two devices where one is the client and the other is the server.

To the best of our knowledge, none of these existing systems of interest apply or implement any

security and privacy techniques. For this reason, we intend to improve the system of Ventura and T.

Höllerer [1] in order to afford some security and privacy layers that will be proposed and

implemented. We will implement several measures: user authentication, encryption and decryption

during data transmission, database encryption, and extracting the features at the client side instead of

sending the whole picture to the server and letting the server extract the features .

When designing such systems with consideration for security, an important question is how secure

the designers want their systems to be . In other words, some of them may need only to authenticate

the users , others may need only to encrypt the connection , or they may only need to keep their

database encrypted . However, we will try to cover all such security considerations and measure how

they are affecting the performance of the system proposed by Ventura and T. Höllerer [1] .

5

Chapter 3 : Creating 3D Model Reconstruction

3.1 Camera Calibration : iPhone 6 plus’s camera was used in this project to take photos for the

outdoor environment . Most of today’s cameras come with some significant distortions , fortunately

such distortions are constant and could be corrected and remapped by a calibration [10] .

After the iPhone camera is calibrated , the following camera matrix should be obtained as an output:

Figure 16 : Camera Matrix , source [10]

As for fx and fy , they are the camera focal lengths . However , commonly axe x and axe y use the

same focal length with a certain ratio :

fy = fx * ratio ……………………….. ( 1 )

Ratio is usually equal to one so fy will be equal to fx [10] . That is , the focal length for both axes in

the iPhone 6 plus camera will be a single focal length . As for cx and cy , they are the optical centers ,

Xcenter and Ycenter , and they are expressed in pixel coordinates .

The calibration program basically calculates the camera matrix through geometric equations [10] .

The calibration accepts three different types of objects to do the calculations of the camera matrix :

1- Classical black-white chessboard

2- Symmetrical circle pattern

3- Asymmetrical circle pattern

The first type of the three above objects was chosen to calibrate the iPone 6 plus camera, several

photos of the chessboard have been taken , at least 30 photos , from different distance , angles . The

calibration program will read the configuration files where the number of inner corners ( cross of

6

rows and columns ) is define , the original chessboard that is provided by OpenCV has a width

(rows) six and the height (columns) nine .

Moreover , the configuration file has the option to choose which kind of object will be used to

calibrate the camera , in our case CHESSBOARD . Also , it contains where the photos are stored so

the calibration program can read them . Additionally , the number of how many photos to be

calibrated should be specified , at least 13 photos out of 30 photos should be calibrated to come up

with a good result of camera matrix .

The calibration program will try to detects the corners in the photo of CHESSBOARD based on the

configuration file . However , it is common the that the calibration could fail to detect such corners

in the photo , for this reason we should supply an enough number of photos to calibrate 13 photos (

in our case , we supplied 30 photos ) .

After running the calibration program with the supplied photos and the right parameters in the

configuration file , XML file will be created and it will contain the Camera Matrix as it is shown in

the following part of calibration program output ( the output_camera_calibration.xml ) :

<?xml version="1.0"?> <opencv_storage> <calibration_Time>"Mon Mar 16 06:05:54 2015"</calibration_Time> <nrOfFrames>16</nrOfFrames> <image_Width>3264</image_Width> <image_Height>2448</image_Height> <board_Width>6</board_Width> <board_Height>8</board_Height> <square_Size>30.</square_Size> <FixAspectRatio>1.</FixAspectRatio> <!-- flags: +fix_aspectRatio +fix_principal_point +zero_tangent_dist --> <flagValue>14</flagValue> <Camera_Matrix type_id="opencv-matrix"> <rows>3</rows> <cols>3</cols> <dt>d</dt> <data> 3.3407986643686490e+03 0. 1.6315000000000000e+03 0. 3.3407986643686490e+03 1.2235000000000000e+03 0. 0. 1.</data></Camera_Matrix>

7

3.2 Taking Photos for the Environment : Building database of 3D model is done as offline stage .

The 3D model database could be created using some tools that VRLT provides . Two different sizes

of database have been created in purpose of examining the effect of database growth on the

localization service that VRLT provides .

Regent Cir , the area between Kraemer Family Library and Lot 224 , in University of Colorado

at Colorado Springs was chosen to be an outdoor environment . The following steps show how

photos have been taken form the outdoor environment :

1- The start point was the west north corner of Lot 224 .

2- Walking on Regent Cir toward Lot 222 .

3- After each four steps of walk , three photos were taken ( appendix G gives an example for

these three photos for one stop :

• The first photo (Center ) was taken when looking straight .

• The second photo ( Right ) : step to right from the center , then the camera was

moved 45 degree to the right to cover more area in the right side , however there is a

30-40 percent common between the second photo and the first photo .

• The third photo ( Left ) : step left from the center . Then the camera was moved 45

degree to the left to cover more area in the left side , however there is a 30-40 percent

common between the third photo and the first photo.

4- Number of stops in the first database is 38 stops ( 114 photos ) , the second database adds

another 38 stops which increases the number of photos with another 114 photos ( double size of the

first database ) .

3.3 Building 3D Model Reconstruction : Figure (3) shows the flowchart of the steps that have been

followed to build the 3D model from VRLT :

8

Figure 17 Flowshart to build 3D from VRLT

• Process 1 : Extracting the features from each photo : The system uses SIFT to extract the

descriptors and detectors in each photo for feature matching . The tool used in this step is

ExtractSIFT . Here is an example for using this tool :

Figure 18 : Feature Extraction

• Process 2 : Matching the features : In this step , the features extracted in one photo is

matched to the same feature in another photo . The feature matching is done linearly , the

tool used in this step is LinearMatch . Here is an example for using this tool :

9

Figure 19 : Matching features linearly

• Process 3 : The matching features are then organized into tracks . The tool used for this step

is MakeTrack . Here is an example for using this tool :

Figure 20 : Making tracks

• Process 4 : Bootstrapping two photos together using the tool Bootstrap to start reconstructing

our database . Here is an example for using this tool :

Figure 21 Bootstrapping two images

• Process 5 : Adding the remaining photos to the bootstrapped photos . The tool used for this

step is AddImages . Here is an example for using this tool :

Figure 22 : Adding the remaining images

• Process 5 : the normal vectors are estimated to come up with our 3D model reconstruction .

The tool used for this step is EstimateNormals . Here is an example for using this tool :

Figure 23 : Reconstruction the 3 D .

10

Chapter 4 : Applying Different Security Levels :

4.1 Security First Level :

4.1.1 User Authentication :

VRLT uses no authentication mechanisms at all . That is , as long as the server used in the system to

localize and track the user is running , it can accept any request from any user in the Internet . It can

be easily seen that the system is not secure since it accepts and serves any request from the Internet .

However , VRLT can be enhanced with any authentication mechanism in order to be assure that the

server is serving the right user . Also , users looking for localization service from a specific server

want to be assured they are connected to the right server .

There are several options that can provide VRLT system with a user authentication mechanism such

as : authenticating the users based on their IP addresses , Authenticating the users using usernames

and passwords , and authenticating the users through certificates signed by a trusted third party . The

latter option is applied to VRLT as user authentication mechanism . In the following section ,

certificate of type X.509 is explained in more details .

4.1.2 Using X.509 Certificates :

Most of the network security applications such as : S/MIME , IP sec , and SSL use X.509

certificates to present their identities to each other Stalling[9] . The main purpose of using X.509

certificates is to authenticate the users when they want to be communicating securely through a

public channel .

X.509 certificates are public key cryptosystem so they are inefficient to be used as direct encryption

to a big data Stalling[9] . However , they are used to encrypt a relatively small blocks , and to

exchange the secret keys between the communicating parties after they become authenticated to each

other . The secret keys exchanged can be then used to encrypt the data by one of symmetric

cryptosystems such Advanced Encryption System AES .

11

Figure 24 : Certifacet Signing

Each certificate has three main sections : (1) User information including the public key and ID , (2)

Information about Certificate Authority ,CA (3) and the Signature . Each user willing to have a

signed certificate has to create a pair of two keys ( private and public key ) , and sends only his

public key along with other public information such as the owner name and his ID to one of known

CAs . Then , CA in turn will hash the user information and encrypt the hash code by the CA private

key . The encryption of the hash code is actually the signature , the third section in each certificate ,

attached along with the other two sections in the certificate .

4.2 Security Second Level :

4.2.1 Encryption of Connection Stream: In this level , the connection between the client and the

server is secured from eavesdropping attacks. The security-enhanced VRLT has a client application

and server application that were built to provide two versions : The first one is to send the image

from the client to the server using a normal TCP connection , the second version is to encrypt the

connection between the client and the server by OpenSSL

Encryption algorithm can be defined to be used for the connection , however , the default encryption

algorithm of OpenSSL is used to encrypt the connection stream. The default algorithm is AES 256

SHA . Wireshark is used to capture the connection when it is unencrypted ( normal TCP ) and

encrypted (SSL above the TCP) . The following the an example of the encrypted and unencrypted

packet .

User  Information  (  Publick  key  ,  ID  )  

Hashing  the  user  

information  

Encrypt  the  the  hashing  code  by  the  

CA  

12

First , reading the image in a hexadecimal as in shown in figure 11 . ( the first 94 bytes from the

image are highlighted :

Figure 25 : Image bytes in Hexadecimal

Second : Figure 12 is the screenshot of Wireshark of first bytes of the image when it was not

encrypted ( normal TCP connection ) :

13

Figure 26 : Wireshark catpture for normal TCP

Third : Figure 13 is the screenshot of Wireshark of the first bytes of the image when it was encrypted

( TCP/SSL connection ) :

14

Figure 27 : Wireshark catpture for TCP/SSL

4.3 Security Third Level :

4.3.1 Database Encryption :

The database contains photos that were taken for the outdoor environment , and the features

extracted by the system along with the key files . In terms of security , it is important to keep the

database protected from unauthorized accesses . Two different sizes of database were created as

follows :

• Database 1 (DB1): has 114 photos , 114 features data files , 114 keys data files ,and 5 xml

15

files .

• Database 2 (DB2): has 228 photos 228 photos , 228 features data files , 228 keys data files

,and 5 xml files .

The VRLT system will read the 3D model through a reconstruction file which is an xml file . The

database is always kept in an encrypted form, and they will be decrypted as long as the user who is

trying to get connected to the server is authenticated by the server . AES algorithm is used as an

de/encryption system .

The encryption and decryption process take much time when the database gets larger . For this

reason , two different sizes of database (DB1 , DB2 ) are used to examine the effect of database

encryption on the performance of the system .

Moreover , two versions of encryption and decryption are used for our database encryption . The

first version is CPU based , whereas the second version is GPU based . The GPU version is used to

see how far the performance , in term of time consumption , could be improved when dealing with

large data .

4.3.2 Using AES Algorithm :

AES is a symmetric block cipher that was accepted to be published by National Institute of Stansards

Technology (NIST) in 2001 , and it was chosen to replace the DES as the approved standard for a

wide range of applications Stallings[9] .

The AES structure is complicated in comparison to the public key cryptosystems such as RSA …. .

AES structure has a number of rounds , the number of rounds depends on the key length . The key

length that AES could be used is : h16 bytes (128 bits ) , 24 bytes (192 bits) , or 32 bytes (256 bits) .

The number of rounds is 10 , 12 , or 14 when the key length is 16 bytes , 24 bytes , 32 bytes

respectively . Each round has number of stages that do the transformation ( permutation and

substitution ) of the input data , the types of stages will be explained in a little bit details in this

section . There is also an initial round which is sometimes called round 0 , this round has only one

stage called ADD ROUND KEY .

16

The input of plaintext to the AES is always a block of 16 bytes ( 128 bit-block ) , these 16 bytes are

arranged in a square matrix of 4 columns and 4 rows , the first four bytes will occupy the first

column , the second four bytes will occupy the second column , the third four bytes will occupy the

third column , and the fourth four bytes will occupy the last column as shown in the following figure

(14) :

Figure 28 : a square matrix of plaintext

The key will be expanded based on the key length itself , as mentioned previously that the key length

specifies the number of rounds . A length of 16 bytes from the key expansion will be supplied to

each round . In case of 16 byte length , the key will be expanded to 176 bytes whereas 24 bytes and

32 bytes will be expanded to 208 bytes , and 240 byes respectively . The Table … shows all the

possible parameters that AES can take :

17

Table 1 : AES parameters . source [9]

Each round has four stages ( Stage 1 : Substitute Bytes , Stage 2 : Shift Rows , Stage 3: Mix

Columns , and Stage 4 : Add Round Key ) unless the initial round has only Add Round Key and the

last stage has three stages (Stage 1 : Substitute Bytes , Stage 2 : Shift Rows , and Stage 3: Add

Round Key ) .

In term of Security , Add Round Key stage adds security to the AES that is why the AES starts and

ends with this stage . The other stages dose not use the key , and they are reversible without

knowledge of key , these stages add no security however they together provide confusion , diffusion

, and nonlinearity Stallings[9] .

The stages in each round are used for permutation and substitution What the four stages do in each

round as described as follows :

1- Substitute bytes: This stage is called SubBytes S-box is used in this stage to perform a byte-by-

byte substitution of the block Stalling[9]

The corresponding function prototype for this task as follows :

A . In CPU version :

18

B. In GPU version :

2- Shift Rows : This stage is called ShiftRows , and it is a simple permutation Stalling [9].

The corresponding function prototype for this task as follows :

A . In CPU version :

B. In GPU version :

3- Mix Columns: This stage is called MixColumns , and it is a substitution that makes use of

arithmetic over GF(28) Stallings[9] .

The corresponding function prototype for this task as follows :

A . In CPU version :

B. In GPU version :

19

4- Add Round Key : This stage is called AddRoundKey , and it is a simple bitwise XOR of the

current block with a portion of the expanded key Stallings [9] .

The corresponding function prototype for this task as follows :

A . In CPU version :

B. In GPU version :

The AES decryption process has the same number of rounds as the encryption process has .

However , each round has the following stages ( Stage 1 : Inverse Shift Rows , Stage 2 : Inverse

substitute Bytes , Stage 3 : Add Round Key , and Stage 4 : Inverse Mix Columns ) unless the initial

round has only Add Round Key and the last stage has three stages (Stage 1 : Inverse Shift Rows ,

Stage 2 : Inverse substitute Bytes , Stage 3 : Add Round Key) .

The AES decryption uses the key expansion in reverse . That means the last 16 bytes in the key

expansion will be used in the initial round in the decryption process , and the first 16 bytes in key

expansion will be used in the last round .

The four stages in each round in the decryption process are simply the inverse to the stages in the

decryption process . The following figure (15) shows the whole picture of how the encryption and

decryption process work in AES . The encryption is on the left side and starts from up to down

whereas the decryption process is on the right side and starts from down to up .

20

Figure 15 : Overview of AES algorithm source [9]

21

Chapter 5 : Evaluation and Results :

Due to the variation of the network measurement factors ( available bandwidth , processing time at

server and client , and transmission time ) , measuring the time overhead of each level that is applied

to VRLT is not easy , and requires to run the system several times , then calculating the average time

. The average time for VRLT to run without any of these security levels is 1636.76 when using DB1

whereas it is 1831.8 when using DB2 . The following sections will talk about each level in more

details .

5.1 Mutual User Authentication :

In this step , no encryption is applied , hence the connection is still plaintext and easily could be

eavesdropped by the attacker . However , the authentication for both peers ( client and server ) is

applied by using SSL connection . Authentication process happens during the 3 Handshaking phase

in SSL connection ( Authenticating the client at the server side and the server at the client ) . The

average time of authentication process when using DB1 is 37.58 ms and 31.72 ms when using DB2 (

the difference is due to the status of the network though they should be the same ) . Equation (2) is

used to calculate the overhead of 3Handshaking phase to the time needed to complete the

localization request in normal TCP .

Overhead = 3 Handshaking average time / The average time needed for the whole process in normal TCP

Equation * 100 ……………………………………………………………………………………………………… (2)

As a result , the overhead of authentication process for DB1 and DB2 is still minimal and has small

effects on the whole localization process as it can be seen in the following :

DB1 : Overhead = (37.58/1636.76)*100 = 2.29 %

DB2 : Overhead = ( 31.72 / 1831.8 ) * 100 = 1.7 %

5.2 Securing TCP Connection with SSL :

After the client is authenticated by the server as well as the server is authenticated at the client side ,

the data being transmitted between the client and the server is secured “encrypted” . Surprisingly ,

this level of security has almost no effect on the localization process . The VRLT has been tested so

many times to see the difference between sending the data securely and not securely . As a result ,

22

there is about 22 times of 50 times that normal TCP connection (unencrypted) took much time than

SSL connection ( encrypted ) . On average , sending the data in normal TCP when DB1 is used is

56.12 ms whereas in SSL is 61.13 , hence the difference is about 5 ms . Based on Equation (2) , the

overhead of adding this security level to the normal localization process when DB1 is used is :

Overhead = (5/1636.76)*100 = 0.3 %

Moreover , the average time of sending data in normal TCP when DB2 is used is 60.24 ms whereas

it is 63.34 ms in SSL connection ( the difference between DB1 and DB2 is due to the status of the

network though they should be the same ) . Based on Equation (2) , the overhead of adding this

security level to the normal localization process when DB2 is used is :

Overhead = (3/1831.8)*100 = 0.1 %

DB2 is actually created to examine the effect of database growth on the localization service that

VRLT provides . DB2 is a double size of DB1 , the average total time when DB2 is used is

compared to the average total time when DB1 is used . Anything else (Sending time , 3 Handshaking

) is the same in either cases ( DB1 or DB2) . The following table shows the comparison between the

average time to handle a localization request in DB1 and DB2 used in VRLT system :

Table 2: Comparison average total time between DB1 and DB2

Appendix A shows the comparison between the Normal TCP connection and the SSL connection for

the 50 runs . In Appendix A , Sending T means the time needed to send the data from peer to the

other peer, Total means the total time needed to make a request from the client till the pose sent back

to the client from the server. 3HS means 3 Handshaking for SSL Connection .

TCP TCP/SSL

Database 1 ( DB1) 1636.76 ms 1656.26 ms

Database 2 ( DB2) 1831.8 1877.45 ms

23

Appendix B shows the times needed for the 50 runs to complete the localization request for both

DB1 and DB2 in TCP connection and SSL connection .

5.3 AES Encryption of Database :

Encrypting and decrypting the database definitely come with overhead. This overhead will be

increased as long as the database increases as we will see in the following results . Table 3 shows the

average times of 20 runs for ( decryption process , decryption process , and total time needed to

handle the request ) when DB1 is used . Tables 4 shows the average time of another 20 runs when

DB2 is used . The average total time to handle the request dose not include the encryption time since

it only needs to consider the decryption time because the database is already encrypted and needs to

be decrypted to complete the localization process. After, serving the request , the database will be

encrypted again but it has no effects on measuring the localization process .

Table 3 : Comparison between CP and GPU when dealing with DB1

Version/process Encryption Decryption Total Time

CPU 1010.9 ms 2572.16 ms 4252.82 ms

GPU 383.975 ms 551.35 ms 2305 ms

24

Table 4: Comparison between CP and GPU when dealing with DB2

From table 3 and table 4 : it is clear that when the database is increased double as in table 4 (DB2),

it needs roughly an addition time of 54 % of the average time that DB1 needs to handle the request

when CPU is used in database decryption , whereas it needs 20 % as an addition time of the average

time that DB1 needs to handle the request when GPU is used . The addition times are calculated

based on the following equation

Addition time ={ ( Database 2 total time – Database 1 total time ) / Database 1 total time } * 100

……………………………………………………………………………………………………………………………… (3)

As result, We observed the following :

1- GPU is about 3 times faster than CPU in encryption process , and it is also 4 times faster than

CPU in decryption process when DB1 is used .

2- GPU is about 2.5 times faster than CPU in encryption process , and it is also 5 times faster

than CPU in decryption process when DB2 is used .

Appendix C shows the details of 20 runs for de/encrypting DB1 , whereas appendix D shows the

details of 20 runs when DB2 is used .

Chapter Six : Discussion and Future Work :

We can summaries the overhead as the following (our default database is DB1):

Version/process Encryption Decryption Total Time

CPU 1931.97 ms 4812.69 ms 6563.795 ms

GPU 719.27 ms 921.2 ms 2781.16 ms

25

• VRLT with a normal TCP connection needs 1636.76 ms to finish the localization service .

• VRLT needs another 37.58 ms for mutual user authentication , and negotiation between the

server and the client for which symmetric cipher to use for data transmission encryption .

The overhead is 2.29 % as an addition time will be needed to the normal TCP version .

• VRLT needs an addition of 5 ms to send the data securely through SSL ( encrypted ) . The

overhead of sending the data as encrypted to the normal TCP connection with the first

security level ( user authentication ) is 0.3 % .

• VRLT needs 2572.16 ms to decrypt the database when CPU is used , whereas it needs only

551.35 ms when GPU is used . The overhead of database decryption to the (

authentication + connection stream encryption )-enhanced VRLT is as follows: in CPU ,

it is 155 % , in GPU , it is 33 % .

However , here is the effects of the database growth on the localization process , and also how far

the GPU could improve the last security level ( Database Encryption ):

• VRLT with a normal TCP connection needs 1831.8 ms to finish the localization service .

• VRLT needs another 31.72 ms for mutual user authentication , and negotiation between the

server and the client for which symmetric cipher to use for data transmission encryption .

The overhead is 1.7 % as an addition time will be needed to the normal TCP version .

• VRLT needs an addition of 3 ms to send the data securely through SSL ( encrypted ) . The

overhead of sending the data as encrypted to the normal TCP connection with the first

security level ( user authentication ) is 0.1 % .

• VRLT needs 4812.69 ms to decrypt the database when CPU is used , whereas it needs 921.2

ms when GPU is used . The overhead of database decryption to the ( authentication +

connection stream encryption )-enhanced VRLT as follows : in CPU , it is 262 % , in

GPU , it is 50 % .

It is clear , The first two levels ( user mutual authentication and connection stream encryption ) add

minimal overhead to the normal TCP version , and this overhead decreases when the database grows

up , however the system is secured with these two essential security levels . As for the database

encryption , it adds much overhead specially when CPU version is used and the database grows .

However using GPU helps reduce the time overhead . Based on these results , it is strongly

recommended to add the first two security levels to VRLT ( user mutual authentication and

26

connection stream encryption ) since it is has small effects on the localization service and this effect

decreases when the database is increases . As for the third security level , it is obvious that it adds so

much overhead , however GPU version will be needed in case of demand of security is so high and it

is OK to make the response delayed .

As for future work , the optional security level ( extracting the features at the client side and sending

the features to the server instead of sending the image itself ) could add more security to the system

, also GPU could be used to extract the image features instead of using CPU version which will

improve the system in term of time consumption . Moreover , the database could be organized in any

DBMS instead of using a directory where every thing is saved in that directory and read through an

xml file .

27

References :

1-­‐  J.  Ventura  and  T.  Ho  ̈llerer,  “Wide-­‐area  Scene  Mapping  for  Mobile  Visual  Tracking,”  in  Int.  Symposium  on  Mixed  and  Augmented  Reality  (ISMAR),  2012    

2-­‐  D.  Wagner,  A.  Mulloni  T.  Langlotz,  and  D.  Schmalstieg.  Real-­‐Time  Self-­‐Localization  from  Panoramic  Images  on  Mobile  Devices.  In  VR  ,  pages  211-­‐218  

3-­‐  G.  Klein  and  D.  Murray.  Parallel  Tracking  and  Mapping  on  a  Camera  Phone.  In  ISMAR,  pages  83–86,  2009.  

4-­‐  R.  Behringer,  “Registration  for  Outdoor  Augmented  Reality  Applications  Using  Computer  Vision  Techniques  and  Hybrid  Sensors,”  Proc.  Virtual  Reality  Ann.  Int’l  Symp.,  pp.  244-­‐251,  1999.  

5-­‐  I.  Skrypnyk  and  D.  Lowe,  “Scene  Modeling,  Recognition  and  Tracking  with  Invariant  Image  Features,”  Proc.  Int’l  Symp.  Mixed  and  Augmented  Reality  (ISMAR  ’04),  pp.  110-­‐119,  2004.  

6-­‐  H.  Kawaji  ,  K.  Hatada  ,  T.  Yamasaki  ,  K.  Aizawa    “Image-­‐based  Indoor  Positioning  System:  Fast  Image  Matching  using  Omnidirectional  .  Panoramic  Images”    Proceedings  of  the  1st  ACM  international  workshop  on  Multimodal  pervasive  video  analysis  Pages  1-­‐4?  

7-­‐  C.  Arth,  D.  Wagner,  M.  Klopschitz,  A.  Irschara,  and  D.  Schmalstieg.  Wide  area  localization  on  mobile  phones.  In  Mixed  and  Augmented  Reality,  2009.  ISMAR  2009.  8th  IEEE  International  Symposium  on,  pages  73–82.  IEEE,  2009.    

8-­‐  S.  Panzieri,  F.  Pascucci,  and  G.  Ulivi,  “An  outdoor  navigation  system  using  GPS  and  inertial  platform,”  IEEE/ASME  Trans.  Mechatron.,  vol.  7,  no.  2,  pp.  134–142,  Jun.  2002.    

9-­‐  W.  Stallings  “  CRYPTOGRAPY  AND  NETWORK  SECURITY  PRINCIPLES  AND  PRACTICE  “  ,  5th  ed  .  Upper  Saddle  River:  Pearson  Education,INc,  2006  .  

10-­‐  www.opencv.rog  ,  “Camera  Calibration  with  OpenCV”  

11-­‐  I.  Dubrawsky,    HOW  TO  CHEAT  AT  Securing  Your  Network,  Burlington:  Amorette  Pedersen  ,  2007  .  

12  –  Y.  Pan,  Y.  Xiao  ,  Security  in  Distributed  and  Networking  Systems  ,  1st  ed  .  Singapore:  World  Scientific  Publishing  Co.  Pte.  Ltd  2007  .  

13  –  Z.  Duan  ,  X.  Yuan  ,  J.  Chandrashekar  “  Controlling  IP  Spoofing  Through  Inter-­‐Domain  Packet  Filters  “  IEEE  INFOCOM  ,  2006  .  

28

APPENDEX A

TCP  Conn   With  DB1    

SSL  Conn   With  DB1      Sending  T   Total  

 3  HS   Sending  T   Total  

 61.304   1617.64    

21.814   40.555   1578.15    42.273   1706.4  

 15.218   42.521   1779.17  

 44.161   1721.62    

14.183   54.751   1591.94    49.732   1632.24  

 25.997   54.87   1621.06  

 30.803   1473.56    

25.34   43.191   1624.55    44.009   1547.87  

 20.355   84.513   1781.34  

 38.171   1612.07    

27.088   59.512   1617.05    62.532   1664.16  

 32.199   50.261   1653.78  

 63.477   1625.09    

51.114   92.866   1635.72    39.593   1535.28  

 30.873   49.095   1732.56  

 38.674   1636.15    

41.075   80.758   1665.53    34.602   1627.3  

 31.245   77.18   1722.47  

 64.625   1614.63    

111.401   82.135   1783.32    43.322   1613.8  

 34.262   60.771   1635.99  

 44.749   1650.37    

51.895   47.717   1725.12    79.801   1609.9  

 33.596   56.066   1732.97  

 47.828   1636.88    

52.008   74.04   1616.24    36.64   1822.57  

 45.87   58.172   1719.26  

 31.633   1510.26    

60.504   78.241   1736.75    88.106   1724.32  

 40.157   61.025   1620.65  

 52.621   1611.79    

24.791   51.913   1701.93    47.18   1634.78  

 187.372   98.131   1639.07  

 64.559   1653.52    

42.638   42.077   1716.03    83.479   1705.58  

 61.647   28.468   1721.38  

 37.946   1606.75    

16.548   51.47   1593.89    63.426   1818.03  

 28.178   40.302   1685.34  

 73.541   1486.39    

30.935   91.713   1625.99    45.279   1610.18  

 14.637   67.51   1770.39  

 74.53   1731.46    

19.438   48.751   1601.81    48.781   1720.07  

 11.95   46.409   1495.28  

 78.352   1721.28    

40.996   71.873   1636.09    88.206   1599.37  

 43.668   74.53   1561.39  

 50.823   1605.92    

12.986   42.423   1591.61    48.93   1713.91  

 23.922   63.734   1616.76  

 46.025   1579.06    

22.244   52.899   1658.53    67.471   1613.4  

 58.041   68.529   1650.55  

 46.167   1618.77    

14.35   50.656   1671.14    39.387   1616.03  

 15.339   70.605   1676.54  

 66.012   1631.68    

37.613   46.34   1726.1    68.269   1653.07  

 35.866   57.543   1694.53  

29

92.044   1674.27    

67.328   55.082   1634.49    62.289   1615.28  

 20.745   44.893   1608.84  

 64.132   1641.7    

24.529   25.157   1591.88    60.156   1600.02  

 43.81   21.231   1517.91  

 62.499   1623.24    

30.874   54.854   1621.82    51.991   1693.16  

 13.838   52.94   1597.34  

 79.9   1614.61    

33.536   44.681   1619.67    65.404   1554.51  

 33.381   48.415   1644.29  

 44.027   1688.1    

63.626   73.969   1745.82    46.975   1620.15  

 38.311   72.827   1622.93  

               56.12872   1636.7638   AVERAGE     37.58662   58.1633   1656.2592   AVERAGE  

30

APPENDEX B

TCP  Conn   With  DB1    

SSL  Conn   With  DB1      Sending  T   Total  

 3  HS   Sending  T   Total  

 61.304   1617.64    

21.814   40.555   1578.15    42.273   1706.4  

 15.218   42.521   1779.17  

 44.161   1721.62    

14.183   54.751   1591.94    49.732   1632.24  

 25.997   54.87   1621.06  

 30.803   1473.56    

25.34   43.191   1624.55    44.009   1547.87  

 20.355   84.513   1781.34  

 38.171   1612.07    

27.088   59.512   1617.05    62.532   1664.16  

 32.199   50.261   1653.78  

 63.477   1625.09    

51.114   92.866   1635.72    39.593   1535.28  

 30.873   49.095   1732.56  

 38.674   1636.15    

41.075   80.758   1665.53    34.602   1627.3  

 31.245   77.18   1722.47  

 64.625   1614.63    

111.401   82.135   1783.32    43.322   1613.8  

 34.262   60.771   1635.99  

 44.749   1650.37    

51.895   47.717   1725.12    79.801   1609.9  

 33.596   56.066   1732.97  

 47.828   1636.88    

52.008   74.04   1616.24    36.64   1822.57  

 45.87   58.172   1719.26  

 31.633   1510.26    

60.504   78.241   1736.75    88.106   1724.32  

 40.157   61.025   1620.65  

 52.621   1611.79    

24.791   51.913   1701.93    47.18   1634.78  

 187.372   98.131   1639.07  

 64.559   1653.52    

42.638   42.077   1716.03    83.479   1705.58  

 61.647   28.468   1721.38  

 37.946   1606.75    

16.548   51.47   1593.89    63.426   1818.03  

 28.178   40.302   1685.34  

 73.541   1486.39    

30.935   91.713   1625.99    45.279   1610.18  

 14.637   67.51   1770.39  

 74.53   1731.46    

19.438   48.751   1601.81    48.781   1720.07  

 11.95   46.409   1495.28  

 78.352   1721.28    

40.996   71.873   1636.09    88.206   1599.37  

 43.668   74.53   1561.39  

 50.823   1605.92    

12.986   42.423   1591.61    48.93   1713.91  

 23.922   63.734   1616.76  

 46.025   1579.06    

22.244   52.899   1658.53    67.471   1613.4  

 58.041   68.529   1650.55  

 46.167   1618.77    

14.35   50.656   1671.14    39.387   1616.03  

 15.339   70.605   1676.54  

 66.012   1631.68    

37.613   46.34   1726.1    68.269   1653.07  

 35.866   57.543   1694.53  

31

92.044   1674.27    

67.328   55.082   1634.49    62.289   1615.28  

 20.745   44.893   1608.84  

 64.132   1641.7    

24.529   25.157   1591.88    60.156   1600.02  

 43.81   21.231   1517.91  

 62.499   1623.24    

30.874   54.854   1621.82    51.991   1693.16  

 13.838   52.94   1597.34  

 79.9   1614.61    

33.536   44.681   1619.67    65.404   1554.51  

 33.381   48.415   1644.29  

 44.027   1688.1    

63.626   73.969   1745.82    46.975   1620.15  

 38.311   72.827   1622.93  

               56.12872   1636.7638   AVERAGE     37.58662   58.1633   1656.2592   AVERAGE  

32

APPENDEX C

 CPU  /  DB1  

     GPU  /  DB1  

                   Total   Decryption   Encryption  

 Total   Decryption   Encryption  

 4385.03   2639.36   1029.43    

2106.2   539.08   360.112    4477.83   2835.67   1052.17  

 2199.18   599.166   377.008  

 4533.51   2685.04   1019.77    

2200.48   494.888   367.934    4350.62   2719.68   1101.06  

 2186.47   569.598   403.979  

 4350.44   2723.43   1079.88    

2238.2   493.411   420.392    4182.66   2560.39   1094.09  

 2487.75   529.319   405.558  

 4299.18   2651.67   1029.53    

2328.23   577.154   376.874    4315.7   2504.88   992.637  

 2283   560.295   372.826  

 4194.18   2435.98   949.142    

2473.6   583.867   388.734    4228.64   2686.57   981.562  

 2087.67   551.428   341.011  

 4348.74   2477.8   991.907    

2297.61   499.05   368.409    4237.07   2582.36   972.267  

 2202.53   551.656   328.274  

 4100.63   2452.27   948.555    

2219.54   555.562   364.976    4029.42   2463.58   970.697  

 2248.78   579.137   393.403  

 4000.67   2459.16   998.682    

2370.21   577.297   397.543    4141.32   2479.67   957.753  

 2654.4   506.995   407.862  

 4597.94   2612.97   950.968    

2348.96   584.228   402.149    4027.45   2456.67   1071.76  

 2577.82   561.959   379.046  

 4028.6   2454.44   994.832    

2449.12   532.701   407.148    4226.9   2561.76   1031.53  

 2141.15   580.212   416.266  

                 4252.8265   2572.1675   1010.9111   AVERAGE   2305.045   551.35015   383.9752   AVERAGE  

33

APPENDEX D

 CPU  /  DB2  

     GPU  /  DB2  

                   Total   Decryption   Encryption  

 Total   Decryption   Encryption  

 6571.35   4739.34   1980.14    

2673.15   909.437   742.516    6509.48   4733.23   1926.76  

 2769.92   971.824   620.587  

 6691.22   4813.73   2163.22    

2853.2   869.072   652.781    6609.46   4918.26   1871.07  

 2613.24   856.677   746.193  

 6368.9   4750.58   1853.36    

2813.67   907.251   758.37    6552.12   4780.52   1860.09  

 2959.81   907.076   715.443  

 6448.41   4730.78   1845.99    

2804.66   920.212   753.906    6471.17   4738.87   1831.31  

 2722   975.859   723.778  

 6477.66   4820.85   2200.94    

2591.31   983.095   621.91    7003.25   5210.09   2023.29  

 2698.75   900.031   703.321  

 6838.14   5036.27   2011.13    

2758.63   881.452   747.811    6429.99   4779.86   1855.76  

 2594.16   868.075   722.938  

 6407.86   4694.59   1875.08    

2985.07   997.91   714.49    6476.85   4789.82   1866.8  

 2956.11   948.994   732.845  

 6389.33   4750.87   1825.6    

2741.87   920.804   756.211    6567.86   4744.67   1882.65  

 3083.74   930.135   768.618  

 6576.57   4766.81   1842.31    

2764.34   957.661   741.406    6507.84   4822.78   1992.7  

 2643.99   874.091   666.164  

 6788.55   4857.4   2025.17    

2852.84   948.892   758.692    6589.89   4774.47   1905.94  

 2742.69   895.642   737.482  

                 6563.795   4812.6895   1931.9655   AVERAGE   2781.1575   921.2095   719.2731   AVERAGE  

34

APPENDEX E

The firs Photo ( Center )

The second photo ( Left )

The third photo ( Right )