Parking Enforcement via Drones User Manual - CS-CSIFParking Enforcement via Drones User Manual Authored by: Mark Machado Kelvin Lu Alexander Sergian Baotuan Nguyen Client: Professor

Parking Enforcement via Drones User Manual

Authored by: Mark Machado Kelvin Lu Alexander Sergian Baotuan Nguyen Client: Professor Ilias Tagkopoulos

ECS 193, Winter/Spring 2016

Preface This project was completed as part of the undergraduate senior design project for the Department of Computer Science at the University of California, Davis. The project spanned 2 quarters and involved 4 students who worked closely with their client, Professor Ilias Tagkopoulos, and faculty mentor, Professor Xin Liu. In doing this project, we wanted to explore ways of automating city parking enforcement. Currently in the city of Davis, parking enforcement involves human officers who routinely chalk tires to keep track of how long cars have been parked and citing if necessary. This is costly and error prone. We thus envisioned a fully automated system where drones would fly rounds through predetermined routes to monitor parking. Specifically using the DJI Phantom 3, we would capture images of each vehicle as well as location data and time and send it off to a server to be logged. We would provide a web interface for authorities to monitor and process any possible citations. We originally intended for this project to be a completely automated system. However, due to limited capabilities of the Phantom 3 and a concern for safety, we were not able to incorporate autonomous flight. Unlike the recently released DJI Phantom 4, which has intelligent obstacle avoidance builtin, the Phantom 3 did not have such provisions. This proved dangerous, as we had a few accidents during development. After a discussion with our client, we decided in would be best to leave out this feature. Thus, for reasons described above, our final product, requires human control of the quadcopters. In the end, we have developed a system that allows the Phantom 3 drone to upload images along with GPS data and time from each capture are sent off to our server for processing and logging. Our server runs license plate recognition as well as a web interface for users to filter through and process citations.

Parking Enforcement via Drone User Guide Page 2

Acknowledgements This work was supported by the NSF CAREER award CCF1254205 to Dr. Ilias Tagkopoulos and the Department of Computer Science, University of California, Davis. Thank you Professor Ilias Tagkopoulos and Professor Xin Liu for your support and guidance on this project.


Table of Contents 1. Installation and Setup

a. Android Client for Drone b. Server Components

2. User Guide a. Android Client for Drone b. Web Interface Functionality

3. Troubleshooting 4. FAQ 5. Contact 6. Appendix (Requirements Document)


1. Installation and Setup a) Android Client for Drone

The first component you will need to install is our Patrol Android Application. It servers as a client for the drone to communicate to our server. It allows you to control the camera on the DJI Phantom 3 and will send each captured image along with GPS coordinates and timestamp to our server for processing. High Level Overview of Android Application:

Installation The app can be run using the provided .apk file or by cloning the Github repository and pointing Android Studio to the /mobile/ folder with the cloned repository. Android Studio should take care of internal configurations itself. You just need to build the code within Android Studio and target it to your Android device. (Make sure your device has USB debugging enabled under developer options)


DJI SDK Key In order for Andorid DJI SDK functionality to work, you need to create a DJI developer account and obtain your own SDK key for this application. You then need to copy and paste the api_keys.xml.template file in /res/values/ and call it api_keys.xml. You then need to insert the SDK key within that file, as directed by the comments. There should be a string defined as such: <string name="dji_key"></string> Token In order to successfully authenticate the app, you need to have a token file on your Android system called auth.txt in the /ParkingEnforcement/config/ folder of your Android device. These are defined in the res/values/string.xml file as such: <string name="token_dir">/Parking_Enforcement/config/</string> <string name="token_filename">auth.txt</string> NOTE: You will need to get an auth.txt file in order for your app to authenticate properly with the server. These can be generated according to server instructions. Additionally, you must manually create these directories (Parking_Enforcement/DJI_Plate_Captures/ and Parking_Enforcement/config/) in Android, otherwise the app will crash. Lastly, you might want to change the Android package name to reflect your organization’s name. URLs In the app’s /res/values/ folder, you should find a file called string.xml. In here are the following defined urls for getting an authentication token and uploading a picture. Note these for reference: <string name="get_token_url">http://10.42.0.1:8000/authenticate_drone</string> <string name="upload_pic_url">http://10.42.0.1:8000/droneimages</string> Also, for reference, captured images are stored in the /ParkingEnforcement/DJI_Plate_Captures/ folder, as also defined in /res/values/string.xml: <string name="capture_dir">/Parking_Enforcement/DJI_Plate_Captures/</string>


b) Setup of the Server Components and Dependencies The system can be seen as an orchestrator between the several components, such as a database, ALPR computer vision library, message queue, and the DJI Phantom proxy. We assume a Linux (Ubuntu 14.04 LTS) environment, users with other environments should be easily able to find analogues in the specific instructions. This section will outline the steps necessary to install each part. Setting up the Python development environment and bits The primary codebase is in Python 2.7. Several Python tools are necessary for the development, testing, and operation of the system. Python 2.x and its core utilities are commonly installed with several operating systems. For Ubuntu 14.04 LTS, Python 2.7 is already included with the distribution. The easy_install utility for installing Python packages can be setup through an installation script provided by the Python Packaging Authority maintainers. wget https://bootstrap.pypa.io/ez_setup.py O | sudo python

The pip Python package manager is used to manage project dependencies. Another setup script can be used for this. wget https://bootstrap.pypa.io/getpip.py O | sudo python

Git is used for the indevelopment version control system, as well as a utility to download projects from distributed repositories. It can be gotten through through Aptitude on Debianbased operating systems.

sudo aptget install gitall

Lastly, the virtualenv Python tool is needed to manage the project’s development and operational environment, which we can now install through pip.

sudo pip install virtualenv

Setting up OpenALPR and its Python bindings OpenALPR is an open source project made to provide computer vision implementations of a license plate detection and recognition system. We mostly treat and utilize the OpenALPR library as a black box.


First, we will use Aptitude to install the dependencies of the OpenALPR project. sudo aptget install libopencvdev libtesseractdev git cmake buildessential

sudo aptget install libleptonicadev liblog4cplusdev libcurl3dev

Then we will download, compile, and install OpenALPR from its project repository on GitHub. git clone https://github.com/openalpr/openalpr.git

cd openalpr/src

mkdir build

cd build

cmake DCMAKE_INSTALL_PREFIX:PATH=/usr DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc ..

make

sudo make install

Then we can install the Python bindings that are included with the project’s source files.

cd ../bindings/python

source make.sh

sudo pip install .

Setting up Redis and its Python bindings The Redis data structure store allows the operation of a distributed and networked message queue our system will utilize to facilitate data flow through itself. Connections to Redis are, by default, unsecure. Redis binds to any network interface without any notion of authentication or encryption. The Redis website’s quickstart page suggests firewalling Redis ports from external connections and/or using a Redis configuration file to limit the available network interfaces used. Requiring the Redis AUTH command or using a secure tunnel is also recommended. The Redis installation page suggests compiling from source, as per the following. wget http://download.redis.io/redisstable.tar.gz

tar xvzf redisstable.tar.gz

cd redisstable

make

sudo make install


Installation of a Python client interface can be readily done through pip.

sudo pip install redis

Setting up the Core System Retrieving this project’s source can be done through Git. git clone https://github.com/quadsquad193/Quadcopter

cd backend

Project package dependencies, such as the web application framework or library clients can be done through pip. Pip install r requirements.txt

Use pip to install the project to your sitepackages or virtualenv. You may need to use the upgrade or forcereinstall flags if previous setups have been done. If you are using a virtualenv, be sure it has access to the global Python bindings of the OpenALPR library. The e flag may be used to install the package in editable mode during development. [sudo] pip install .

Each service requires particular configuration and exposes different invocation points. The Git repository has a folder named launch which comes with basic configuration and runnable scripts to fire up each service. You may wish to use these as examples to guide your own rollout. Tests may be performed through the pytest framework and invoked by running py.test spec/. Setting up MySQL MySQL is used as for our persistent database. Aptitude allows simple installation from its repositories. sudo aptget install mysqlserver


2 User Guide

A. Android Application Use Instructions

1. Make sure your Android device has a connection to the Internet. The Patrol app requires this for: 1) Verifying the DJI API key 2) Communicating with the server.

2. Turn on the Phantom 3. Once the remote controller has synced with the Phantom 3,

connect your Android device, via USB, to the controller.

3. Run the Patrol application on your Android device. The app should indicate that it has connected successfully to the Phantom 3, and you should be able to see the live camera feed from the drone. The system is now ready for use.

4. Fly the drone to a vehicle. Align the drone so that the license plate is in clear view in the

camera feed. When ready, push the camera button on the Android app to take a picture. The image, along with GPS coordinates and timestamp will then be sent to the server. Note, it may be easier to do this with two people, one person manning the drone, the other controlling the Android device.

5. To log more vehicles, proceed to next vehicle location and repeat step 4.

6. To view previously taken images, select the gallery tab at the top of the app. This may

be useful to see how your images turned out if license plate recognition is inaccurate or failing.


b. User Guide for Web Interface Overview The primary use of the web interface is to verify citations that are flagged by our system. Users can verify or dismiss flagged citations, and can delegate duplicate citations to already verified citations. The web interface also allows users to view a variety of summary information about citations. Account Management Default User Upon creation of an instance of the webservice, a default administrator account is created with the username ‘admin’ and password ‘password’. This default administrator account will allow users to grant other users administrative privileges. Account Creation and Sign In On navigation to the application root url, a user who has not yet signed in will be redirected to the sign in page. Existing users can sign in with their username and password on this page. If a user does not yet have an account, they can create one by clicking the “Create account” link beneath the sign in form. This will cause a new form to load through which a user can create a new account. Upon successful account creation or sign in, the user will be redirected to the home page. Citations The citations page is used to verify, dismiss, and delegate citations. The page provides a list view of recent citations and allows users to verify citations as being valid, dismiss them as invalid, or delegate them to already verified citations referring to the same overparked vehicle. Users can filter the shown citations by the Verified, Dismissed, and Delegated flags. Users can also toggle whether evidence (all images of a cited vehicle taken over the time it was parked) is visible. The interface also allows users to search through past citations by both date and license plate. Summary The summary page provides five visualization of citation data. They are:

1. A counter displaying the number of citations issued that day by the system. 2. A map showing the locations of citations issued in the last week. 3. A counter displaying the number of citations ever issued by the system. 4. A bar chart showing the average number of citations issued by day of week. 5. A bar chart showing the number of citations issued each day in the last week.


3. Troubleshooting

Problem Possible Reason(s) Solution

Patrol App does not show drone’s camera feed.

1. DJI API key could not be registered. 2. Phantom 3 is connected to another app on your phone.

1. Make sure you create an API key from DJI developer website and paste it into mobile / app / src / main / AndroidManifest.xml file. Additionally, make sure your Android device has an Internet connection to register key with DJI. 2. Reconnect drone, and when prompted, select the Patrol app to communicate with the Phantom 3. If defaults have already been set, please remove them in your system settings.

GPS values are received as NaN.

GPS signal weak. Make sure you are in a location with a clear view of the sky and minimal EMI. This is especially important upon starting of the quadcopter.

Pictures I took do not appear on the web interface.

1. Vehicles will be flagged for citation and displayed only when sufficient evidence is collected to prove a violation. Noncitable images will not be displayed. 2. License plate recognition is failing and cannot identify plate text.

1. Continue collecting evidence until a vehicle is within citable window. 2. Try a different angle, closer distance, or different lighting. Refer to OpenALPR documentation on how to improve recognition accuracy.

Pictures I took not visible in gallery

Because images are pulled programmatically, the Android OS isn’t able to index them for viewing by the Gallery application

This may get solved in future Android versions. In the worst case, you’ll have to download a separate app and go to the folder where the images should be stored, and you should see the captures there.

Patrol application crashes Need to make directories and auth.txt file.

See note on page 6.

4. (FAQ) Frequently Asked Questions Parking Enforcement via Drone User Guide Page 12

Q: Which DJI Phantom models does this system work with? A: We specifically developed this system for the Phantom 3 Advanced quadcopter. We do not guarantee functionality with other Phantom models. Q: What Android versions are supported by the Patrol application. A: The minimum SDK version supported is SDK 19 (Android Kitkat, 4.4+) Q: How can I improve license plate recognition accuracy? A: OpenALPR can be configured for specific regions (i.e. California license plates). There are also settings to preskew an image before processing. This is helpful if images are taken at extreme angles. Q: How can I further develop this project? A: You can find our code upon github at https://github.com/quadsquad193/phantomboreas Important: It is against the law to fly any UAV upon camera feed alone. The UAV must be in in line of sight with the operator at all times.


5. Contact Information

Name Email

Professor Ilias Tagkopoulos [email protected]

Baotuan Nguyen [email protected]

Alexander Sergian [email protected]

Kelvin Lu [email protected]

Mark Machado [email protected]

Other references:

Development Blog: https://quadcopterucd.wordpress.com


6. Appendix

Parking Enforcement Via Drones Requirements Document

Authors Alexander Sergian Kelvin Lu Mark Machado Baotuan Nguyen Introduction Goals and Deliverables The goal of this project is to repeatedly fly our drone over a designated parking area and monitor parking times. We want to enforce against overparking in spots designated to be parked at for only a few hours at a time. Significance For most parking spots with timeconstraints, you rely on a city employed officer to drive around parking lots and chalk tires to keep track of violations. With the drone, this process is automated. We only need a single person to periodically monitor our web interface, to confirm citations. This greatly enhances parking enforcement. Better enforcement would also reduce parking abuse, making it a better experience for everyone. Background As far as we have found, no one has addressed this exact problem. There are groups who have addressed similar problems though. Dr. Hui Feng and a team from Fudan University used a DJI Matrice 100 drone with some additional hardware to detect illegally parked cars (parked over the lines). A Houston based startup called ParkiT has software that uses parking lot cameras to detect what spots are filled in a lot using computer vision and machine learning techniques.


System architecture overview

Android App Our Android app currently integrates three functionalities: the ability to control the DJI camera (i.e. take a picture), the ability to read from the quadcopter’s SD Card to download the required image onto the connected Android device, and the ability to upload an image to a server using an HTTP multipart post. The quadcopter’s camera stream is visible from the Android app. The following process gets the drone image to the server:

1) The remote controller downloads the image from the drone using DJI’s Media Scanner 2) The remote controller sends the image to the Android device using a serial USB

connection. 3) The Android App uploads the image to a given server using an HTTP POST request. It

sends the image, the latitude and longitude of the quadcopter’s location during image capture, and the timestamp of the image capture.


Server Processes

Our backend which receives the DJI quadcopter’s (using the Android app as a proxy) HTTP requests, processes the captured images and metadata, persists the results to a store, and serves the information has been split up into many separate processes by role. Each “microservice” process is meant to be run independently. Information is passed between each process through a message queue provided by a Redis server and persisted on a database. This allows multiple instances of each service to be run on many possibly separate hosts over the internet, as all communication is orchestrated through a “centralized” channel. We give a brief description of each services’ role and implementation below.

Drone Service A thin web application built on the Flask web framework. This provides a single HTTP endpoint through which an AndroidDJI application, acting on behalf of the Phantom 3 quadcopter, can make a request to over the internet. The application accepts an image and metadata, which is simply placed in a Redis message queue per request. We consider this unit of data a “payload” internally.


OpenALPR Service Payloads placed onto the Redis message queue are consumed by this process. The image is processed by the OpenALPR library for license plate detection and recognition. If an image contains some information extracted by the library, this service then augments the payload with the LPDR results and places this data into another message queue. ParkingLog Service This service has the responsibility of persisting the information gathered from the captured images into a data store. Again, payloads are consumed from the previous process through the message queue. A database schema defines a relational storage scheme in which results are persisted for business logic. In particular, the bulk of the work is done through an ORM to some database. Web Service The web service is a thin web application built on the Flask web framework. It currently only has a single route that provides a description of the project and an index of the captured images. In the future, its main purposes will be to allow users to query captured images by associated metadata and allow users to flag cars as being overparked.

Database Our database currently consists of a hierarchy of three tables: CaptureLog, PlateLog, and CandidateLog. CaptureLog is our toplevel model that maintains information about captured images and associated metadata including GPS coordinates, filename, and timestamps. CaptureLog has a onetomany relationship with PlateLog which tracks each license plate detected in a captured image. This table keeps track of the of the vertices of the plate images’ corresponding bounding boxes. PlateLog has a onetomany relationship with CandidateLog. CandidateLog keeps tracks of the candidate strings read from a plate by OpenALPR as well as the associated confidence. Glossary of Terms OCR Optical Character Recognition, extracting text from an image. OpenALPR Open Source library for “Automatic License Plate Recognition.” Waypoints DJI built in autonomous flight capability of Phantom 3 quadcopter. Drone will fly at a fixed altitude to a predetermined set of GPS coordinates. Gimbal Motorized camera mount used on the DJI Phantom 3 to stabilize and control pan the camera. Controllable from Android or iOS device. DJI Phantom 3 Advanced Quadcopter Model Redis An open source, inmemory data structure store Requirements


User Cases/User Stories

As the city’s parking enforcement, I want a easy to use web interface to keep track of and process citations.

As city parking enforcement, I would like the system to be robust and for the

transmission to be secure.

As city parking enforcement, I would like the drone to be out of reach and not be tampered with/stolen.

As a user, I want to know that the OCR used on license plates is reliable and will consistently and accurately read licence plates.

As a user, I want to know that the parking monitoring system will only mark cars as

parked past the allowable time if they truly are, so citations are not erroneously issued.

As a user, I want be confident that the drone will operate safely, avoiding any damage or injury.

As a motorist, I want to be reminded of parking times, so that I do not get cited.

As a motorist, I want a better city parking experience, so that I don’t have to waste time

trying to find a parking spot.

As the city’s parking enforcement, I want an easy (possibly even autonomous way) to monitor parked vehicles, so that we can enforce parking efficiently.

As the city’s parking enforcement, I want to know that the drone will be able to adjust for

obstructions preventing it from being able to read a licence plate. Prototyping Code w/ Testing (Github URL)* https://github.com/quadsquad193/phantomboreas *Testing code is inside phantomboreas/core/spec Appendices ◦ Technologies employed Image Recognition OCR through OpenALPR project DJI Phantom 3 Advanced Redis


https://github.com/quadsquad193/phantomboreas

Parking Enforcement via Drones User Manual - CS-CSIFParking Enforcement via Drones User Manual Authored by: Mark Machado Kelvin Lu Alexander Sergian Baotuan Nguyen Client: Professor

Documents