Report

i

DEVELOPING APPLICATIONS FOR LEGO MINDSTORMS NXT ROBOTS

IN ANDROID

A SUMMER INTERNSHIP REPORT

Submitted by

SOUVIK DAS

DEPARTMENT OF ELECTRONICS AND COMUNICATION HERITAGE INSTITUTE OF TECHNOLOGY,

KOLKATA-700107

Under the guidance of

Dr. C S Kumar DEPARTMENT OF MECHANICAL ENGINEERING

INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR

JULY 2014

ii

iii

ACKNOWLEDGEMENT I take great delight in expressing my deep felt gratitude to Prof. C S Kumar sir under whose guidance, I completed my summer internship. It was an honor and pleasure to work under him. I thank him for being patient and taking good care to see that I think and act in the right direction. I wholeheartedly thank Research Scholars-Roshan Kumar Hota, Deboshree, Nava Raja Poonam Anthony and Neha Jain for helping me throughout the project and making my summer internship a cakewalk. The kindness and concern they showed towards me is unforgettable. I would like to thank the Lab in charge – Kouhik Nag for their help and generosity. I thank my co-interns - R.Vishnu Vardhan, Teja Krishna, G.Vinod, Sunil Kumar Dutta and Sai Kaushik for being with me on and off times and making my intern days delightful. Finally, I thank Robotics and Intelligence Lab and CAD/CAM Lab, IIT Kharagpur for providing me with this oppurtunity to work and avail the facilities. I would also like to thank my parents for believing in me and have patience. I am highly indebted to my dear friends, Debanjan Lahiri and Samujjal Das for encouraging me and for letting me have this golden opportunity. SOUVIK DAS Department of Electronics and Communication Engineering, Heritage Institute of Technology, Kolkata-700107 [email protected]

iv

ABSTRACT

The report presents the work that has been carried out during the internship in

IIT Kharagpur. The study on using the android applications to control the robots is

done and previous works, different robotic platforms and available hardware and soft

wares are discussed briefly.

LABVIEW 2012 SP1 with MINDSTORMS NXT/EV3 extension is used to test

the sensors and actuators of NXT and the simple algorithms are implemented to make

some applications like Line Follower and Intelligent Vehicle for testing the NXT brick

and its functionality.

Finally, the attention is shifted to App Inventor 2 and its communication with

NXT Brick's Bluetooth module. A decent study is made on how the App Inventor sends

signals to the NXT over Bluetooth and started with developing simple applications to

communicate with NXT Bluetooth. Then, focused on using the sensors like

accelerometer, gyroscope and touch sensor that there in the android device. The

graphs of the sensor reading are generated. Finally, PID algorithm is implemented

using App Inventor 2 to build application like self-orientation robot and self-

balancing robot. These applications are tested on LEGO MINDSTORMS NXT and

results are documented.

Keywords: Android, LEGO MINDSTORMS NXT, App Inventor2, LabView, Control

systems

v

CONTENTS

Title Page i

Certificate ii

Acknowledgment iii

Abstract iv

Contents vi

List of Figures vii

List of Photographs viii

List of Abbreviations viii

CHAPTER 1: INTRODUCTION 1

CHAPTER 2: PROBLEM DESCRIPTION 1

CHAPTER 3: LITERATURE REVIEW 2

3.1 Related Previous Works

3.2 IOIO – An Alternative

3.2.1 Control Flow:

3.2.2 Modes of Communication

3.3 Lego Mindstorms NXT

3.3.1 PROGRAMMING LEGO MINDSTORMS NXT:

3.3.2 Connectivity and communication:

3.4 Lejos Mindstorms: NXJ Technology

3.5 App Inventor:

3.5.1 BUILDING THE USER INTERFACE

3.5.2 PROGRAMMING THE BEHAVIOUR

CHAPTER 4: ANDROID BASED ROBOTICS 10

4.1 Labview and Lego Mindstorms NXT

4.1.1 LINEFOLLOWER ROBOT

vi

4.1.2 INTELLIGENT VEHICLE

4.2 Android and Lego Mindstorms NXT

4.2.1 CONNECTION

4.2.2 TOUCH CONTROL

4.2.3 ACCELEROMETER CONTROLLED NXT

4.2.4 SELF-ORIENTATION ROBOT

4.2.5 SELF-BALANCING ROBOT

CHAPTER 5: METHODOLOGY 17

5.1 Programming the NXT with LabView

5.2 App Inventor 2 and Control Systems

CHAPTER 6: RESULTS AND DISCUSSION 25

CHAPTER 7: CONCLUSION AND FUTURE SCOPE 26

REFERENCES 27

APPENDIX I 28

OVERVIEW OF DIFFERENT SENSORS

Touch Sensor

Accelerometer

Gyro/Tilt Sensor

Light Sensor

Ultrasonic Sensor

APPENDIX II 31

LEGO COMMUNICATION PROTOCOL

APPENDIX III 33

PID Controller

APPENDIX IV 34

LABVIEW CODES

vii

Line Follower Robot

Intelligent Vehicle

APP INVENTOR 2

Bluetooth Connection

Lego Remote

Lego Accelerometer Remote

Self-Orientation Robot

Self-Balancing Robot

Pseudo Codes

LIST OF FIGURES

1. Fig 3.3 Block diagram

2. Fig 3.4 Modes of communication with remote server

3. Fig 3.6 Communication Block Diagram

4. Fig 4.3 AI2 Companion

5. Fig 4.4 Interface (a) Bluetooth is connected (b) Bluetooth is not connected

6. Fig 4.5 Interface of Remote Control

7. Fig 4.7 Screenshot of the Android application

8. Fig 4.9 Application for balancing Segway

9. Fig 5.1 Different conditions for color sensor

10. Fig 5.2 Lego NXT ultrasonic sensor distance measurement

11. Protocol Payload Data

12. Data packages when sending Bluetooth Commands

13. Fig PID controller Block Diagram (source: Wikipedia)

viii

LIST OF PHOTOGRAPHS

1. Fig 3.1 (a) PhoneSat

2. Fig 3.1 (b) Spheres

3. Fig. 3.2 IOIO-OTG Board

4. Fig 3.5 LEGO MINDSTORMS NXT 2.0

5. Fig 4.1 LEGO LINE FOLLOWER

6. Fig 4.2 LEGO intelligent vehicle

7. Fig 4.6 Control of NXT using Accelerometer

8. Fig 5.3 Self-Orientation Robot -- NXT Crab Chimera

9. Fig 5.5 Self-balancing Robot

10. Fig. 5.6 Coordinate system and rotation axes

11. Fig. Touch Sensor in NXT 2.0

12. Fig. A typical Accelerometer

13. Fig. Gyro Sensor compatible with NXT 2.0

14. Fig. Ultrasonic Sensor

15. Fig. An NXT Light Sensor

LIST OF ABBREVIATIONS USED

1. BCI - Brain Computer Interaction 2. USB - Universal Serial Bus 3. API - Application Programming Interface 4. WIFI - Wireless Fidelity 5. TCP - Transmission Control Protocol 6. LCP - LEGO MINDSTORMS NXT Communication Protocol 7. EDR - Enhanced Data Rate 8. SPP - Serial Port Profile 9. I2C - Inter Integrated Circuit 10. UART - Universal Asynchronous Receiver/Transmitter 11. PID - Proportional, Integral, Differential 12. UHF - Ultra High Frequency

1

CHAPTER 1: INTRODUCTION

Robot, any automatically operated machine that replaces human effort, though

it may not resemble human beings in appearance or perform functions in humanlike

manner [1]. Robotics is the branch of technology that deals with the design,

construction, operation, and application of robots as well as computer systems for

their control, sensory feedback, and information processing. It is the art and

commerce of robots and their practical use.

Years ago, cell phones were being used only for communication and tablets

were unheard of. Once with the invention of touch sensitive displays and processor

manufacturers consistently delivered processors with increased power, the

smartphones and tablets have become powerful computing resources that can run

complex operating systems and can be connected via various technologies with other

devices or gadgets. The gap between a mobile device and a computer in terms of

processing capabilities diminished in time and we can see every day more and more

applications which can run on these little devices. The mobile phones with such

processing capabilities are exploited to control the robots in place of computers [2].

The robots are becoming more common and can be found in many places like

homes, offices, or public spaces. All these robots have to be controlled and

monitored. There are many platforms through which the mobiles and the robots can

be interfaced. Using mobile resources like smartphones or tablets which runs the

Android operating system, any user can control and monitor robots with just a click

or simple movements.

CHAPTER 2: PROBLEM DESCRIPTION

This project focuses on the interaction of android platform with robots. The

objectives include investigating the different options that are present for the android

platform to control robots and checking the compatibility of different robotic

platforms with android. Finally, to implement PID algorithm to make a self-balanced

robot using android device.

2

CHAPTER 3: LITERATURE REVIEW

3.1 RELATED PREVIOUS WORKS

A lot of research has been done and on-going in this field of using Android

Platform to remotely access various Robotic platforms. A lot of robotics platforms are

available commercially for education/research purpose (like LEGO

Mindstorms, MobsyaThymioII, IRobotCreate, VEX, TETRIX, Surveyor

(SRV-1) and Bioloid) and for industrial and commercial purposes (like PUMA-

560, KUKA, EUROP etc.) [3]. All these platforms can be accessed and controlled

remotely by various tools that are provided by the manufacturer.

Smartphone robots and applications are becoming increasingly prevalent in

research projects. Scientists at NASA and MIT built free-flying satellites called

SPHERES and PhoneSats that are self-contained with power, propulsion, and

computing and navigation equipment. These satellites use smart phones as control

systems and a UHF radio beacon to transmit data and images to the ground. The

smart phones monitored the cameras, accelerometers, magnetometers, and

gyroscopes, which were onboard the satellites [4].

(a) (b)

Fig 3.1 (a) PhoneSat[4] (b) Sphere [5]

3

Android phones have also been used successfully with the LEGO

MINDSTORMS NXT for robotics/software engineering classes. An Android based

robot was also used for brain-computer interface (BCI) research where the brain

activity of a subject controlled a robot over the internet. The robot was composed of a

Lego MINDSTORMS NXT 2.0 and an Android phone which sent video feedback to

the subjects.

3.2 IOIO – AN ALTERNATIVE

The IOIO is a board that provides a host machine the capability of interfacing

with external hardware over a variety of commonly used protocols. The original IOIO

board has been specifically designed to work with Android devices. The newer IOIO-

OTG ("on the go") boards work with both Android devices and PC's The IOIO board

can be connected to its host over USB or Bluetooth, and provides a high-level Java

API on the host side for using its I/O functions as if they were an integral part of the

client

Fig. 3.2 IOIO-OTG Board [6]

This IOIO board can also be used in our work as an alternative. This will act as

an interface between our android platform and the system actuators and sensors. A

block diagram of the concept has been described in the next page .Also different

mode of communication has also been mentioned and explained.

4

3.2.1 Control Flow:

Fig 3.3 Block diagram [3]

A two way communication between the Android Phone (Platform) and the

IOIO Board can be observed in the block diagram. The Android Phone generates a

signal according to the inputs received by the Android Phone and send it to the IOIO

Board. The IOIO Board in turns pass this signals to the actuators (which are motors in

our case) and drives the motors. It also instruct the other sensors and R/C vehicles

which are present in our system. In other words, the Android phone interacts

with actuators, such as speed controllers or pan/tilt units, via the IOIO

board. The modes of communication that can be used to connect an Android Phone

and an IOIO Board are Bluetooth, WIFI or by using

USB Cables, which are explained in detail.

3.2.2 Modes of Communication

Fig 3.4 Modes of communication with remote server [3]

5

The Android Phone connects with the IOIO using a Bluetooth Connection

or by using a USB Cable.

This whole system, Android Based Robotics in turns is connected to remote

location (which is a PC in our case) using Wi-Fi (TCP Protocols).

The IOIO and all other peripherals (Actuators, R/C vehicles, Sensor and

Android Phones) communicates by sending and receiving PWM signals.

3.3 LEGO MINDSTORMS NXT

The Lego Mindstorms series of kits contain software and hardware to create

customizable, programmable robots. They include an intelligent brick computer that

controls the system, a set of modular sensors like color, ultrasonic, touch sensors and

motors, and LEGO parts from the Technics line to create the mechanical systems.

Programming and connectivity of the LEGO MINDSTORMS NXT is discussed

briefly in next section.

Fig 3.5 LEGO MINDSTORMS NXT 2.0 [7]

6

3.3.1 PROGRAMMING LEGO MINDSTORMS NXT:

There are several ways to program Lego NXT:

• Using the LEGO MINDSTORMS NXT official software - This software

is free and comes with LEGO NXT package. It has a powerful UI

programing side, and it is basically a Drag & Drop programming. This kind

of programming is very simple and can be adopted even by a non-

programmers or children.

• Using third-party software’s- There are a lot of free and non-free

wrappers for Lego communication protocol and firmware. LabView with

LEGO MINDSTORMS NXT add-on is one such type.

• Using a Lego NXT communication protocol (LCP) - It is completely free,

and a low-level programming. This protocol obviates the usage of

MINDSTORM or any other third—party software. It is used for controlling

the LEGO from a remote device.

3.3.2 Connectivity and communication:

The LEGO MINDSTORMS NXT includes the following communication

possibilities [8]:

• Bluetooth Communication, V2.0 with EDR

o Supporting the Serial Port Profile (SPP)

• USB communication, V2.0

Beside the above mentioned two main communication protocols, the LEGO

MINDSTORMS NXT will also include two communication interfaces which primary

purpose is communication with external embedded devices.

7

• One 6 wired digital communication port, maximum communication

speed at 1Mbps (High speed port)

• Four 6 wired digital communication ports, maximum communication

speed at 9600 bps (Low speed ports)

The low-speed communication is using the I2 C communication standard. The

primary usages for the high-speed port are to enable the NXT to communicate with

external devices which requires high communication speeds.

The figure below shows the main layers in the communication stack between the

embedded device and PC.

Fig 3.6 Communication Block Diagram

It will be possible to access the communication with NXT either using the LCP

or by writing and reading raw data directly to and from the communication buffers.

When writing and reading directly to and from the buffers, there is no way to check

on the data being sent or read back from the NXT unit.

8

3.4 LEJOS MINDSTORMS: NXJ TECHNOLOGY

LeJOS is a tiny Java Virtual Machine. In 2006, it was ported to the LEGO NXT

brick. LeJOS NXJ has got all the classes and API's to support all the functionalities

of MINDSTORMS NXT and EV3 (The updated version of NXT). Those libraries can

be used to develop the android applications and control NXT/EV3 Robot remotely.

To be able to use Lejos Libraries, the firmware of the Brick must be changed, which

is one of the basic disadvantages of this method. But nevertheless many applications

can be built using its full potential [9] [10].

3.5 App Inventor:

App Inventor is a visual blocks language provided by Google that allows

people who are not familiar with computer programming, to create mobile

applications [11]. App Inventor graphical interface allows users to drag-and-drop

visual objects to create application that run on many mobile phones with the Android

OS.

App Inventor has two main windows: a component designer for building the

user interface and a blocks editor for defining the application behavior. Application

can be tested directly on the phone of an emulator.

3.5.1 BUILDING THE USER INTERFACE

Building the User Interface with the Component Designer. The Component

Designer is a tool for designing user interfaces. The left side palette has all the

components that can potentially be added to an application. The left-middle panel or

Viewer, is the surface representing the way the phone’s screen well appear hen the

applications runs. The use drags components from the palette to the viewer to specify

components of particular application. Here, non-visual components the applications

need for accessing the technology available on the Android device, can also be

specified.

9

3.5.2 PROGRAMMING THE BEHAVIOUR

Programming the Behaviors with the Blocks Editor. The behavior of the

application is defined in the Blocks Editor. This Editor has two palettes from which

blocks are dragged, the Built-in pallet and the User-specified Blocks palette.

• The Built-in palette contains blocks for standard programming control and

functionality, for text and list manipulation, and mathematical, logical and

control operators.

• The user-specified palette contain blocks representing the components of the

applications that were added in the Component Designer. The application

behavior is directly defined through a set of event-handlers.

Live testing can be performed with a plug in phone or with an Android phone

emulators, which helps testing behaviors.

The App Inventor enables the usage of Android device to control the LEGO. It has

a library for LEGO NXT, with components and their behaviors defined. App Inventor

uses the Lego MINDSTORMSNXT communication protocols to send the commands

over Bluetooth network.

10

CHAPTER 4: ANDROID BASED ROBOTICS

Android Based Robotics is a huge area where numerous researches are going

on in various interdisciplinary fields. The work focuses on developing an Android

applications using App Inventor 2 and use that application to connect with NXT

Brick and control the NXT brick. It has been emphasized particularly on using this

Android Platform to develop algorithms for PID Controller to make the robot

autonomous.

Before started working with android and LEGO NXT, some applications are

developed using LABVIEW to get familiar with the hardware of the NXT robot [12].

The applications are briefly introduced.

4.1 LABVIEW AND LEGO MINDSTORMS NXT:

4.1.1 LINE FOLLOWER ROBOT:

In this application a wheel robot is made with Legos and a color sensor. The robot

follows a black line a white surface using single color sensor.

Fig 4.1 LEGO LINE FOLLOWER

11

4.1.2 INTELLIGENT VEHICLE:

In this application the robot follows a line and at the same time it stops moving if it

faces any obstacle. The robot adjusts its speed depending upon the distance of the

obstacle. The color sensor and ultrasonic sensors are used to detect the line and an

obstacle. It is an extension of line follower robot.

Fig 4.2 LEGO intelligent vehicle

4.2 ANDROID AND LEGO MINDSTORMS NXT:

Four Applications have been developed using App Inventor 2 to

• Control the robot remotely.

• Control the robot using the accelerometer of the Android Device.

• Orient the robot to any user specified direction using the Orientation

Sensor/Gyro Sensor.

• Develop a PD controller to make the robot self -balance on two wheels using

the accelerometer of the Android Device (Samsung Galaxy Y duos).

12

4.2.1 Connection

To access the NXT Brick, a connection has to be setup between the Android

Device and the NXT brick. At first the App Inventor (PC) has to be connected to the

Android Device. This can be done two ways.

• A temporary connection between the two devices by using a cloud connection

with the help of AI2 Companion. The Android device should have this

application installed in its system. See the picture in the next page.

• The second way is to build an '.apk' and directly install the application in the

android device.

Next the connection between the Android Device and the NXT Brick has to be set-

up. For this Bluetooth Connection has been used. An application has been built to

connect the Android Device to the NXT Brick. The codes for the Bluetooth

connection can be found in Appendix IV. The screen shot of the interface is shown in

the next page.

Fig 4.3 AI2 Companion

13

(a) (b)

Fig 4.4 Interface (a) Bluetooth is connected (b) Bluetooth is not connected

4.2.2 Touch Control

In this application touch sensors of the android device have been used to

navigate the Robot. The screen shot below shows the interface of the application.

Fig 4.5 Interface of Remote Control

14

Touch Control is a basic project that highlight how to establish connection with

NXT Brick and actuate the motors. Here the four buttons on the screen are used to

navigate the robot in four different directions. The Bluetooth connection is

established between NXT Brick and the Android Device and then the buttons are

pressed by the user to control the robot remotely.

4.2.3 Accelerometer Controlled NXT

In this application, the accelerometer is used to control the movement of the

Robot. The raw data from the accelerometer has been extracted by using the Android

Device and those data are used to give power to the motor and steer the Robot in

different directions. App Inventor 2 is used to build the application which will extract

the sensor's raw data and send the data to the NXT brick by following the Bluetooth

Communication Protocols.

It can be seen from the picture that the Tab is inclined in various directions to

navigate the Robot.

Fig 4.6 Control of NXT using Accelerometer

15

4.2.4 LEGO Orientation

In this application the Orientation Sensor/Gyro Sensor of the Android Device is

used to orient the Robot in a user specified direction. An Android application is

developed that can detect orientation in three different axes: Azimuth(Z-axis), Roll

(X-axis) and Pitch (Y-axis). The Azimuth value reads 0 when kept in north direction.

The android device is kept on the Robot and is fixed to the Robot. After the user

specified the azimuth value (which is taken as an input directly in the application) the

Robot moves in clockwise or anticlockwise direction to orient itself in the direction

specified by the user.

Initially a Bang-Bang Controller is used to orient the Robot in the desired

direction. But the Robot was overshooting every time it reaches its desired

orientation. So finally P Controller is used to control the Robot efficiently. The P

controllerisimplemented using simpleArithmeticExpressionsand Graphical

Fig 4.7 Screen shot of the Android application for self-orientation robot

Interface provided by App Inventor 2. The error is calculated and a corresponding

power is sent to the Robot which are related by the following equation:

Power = Kp * Error

When the Android Device is put on the Robot, it directly takes the value of power to

actuates the motors clockwise or anti-clockwise and orient itself accordingly. The

16

codes and the algorithm for the P Controller can be found in the Appendix IV and

Chapter 5 in details. The picture of the experiment is shown in the next page.

4.2.5 Two Wheel Balancing Robot using Android Phone

This experiment has been carried out by using a Samsung Galaxy Y Duos. The

main aim of this experiment is to use an Android Phone to balance a Two-Wheeled

Robot (LEGO MINDSTORMS NXT 2 is used) by making use of the accelerometer

of the corresponding phone to calculate the error.

Fig 4.9 Application for balancing Segway

This error can be positive or negative if the Robot falls in the forward direction

or backward direction respectively. Then the difference of the final error and initial

error is also calculated for the PD Controller. This error is converted to power and is

sent to the motors of the Segway that drives back and forth to balance itself according

to the sign of the error. The power is calculated by using the following equation:

power = KP * error + KD * diff + KI * sum .................... (2)

The above formula is implemented in App Inventor 2 to develop a controller for the

Segway. The methods followed and algorithms discussed in Chapter 5.

17

CHAPTER 5:METHODOLOGY ADAPTED

The LEGO MINDSTORMS kit is arguably the most widely used educational

platform as it is relatively inexpensive, reusable, robust, reconfigurable. Labview is

used to get familiar with the kit and its working and program the brick. Then,

Android applications were developed using App Inventor 2 to control the brick.

5.1 LABVIEW

National Instruments LABVIEW 2012 SP1 is used to get familiar with LEGO

MINDSTORMS NXT robot and programming. Few applications like line follower

robot, intelligent vehicle are developed. PID controller algorithm is implemented to

correct the error and make the robot stable.

5.1.1 LINE FOLLOWER:

Single color sensor is used to sense the black line on the white surface. The

color sensor is used in ‘Reflection Mode’. The sensor reads 100% when it encounters

complete white (Fig 5.1 c), 0% with complete black (Fig 5.1 a) and reads

intermediate values in all other cases (Fig 5.2 b), ideally. In practice, as the values

vary depending upon the ambient conditions, the sensor needs to be calibrated and

threshold values should be changed accordingly, every time the environment changes.

Fig 5.1 Different conditions for color sensor

18

5.1.2 INTELLIGENT VEHICLE:

This is an extension to the line follower robot, where ultrasonic sensor is used

to sense the obstacles. The ultrasonic sensor reads the distance of the obstacle. P

controller is used to adjust the speed of the robot depending on the distance.

Fig 5.2 Lego NXT ultrasonic sensor distance measurement [13] [14]

The P controller is used to correct the speed of the left and right motors according to

the error in both Line follower robot and intelligent vehicle. The Lab View codes and

description about the PID controller is mentioned in Appendix III

5.2 APP INVENTOR AND CONTROL SYSTEMS

In this section the methodology for all the applications have been discussed in

details. It includes the procedures adopted and algorithms followed of the control.

In all the four experiments, the NXT Robot is connected to the Android Device

over Bluetooth by providing a Bluetooth Client. This provision is present in App

Inventor 2 MINDSTORMS NXT Blocks.

The App Inventor 2 programs are given in the Appendix IV Section.

5.2.1 Remote Control

For this experiment, LEGO MINDSTORMS NXT Crab Chimerahas been

used(shown in the picture below).

19

Fig 5.3 Self-Orientation Robot -- NXT Crab Chimera

First, a provision for Bluetooth connection(a Bluetooth Client is provided for

the Android Device) is made. Then using App Inventor 2 Block Programming

interface the actual program is built. The outline of the algorithm followed is given

below:

- Establish Bluetooth connection

- Read the button

- Execute corresponding action

- Repeat the steps

The above algorithm is followed and implemented in App Inventor2(the programs are

given in Appendix IV Section). The Application is then tested on the Robot and is

concluded to be working fine.

5.2.2 Accelerometer Controlled NXT

The application uses the accelerometer of the Android Device to navigate the

robot. The Accelerometer Reading in Z-direction and Y-direction is used for giving

power to motors and enable steering. When the Tab is being held in landscape

orientation, inclination of the top of the Tab in downward sense gives forward motion

and in upward sense gives backward motion. If the Tab is inclined in left sense then it

moves the Robot in Anti-clockwise direction and vice-versa. A Samsung Galaxy Tab

is used to run the application.

20

The outline of the algorithm which is followed is given below:

- Read Y and Z values from the accelerometer

- Use Z value for powering the motors

- Use Y value for steering

- Set left motor power to the difference of Z and Y values

- Set right motor power to the sum of Z and Y values

- Repeat all steps

5.2.3 LEGO Orientation

In this application, the android device is a part of the robot.As has been mentioned earlier, this experiment uses the Gyro Sensor to orient itself to the given user-specified initial value. For this experiment Samsung Galaxy Tab 10.1 is used. The Tab is attached on the NXT Robot in Crab Chimera formation. The Application extracts raw data from the Orientation Sensor of the Tab and implements a P-Controller Algorithm to generate a power control. The error is calculated by the following formula:

error = final_pos - initial_pos ;

Wherefinal_pos is the current position of robot and initial_pos is the position specified by the user (the position at which the Robot will orient itself ultimately). The outline of the algorithm that is followed to implement the P-controller is given below:

- Get the azimuth value (which corresponds to direction) from user - Set it as the initial position - Read gyro sensor - Subtract initial value from reading to get the error - Multiply the error with proportional constant of P-Controller (P-gain) - Set the value to power - if the error is negative power the left motor, else power the right motor - Repeat all the steps

5.2.4 Two Wheeled Balancing(Segway) Robot using Android Device

The mobile is fixed to the robot, like has been done for the self-orientation

robot. The model (shown in the figure) has been chosen to make the Segway more

21

Fig 5.5Self-balancing Robot

stable. The main objective of choosing this model is to keep the center of mass just

above the axis of the wheel so that the net torque on the robot can be minimized.

Moreover the Android phone is attached with the robot in the front so the robot has a

natural tendency to fall in the forward direction. To cancel the effect certain changes

have been introduced in the algorithm.

PID Controller is implemented in App Inventor 2 which directly balances the

Segway. The phone is used as both the sensor and the processor to balance the Robot.

The accelerometer senses only the linear movements in three directions and gives

data. Thisdataisused to measure the roll and pitch angle. At this point the X, Y, and Z

readingsare pretty noisy so a Low-pass filter is applied to remove the short-term

fluctuations. The formula for a low-pass filter is pretty easy and it uses the previous

read values. The low pass filter implemented is given below,

𝑋𝑛 = 𝛼𝑋𝑛−1 + (1 − 𝛼)𝑋𝑛 (1)

After obtaining the raw values with less noise, roll and pitch values can be

computed. The formulae to calculate roll and pitch from raw accelerometer values

and derivation of the formulas are given below.

22

PITCH AND ROLL ESTIMATION

The orientation of the smartphone can be defined by its roll, pitch and yaw

rotationsfrom an initial position. The roll, pitch and yaw rotation matrices, which

transform a vector (such as the earth's gravitational field vector g) under a rotation of

the coordinate system of Figure by angles φin roll, θ in pitch and ψ in yaw about the

x, yand zaxes respectively, are:

Fig. 5.6 Coordinate system and rotation axes [16]

The raw values need to be converted to the g unit (1g = 9.8 m/s²), before applying the corresponding equations. The process of obtaining and converting the accelerometer readings depends on the accelerometer

𝐺𝑎𝑐𝑐 = 𝑅𝑎𝑤𝑎𝑐𝑐 × 𝑅𝑎𝑛𝑔𝑒2𝑟𝑒𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛−1

(2)

Now the sensor readings (converted to g unit) are,

𝐺𝑝 = �𝐺𝑝𝑥𝐺𝑝𝑦𝐺𝑝𝑧

� = 𝑅𝑔 = 𝑅 �001� (3)

R is the rotation matrix describing the orientation and ‘g’ is the earth’s gravitational

field matrix.

23

𝑅𝑥(𝜑) = �1 0 00 cos (𝜑) sin (𝜑)0 −sin (𝜑) cos (𝜑)

� (4)

𝑅𝑦(𝜃) = �cos (𝜃) 0 −sin (𝜃)

0 1 0sin (𝜃) 0 cos (𝜃)

� (5)

𝑅𝑧(ψ) = �cos (ψ) sin (ψ) 0−sin (ψ) cos (ψ) 0

0 0 1� (6)

There are six possible orderings of these three rotation matrices and, in

principle,all are equally valid. The rotation matrices do not, however, commute

meaning that the composite rotation matrix Rdepends on the order in which the roll,

pitch and yaw rotations are applied. It is instructive to compute the values of the six

possible composite rotation matrices Rand to determine their effect on the earth's

gravitational field of 1g initially aligned downwards along the z-axis

𝑅𝑥𝑦𝑧 �001� = 𝑅𝑥(𝜑)𝑅𝑦(𝜃)𝑅𝑧(ψ)�

001� (7)

After substituting (3), (4), (5) in (6), we get

𝑅𝑥𝑦𝑧 �001� = �

−sin (𝜃)cos (𝜃)sin (𝜑)cos (𝜃)cos (𝜑)

� (8)

Above equation can be written in the form of (7)

𝐺𝑝||𝐺𝑝||

= �−sin (θ)

cos (𝜃)sin (𝜑)cos (𝜃)cos (𝜑)

� (9)

1

�𝐺𝑝𝑥2+𝐺𝑝𝑦2+𝐺𝑝𝑧2�𝐺𝑝𝑥𝐺𝑝𝑦𝐺𝑝𝑧

� = �−sin (𝜃)

cos (𝜃)sin (𝜑)cos (𝜃)cos (𝜑)

� (10)

24

By solving equations (8) and (9) for pitch and roll, we get,

𝜑𝑥𝑦𝑧 = �−𝐺𝑝𝑦𝐺𝑝𝑧

� (11)

𝜃𝑥𝑦𝑧 = � 𝐺𝑝𝑥

�𝐺𝑝𝑦2+ 𝐺𝑝𝑧2� (12)

The mobile is fixed on the robot in landscape orientation. The error calculated

using the change in roll angle. This error is fed to the PID controller which sends the

control signals for the NXT brick to drive the motors in order to balance the robot.

The outline of the algorithm that is followed to implement the PID-controller is given below:

- Calibrate the sensor in stable position

- Read accelerometer and calculate pitch

- Calculate error by subtracting the calibrated value from the current value

- Calculate the difference term by subtracting the current error from previous

error

- Calculate the sum term by adding the current error previous sum

- Add all the three terms after multiplying error with P-gain, difference with

D-gain, sum with I-gain to get the power

- Repeat all the steps

The pseudo codes for all algorithms are given in appendix IV.

25

CHAPTER 6: RESULTS AND DISCUSSIONS

Android apps are developed using App Inventor to control LEGO MINDSTORMS

NXT robots. Two kinds of applications are made to control the robots:

1) Applications which control the robot remotely

2) The applications which need the mobile to be attached to the phone.

The robot is made to be remotely controlled by android applications viz. LEGO

Touch Control and LEGO Accelerometer Control successfully. The P controller is

successfully implemented in the android application for self-orientation robot. The

algorithm, to calculate the roll and pitch from raw values using rotation matrices and

low pass filter to reduce the noise, and PID controller are successfully implemented in

the android application. But it is partially successful when implemented in hardware.

The issues observed are noise in sensor readings, delay caused by the motors and

communication delay.

26

CHAPTER 7: CONCLUSION AND FUTURE WORK

All the android applications developed, are successfully implemented on

LEGO MINDSTORMS NXT platform. As mentioned, the self-balancing robot is not

successful on hardware. The noise in the sensor readings is reduced up to some extent

by implementing simple low pass filter. But, that did not suffice mobile application to

give stable control signals for the robot.

In future, Kalman filter will be implemented to get the accurate readings from the

sensor. As App Inventor 2 is drag and drop kind of programming, it does not

facilitates to code the advanced algorithms. In this context, LeJos would better serve

the purpose than App Inventor. In LeJos, as Java is used, many control algorithms can

effectively be implemented to make the systems robust.

The Bluetooth capabilities of the NXT brick have been used by connecting it with

PC or by connecting it to an Android Device. But the idea of Android Based Robotics

does not limit itself to establish a connection between the NXT platform and Android

Device. This can be extended and further carried out to establish a connection

between all Robotic Platforms available commercially and Android phone/tab.

27

REFERENCES

[1] Hans Peter Moravec: Robot definition http://www.britannica.com/EBchecked/topic/505818/robot

[2] Best of LEGO NXT Projects over Timehttp://www.intorobotics.com/tag/mindstorms-nxt/

[3] Oros N., Krichmar J.L. 2013. "Smartphone Based Robotics: Powerful, Flexible and Inexpensive Robots for Hobbyists, Educators, Students and Researchers." CECS Technical Report 13-16. November 26, 2013. Center for Embedded Computer Systems, University of California, Irvine.

[4]PhoneSats- http://www.phonesat.org/

[5] image source: http://www.nasa.gov/mission_pages/station/main/spheres_smartphone.html

[6] Imagesource: https://github.com/ytai/ioio/wiki/Getting-To-Know-The-IOIO-OTG-Board

[7] image source: http://robotworkshopfinland.blogspot.in/2013/03/lego-mindstorms-nxt.html

[8] Appendix 1-LEGO MINDSTORMS NXT Communication protocol

[9] S. Goebel, et al., "Using the Android Platform to control Robots," in Proceedings of 2nd International Conference on Robotics in Education (RiE 2011), 2011, pp. INNOC - Austrian Society for Innovative Computer Sciences--142.

[10] http://www.lejos.org/nxj.php

[11] Sergio Sandoval-Reyes, Pedro Galicia-Galicia and Ivan Gutierrez-Sanchez “Visual Learning Environments for Computer Programming” -- 2011 Electronics, Robotics and Automotive Mechanics Conference

[12] Jesús M. Gómez-de-Gabriel, Anthony Mandow Jesús Fernández-Lozano, and Alfonso J. García-Cerezo -- “Using LEGO NXT Mobile Robots With LabVIEW for Undergraduate Courses on Mechatronics” , IEEE International Conference on Social Computing / IEEE International Conference on Privacy, Security, Risk and Trus

[13] Image source: http://www.cdn.sciencebuddies.org/Files/4703/5/lego-nxt-ultrasonic-sensor-distance_img.jpg

[14] Image source:

[15] Image source:

http://www.tik.ee.ethz.ch/mindstorms/sa_nxt/content/images/us_setup3.jpg

[16] Pitch and Roll Estimation

http://www.legoengineering.com/

http://www.freescale.com/files/sensors/doc/app_note/AN3461.pdf

28

APPENDIX I:

OVERVIEW OF DIFFERENT SENSORS

Touch Sensor

In general a touch sensor is a type of sensor that only has to be touched by an object

to operate. A touch screen includes an array of touch sensors on a display. A touch

sensor is the simplest kind of tactile sensor.

There are mainly two type of touch sensors:

1. Capacitive Touch Sensor

2. Resistive Touch Sensor

Fig: Touch Sensor in NXT 2.0 [15]

Touch Sensors in LEGO MINDSTORMS NXT:

The Touch Sensor gives the robot a sense of touch. The Touch Sensor detects when it

is pressed by something and when it is released again.

An accelerometer is a device that measures proper acceleration. The proper

acceleration measured by an accelerometer is not necessarily the coordinate

acceleration (rate of change of velocity). Instead, the accelerometer sees the

acceleration associated with the phenomenon of weight experienced by any test mass

Accelerometer

29

at rest in the frame of reference of the accelerometer device. Another term for the

type of acceleration that accelerometers can measure is g-force acceleration.

Fig. A typical Accelerometer

Conceptually, an accelerometer behaves as a damped mass on a spring. When the

accelerometer experiences an acceleration, the mass is displaced to the point that the

spring is able to accelerate the mass at the same rate as the casing. The displacement

is then measured to give the acceleration.

A LEGO MINDSTORMS NXT 2.0 has also got its accelerometer/tilt sensor.

This Accelerometer/Tilt Sensor measures acceleration in three axes. It also measures

tilt along each axis. Using the sensor, the acceleration of the robot can be measuredin

the range –2g to + 2g...

Gyro sensors detects if the device's axis are rotated from the real-world; it detects tilts

and degrees from the magnetic North. They are often used in self-balancing robots for

keeping upright. They are also used in autonomous robots for keeping track of

orientation. A gyro sensor measures the angular velocity. In other words they measure

how fast the sensor is rotating. This is most often expressed as degrees per second or

radians per second. Some gyro sensors measure the angular velocity over one axis,

others take measurements over two or three axes. LEGO Mindstorms NXT features a

GYRO Sensor.

Gyro Sensor/Tilt Sensor

30

Fig. Gyro Sensor compatible with NXT 2.0 [15]

Ultrasonic Sensors

The Ultrasonic Sensor is used to measure the distance of an obstacle. It

measures distance in centimeters and in inches. It is able to measure distances from 0

to 255 centimeters with a precision of +/- 3 cm.

The Ultrasonic Sensor uses the same scientific principle as bats: it measures

distance by calculating the time it takes for a sound wave to hit an object and return –

just like an echo. Large sized objects with hard surfaces return the best readings.

Objects made of soft fabric or that are curved [like a ball] or are very thin or small

can be difficult for the sensor to detect.

Fig. Ultrasonic Sensor [15]

Light Sensor

The Light Sensor enables a robot to distinguish the intensity of the reflected

light. This information can be used to distinguish the colors, also. Thus, a light sensor

can also be used as a color sensor.

31

Fig. An NXT Light Sensor [15]

APPENDIX II:

Lego Communication Protocol [8]:

The communication protocol is placed above the USB and the Bluetooth

communication ports because this layer will handle the data that will be the payload

within the USB and the Bluetooth communication layers.

All the calculation of checksum and setting of package numbers will be

handled by the individual communication layers.

The communication protocol handles the following communication purposes:

• Download files (File write)

• Upload files (File read)

• Delete user-defined data in the embedded system

• Direct communication with the NXT system

General protocol overview, payload data from the USB or Bluetooth

communication channel:

Protocol Payload Data

32

Byte 0: Command type. The 7th lowest bit of this byte is used for identifying the

command type.

Byte 1: Command byte. Used by the loader module to identify what should happen

with the data. Open, Read, Write, Delete data, and direct communication with NXT.

Byte 2 - N: These bytes offer additional information.

This protocol will be wrapped into either the Bluetooth serial profile or the USB 2.0

communication protocol.

Bluetooth Communication for LEGO

The Bluetooth functionality with LEGO MINDSTORMS NXT is using the

SPP. The interface to the Bluetooth chip in the embedded system is a high speed

UART interface on the ARM7 processor. All the baseband handling is done within

the Bluetooth chip which means that the data sent to and from the Bluetooth chip is

the same data that is sent to and from LEGO MINDSTORMS NXT communication

protocol layer on the PC side.

Commands are sent to the Lego NXT through Bluetooth by transferring a data

package (a bulk of bytes) to the robot for immediately execution. These commands

are divided into two groups:

• System commands – Administrative commands that control the administration

of the brick or the protocol. Example for administrative commands can be -

access the file system on the brick or communication configuration.

• Direct commands – Commands that control robot’s motion, sensor’s

reading…

Data packages when sending Bluetooth Commands

33

Some of the commands are a request for feedback from the robot while other

commands are just a set for orders. When using Bluetooth for communication, 30

milliseconds must pass between the time the NXT sends a response and the receipt of

a new command. The structure of the data package is different when it is sending to

the brick from the structure when it is receiving from the brick

APPENDIX III: ALGORITHM

PID Controller

A proportional-integral-derivative controller (PID controller) is a control

loop feedback mechanism (controller) widely used in industrial control systems. A

PID controller calculates an error value as the difference between a measured process

variable and a desired set point. The controller attempts to minimize the error by

adjusting the process through use of a manipulated variable.

Fig PID controller Block Diagram (source: Wikipedia)

34

APPENDIX IV:

LABVIEW AND APP INVENTOR CODES

LABVIEW CODES:

Line Follower Robot:

35

Intelligent Vehicle:

36

APP INVENTOR 2:

BLUETOOTH CONNECTION:

37

LEGO REMOTE:

38

ACCELEROMETER LEGO REMOTE:

39

SELF ORIENTATION ROBOT:

40

SELF-BALANCING ROBOT

Self-Balancing Robot Sensor Calibration:

41

Self-Balancing Robot Controller:

42

PSEUDO CODES:

Bluetooth Connection:

If Bluetooth connection is enabled:

If (number (Bluetooth devices) = 0)

Print “Pair the device”

Else

Connect to Bluetooth

Else

Print “Enable Bluetooth connection

Lego Remote Control:

check: if Bluetooth connection = ' enabled ' {

then

check: if button up = ' click '

move forward ;

check: if button left = ' click '

move anticlockwise ;

check: if button right = ' click '

move clockwise ;

check: if button down = ' click '

move backward ;

check: if button stop = ' click '

stop ; }

else check: if Bluetooth connection = ' disabled ' then connect to Bluetooth ; else check: if no. of Bluetooth devices = 0 then pair the device ;

43

Lego Accelerometer Control

while( accelerometer reads ' value ')

-do-

power = z_direction accel_reading ;

steering = y_direction accel_reading;

LeftMotorPower = (power*10)-(steering * 10);

RightMotorPower = (power*10)+(steering*10);

-end loop-

Self-Orientation Robot:

ref_posuser input

While (gyro_read)

-do-

if(error<0)

power_left_motor = Kp * error;

else

power_right_motor = Kp* error;

-end loop-

44

Self-Balancing Robot

Calibrated Value = initial accelerometer readings;

Calibrated_roll = initial roll;

Current_X = accelerometer X reading;

Current_Y = accelerometer Y reading;

Current_Z = accelerometer Z reading;

// Low pass filter

while(Accelerometer_reading)

-do-

FX = a*current_X + (1-a)*FX;

FY = a*current_Y + (1-a)*FY;

FZ = a*current_Z + (1-a)*FZ;

roll = atan2(-FY,FZ);

error = roll - calibrated_roll;

final_error = error;

diff = final_error - initial_error;

initial_error = error;

sum = sum + error;

Power = Kp*error + Kd*diff + Ki*sum;

if(power > 100)

power = 100;

else if (power<-100)

power = -100;

left_motor_power = power;

right_motor_power = power;

-end while-