Teleoperation of Mobile Robot Using Event Based Controller and …conf-scoop.org/ACE-2014/2.Shahzad_ACE.pdf · 2014. 9. 2. · telesurgery [15], and recently teleoperation of mobile

Teleoperation of Mobile Robot Using Event Based Controller and Real Time Force Feedback

Aamir Shahzad Automatic Control Engineering Department

University of Siegen Siegen, Germany

[email protected]

Hubert Roth Automatic Control Engineering Department

University of Siegen Siegen, Germany

[email protected]

Abstract—Event based controller has been implemented to teleoperate the real mobile robot efficiently. The system consists of master haptic device, slave robot and a communication network. On master side with the help of visual aid and real time force feedback acting on the robot the operator control and navigate the robot and receive sensory feedback. Environmental force which is acting on slave robot is modeled as virtual force based on obstacles in front of mobile robot using proximity sensors and it has been reflected to human operator in real time using perditor block. Thus the operator can feel that he is driving the robot like a car while he is present at remote location. The designed controller shows the excellent coordination between master haptic device and slave robot. The slave robot follows the master device and communication delay has no effect on the performance and stability of teleoperated robot.

Keywords— teleoperation; event based controller; force feedback; haptic device

I. INTRODUCTION

In fact, teleoperated robots are excellent mean to work in hazardous environments where human safety is at high risk like nuclear power plants, landmines clearance and space exploration[1-4]. Also teleoperation provides solutions in cases where human operators simply can’t manipulate given objects like surgery inside human body through micro-robots which is called tele-surgery. Teleoperation is finding applications in these areas because the technology can save lives and reduce cost by removing the human operators from the operation sites. However, most of these areas still need humans in the control loop because of their very high level of skills and because machine intelligence is insufficiently advanced to operate autonomously and intelligently in such complex and unstructured environments. Teleoperation has become one of the most rapidly expanding areas in mechanical, electrical, computer and control systems engineering.

Today many industries utilize robots because they offer advantage of being able to perform set routines more quickly, cheaply, and accurately than humans. Instead of using programmed routines to maneuver the robots, tele-robotics allows to operate the robot from a distance and make decisions in real time[5]. With the development of more powerful and efficient computers, the future for teleoperation seems

extremely promising. On the other hand, the active research in teleoperation is being conducted using Internet as communication medium. This has happened due to the fact that the Internet has changed from a simple data transmission medium to a virtual world application like control. The system which uses real time control over the Internet has many difficulties. One of the most important difficulty is the delay due to the data packets transmission between two points over network. This delay due to its random nature plays a significant role in the stability and efficiency of the system when the commands are sent and received in real time applications. Furthermore, when the Internet began to be used for communication, packet switched networks presented the already established time-delay analysis with difficulties due to randomly varying delays, discrete-time exchange of data and loss of information. So that earlier delay related results were adapted to the new setting as it was studied in detail in [6] as well as to discrete-time setting in [7-10] and information loss in [11]. These methods found their way to several applications in handling radioactive material [12], space robotics [13,14], telesurgery [15], and recently teleoperation of mobile robots [16,17].

Moreover, several Internet based robots have been developed and studied. Reference [18] where they considered the bilateral teleoperation of a wheeled mobile robot over communication channel with constant delay to enable the user to control the mobile robot by operating a master haptic joystick. The passivity of the closed-loop system is also enforced so that, even with communication delays, humans can stably and safely teleoperate the wheeled mobile robot with force-reflection. However, this study was based on simulation framework. Reference [19] the use of a haptic interface is proposed to increase the user’s perception of the workspace of the mobile robot. The passivity of the overall system is preserved, so that the stability of the virtual interaction is guaranteed. But the system behavior was not evaluated in complex tasks and also it did not take into account the significant time delay in the data transmission. Reference [20] presented the Internet-based tele-rehabilitation sharing system, whose aim is to achieve the situation where multiple stay home patients in different places can share rehabilitation instruction of one physiotherapist at the same time. However, they have done simulation which is hard to

Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields 22-23 August 2014, Koc University, ISTANBUL/TURKEY

7

implement and the real experiment is under planning in that research.

In this work a bilateral control of mobile robot is presented with real time force reflection to operator which is acting on mobile robot without any assumption on time delay using event based control approach. The virtual interaction force is computed on the basis of obstacles in front of the mobile robot. Thus, the live video feed and force feedback from the robot help the operator to drive robot like a car by generating linear velocity equivalent to gas pedal adjustment and heading angle equivalent to steering wheel rotation in car, commands from haptic joystick.

II. PROBLEM DESCRIPTION

In fact, the teleoperation over Internet suffers with time delay. This delay happens due to latency in communication via Internet. The main effects of this delay are instability and de-synchronization. The previous researches assumed the delay time is constant or has upper bound limit[18][21]. In order to avoid these assumptions over delay an event based controller has been implemented which has no impact of delay on it. Also, a predictor block is implemented in the feedback loop that reflect real time force acting on the robot to the operator as shown in Fig. 3.

III. NON-TIME BASED CONTROL FOR TELEOPERATION WITH FORCE REFLECTION

Dif ferent approaches have been used to stabilize the teleoperated robots. The stability in this work is ensured by using event based controller. The Fig. 1, and Fig. 2, show the conventional control block and event based control block respectively. Thereom1 explains the stability of the event based controller. The proof of this theorem has been done in [22].

Fig. 1. Conventional Control Loop

Fig. 2. Non-time based control

A. Theorem1 If the original robot dynamic system(without remote

human/autonomous controller) is asymptotically stable with time t as its action reference and the new non-time action reference, e=Π(y) is a (monotone increasing) non decreasing function of time t, then the system is (asymptotically) stable with respect to the new action reference e. The advantage of this approach is that stability is independent of random time delay.

IV. THE CONTROL APPROACH

Fig. 3. Block diagram of the teleoperated system.

The telecontrol of mobile robot has been implemented as shown in Fig. 3. Haptic feedback is very crucial in telecontrol along with vision and sensory feedback to perceive the environment around the robot. The force acting on the slave robot is fed to master device so that operator can feel the real impact of force acting on slave robot. In telecontrol there is delay due to which force acting on the hapatic device is a delayed response. The force is modeled as virtual force which is acting on robot and is inversely proportional to distance to obstacle in front of robot. With the predictor block it can be made sure that the real time force is generated by using the Tdb(delay time backward), velocity of slave and onboard proximity sensors to calculate the real time position and hence virtual force.

The operator generates joystick position Xm(e), as it is given in the (1).

Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields 22-23 August 2014, Koc University, ISTANBUL/TURKEY

8

hC

ehFe)(

)(mX = (1)

Where Ch is constant and e is the event.

=)(mX

)(myX

)(mxX

)(mX

e

e

e

e

φ

(2)

=

)(

)(

)(

)(

ehF

ehyF

ehxF

ehF

φ

(3)

Where Fh(e) given in (3) is applied force by human operator and Fm(e) is reflected force. Fhx(e), Fhy(e) and Fhφ(e) generate Xmx(e), Xmy(e) and Xmφ(e) positions of the joystick respectively. The dynamics of joystick is given in the (4), where Mm is mass of master joystick and Vm(e) is velocity of master joystick. Each event is triggered by the previous event as can be seen in the (4).

)()()1( emFehFemV

mM +=+

• (4)

=)(mV

)(myV

)(mxV

)(mV

e

e

e

e

φ

(5)

)()()(sV eenVemVe −= (6)

=)(oV

)(oyV

)(oxV

)(oV

e

e

e

e

φ

(7)

Vm(e) travels through communication channel with some delay but this delay has no effect on the performance of system since advancement in time has no effect on slave robot only advancement in event e stimulates the slave robot. Therefore when there is connection loss then the slave robot will stop and wait for new event. After reconnection the slave will start following master again. Vs(e) mentioned in the (6) is slave input velocity and it is same as Vm(e) when there is no obstacle in front of robot. Ven(e) is reduction in Vs(e) when there is some obstacles to reduce the speed of robot. The proximity sensors mounted on the slave robot scan the environment and adjust the Vs(e). Vo(e) is output velocity of slave robot given in the (7).

V. TELEOPERATED SYSTEM

A. Hardware System Fig. 4, illustrates the pictorial view of the teleoperated

system. It consists of force feedback joystick, client pc and AutoMerlin along with server PC. Haptic device i.e. force feedback joystick from Wingman generates the commands for robot navigation and reflects the force feedback to human operator. AutoMerlin (Auto Mobile Experimental Robot for Locomotion and Intelligent Navigation) is a four wheel car like mobile robot as shown in Fig. 5, which is equipped with dspic microcontroller and onboard sensors. The robot is driven by two dc motors and the power is transmitted equally to all four wheels. The front wheels are steered by servo motor. The controls are realized by pulse width modulation (PWM) signals from the microcontroller to the drive and steer motors, respectively. The data transfer between AutoMerlin and the server PC has been realized by UART communication using RS232 serial port.

Fig. 4. Pictorial View of the system.

Fig. 5. AutoMerlin

Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields 22-23 August 2014, Koc University, ISTANBUL/TURKEY

9

B. Software Framework Two algorithms have been developed in C# language for

both client and the server. The client part is hosted in client PC and it is taking inputs (velocity and steering angle) from the joystick. Then, it has the ability to send information to the server PC via TCP/IP. It reflects force feedback to haptic device that is the force feedback joystick. Second part is the server which is hosted in the server PC. The server algorithm receives velocity commands from the client and process them according to environment e.g. if there is no obstacle then Vm(e) is fed as Vs(e) and if there is obstacle in the range of 0.5m then Vm(e) is adjusted to reduce the Vs(e), and if the object is in the range of 0.2m then the robot will stop. The server execute the velocities commands for 200ms and then wait for the next event. So each event has duration of 200ms. Client generates new commands as next event. These commands travel through network and reach server algorithm which sends them to AutoMerlin via serial port. Server algorithm receives sensor data from the robot and sends them back to the client. Also, it sends the visual feedback from the vision sensor (camera) to enhance the perception of operator about the environment around the slave robot as shown in Fig. 6.

TCP/IP is reliable and guarantees sending-receiving data, therefore it has been used for important data transfer (velocity, heading angle, and sensor data). While UDP is faster than TCP and packet loss is acceptable in video streaming, therefore it has been used for sending live video stream from robot to client.

Fig. 6. Client GUI

VI . EXPERIMENTS

Some experiments were carried out over local network to check the behavior of the controller when there is perfect connection between client and server and then we made deliberate disconnection and reconnection to evaluate the performance of controller.

Fig. 7. The master heading angle.

Fig. 8. The slave heading angle.

Fig. 7, and Fig. 9, show the heading angle and linear velocity of master device i.e. haptic force feedback joystick when there is perfect connection between master and slave. Fig. 8, and Fig. 10, show the slave robot heading angle and linear velocity. It is clear from these four plots that the slave robot is following the master device i.e. haptic force feedback joystick.

Fig. 9. The master linear velocity.

Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields 22-23 August 2014, Koc University, ISTANBUL/TURKEY

10

Fig. 10. The slave linear Velocity.

Fig. 11, and Fig. 12, show the master linear velocity and slave linear velocity with disconnection. In the beginning the master and slave linear velocity is zero then slave follows master until there is connection loss and both master and slave velocities become zero. When the connection is reestablished the slave starts following the master again.

Fig. 11. The master linear velocity with disconnection.

Fig. 12. The slave linear velocity with disconnection.

Fig. 13, plots the master linear velocity, Fig. 14, plots the slave linear velocity and Fig. 15, the force acting on the robot. These plots are related with each other. When there is no obstacle the slave follows the master. At event 50 there is some obstacle so velocity is reduced and force is increased as shown in Fig. 14, and Fig. 15. Slave is not following master now but instead it is following the algorithm developed in server program to avoid obstacles. So when there is maximum force the velocity is of slave is minimum regardless the velocity of master device and when the object is in the critical range of 0.2m the slave will stop and force will become maximum and it will not listen to master device.

Fig. 13. The master linear velocity when there are obstacles.

Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields 22-23 August 2014, Koc University, ISTANBUL/TURKEY

11

Fig. 14. The slave linear velocity when there are obstacles.

Fig. 15. The force acting on slave reflected to master device.

VI I. CONCLUSION AND FUTURE WORK

The above mentioned results have been plotted to analyze the performance of controller and the coordination between master and slave. The results are quite impressive and exhibit the excellent coordination between master and slave. In future work the map building will be added to the GUI so that the human operator can understand the environment around slave robot more precisely and the vision system will be used to detect and localize humans in the environment and then send to them the rescue robot after detection.

REFERENCES [1] O. Linda and M. Manic, "Self-organizing fuzzy haptic teleoperation of

mobile robot using sparse sonar data", IEEE Transactions on Industrial Electronics, vol. 58, no. 8, august 2011.

[2] I. Farkhatdinov, J. H. Ryu, and J. An," A preliminary experimental study on haptic teleoperation of mobile robot with variable force feedback gain," IEEE Haptics Symposium Waltham, Massachusetts, USA,25 - 26 March 2010.

[3] K. M. Al-Aubidy, M. M. Ali, A. M. Derbas, and A.W. Al-Mutairi," Gprs-based remote sensing and teleoperation of a mobile robot, "10th International Multi-Conference on Systems, Signals & Devices (SSD) Hammamet, Tunisia, March 18-21, 2013.

[4] S. K. Cho, H. Z. Jin, J. M. Lee, and B. Yao," Teleoperation of a mobile robot using a force-reflection joystick with sensing mechanism of rotating magnetic field," IEEE/ASME TRANSACTIONS ON MECHATRONICS, VOL. 15, NO. 1, FEBRUARY 2010.

[5] Z. Szanto, L. Marton, P. Haller, and S. Gyorgy,"Performance analysis of WLAN based mobile robot teleoperation," IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, 5-7 Sept. 2013.

[6] R. Lozano ,N. Chopra , and M. W. Spong , "Passivation of force reflecting bilateral teleoperators with time varying delay," In Mechatronics’02, Entschede, Netherlands,2002.

[7] P.Berestesky ,N. Chopra , and M. W. Spong, "Discrete time passivity in bilateral teleoperation over the Internet," In Proceedings of the IEEE international conference on robotics and automation, New Orleans, LA, USA,2004.

[8] J. H. Ryu, D. S. Kwon, and B. Hannaford , "Stable teleoperation with time domain passivity control," IEEE Transactions on Robotics and Automation, vol. 20, no. 2, April 2004.

[9] C. Secchi, S. Stramigioli, and C. Fantuzzi, "Dealing with unreliabilities in digital passive geometric telemanipulation," In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems Vol.3, 2003, pp. 2823–2828.

[10] Y. Yokokohji,T. Imaida, and T. Yoshikawa, "Bilateral control with energy balance monitoring under time-varying communication delay,". In Proceedings of the IEEE international conference on robotics and automation,Vol. 3, San Francisco, CA, USA,2000, pp. 2684–2689.

[11] C. Secchi, S. Stramigioli, and C. Fantuzzi, "Digital passive geometric telemanipulation," In Proceedings of the IEEE international conference on robotics and automation Vol. 3,2003, pp. 3290–3295.

[12] W. Wang and K. Yuan, "Teleoperated manipulator for leak detection of sealed radioactive sources," In Proceedings of the IEEE international conference on robotics and automation, Vol. 2,2004, pp. 1682–1687.

[13] W. K. Yoon, T. Goshozono, H. Kawabe, M. Kinami, Y. Tsumaki, and M. Uchiyama, "Model-based space robot teleoperation of ETS-VII manipulator," IEEE Transactions on Robotics and Automation,2004.

[14] T. Imaida, Y. Yokokohji, T. Doi, M. Oda, and T. Yoshikwa, "Groundspace bilateral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition," IEEE Transactions on Robotics and Automation, 2004.

[15] A. J. Madhani, G. Niemeyer, and J. K. Salisbury, "The black falcon: A teleoperated surgical instrument for minimally invasive surgery," In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems Vol. 2,1998, pp. 936–944.

[16] N. Diolaiti and C Melchiorri, "Teleoperation of a mobile robot through haptic feedback" In IEEE international workshop on haptic virtual environments and their applications,2002,pp. 67–72.

[17] O. J. Rösch, K. Schilling, and H. Roth, "Haptic interfaces for the remote control of mobile robots" Control Engineering Practice, 2002.

[18] L. Dongjun, M. P. Oscar, and M. W. Spong, " Bilateral teleoperation of a wheeled mobile robot over delayed communication network," Proceedings of the IEEE International Conference on Robotics and Automation, Orlando, Florida - May 2006, pp. 3298- 3303.

[19] N. Diolaiti and C. Melchiorri, "Haptic tele-operation of a mobile robot", IFAC 2003.

[20] Z. Xiu, A. Kitagawa, H. Tsukagoshi, C. Liu, and M ido," Internet-based tele-rehabilitation system bilateral tele-control with variable time delay," Proceeding of the 2006 IEEE/RSJ, International Conference on Intelligent Robots and Systems, pp. 5208-5213, Beijing, China, October 2006.

[21] M. P Oscar, L. Dongjun, M. W. Spong, I. Lopez, and C. T. Abdallah, "Bilateral teleoperation of mobile robot over delayed communication network: Implementation," Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China, October 9 - 15, 2006.

[22] N Xi and T J Tarn, “Action synchronization and control of Internet based telerobotic systems,” IEEE Int. Conf. on Robotics and Auto, May 1999.

Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields 22-23 August 2014, Koc University, ISTANBUL/TURKEY

12