Open Access Open Access  Restricted Access Subscription Access

Control of Robot Arm Based on Hand Gestures Using Leap Motion Sensor Technology

Hiroshan Gunawardane, Nimali Medagedara, Achintha Madhusanka


This paper presents the mathematical model, design and the implementation of the hand gesture controlled robot manipulator. Nowadays various types of robots are used in medical industries. For the case of surgeries, surgeon uses either direct or through the computer control methods to control instruments required.  For electro mechanical designing of a bilateral robot manipulators Controlling is identified as one of the key issues. Hand Gesture controlling is one of the major developments emerged in recent past. A technique of controlling robot manipulators in slave side of the bilateral system using Leap Motion Contoller(LMC) based gesture detections is developed with the help of this research. The mathematical model for inverse kinematics, dynamics and trajectory generation is developed and simulated in the “Wolfram Mathematica 10” . This paper shows the possibilities of optimizing the joint actuator by improving the trajectory planning. The model is verified using the MATLAB simulation in real-time. The simulation is developed using the MATLAB Robotics Tool Box 9.10 to imitate the fingertip. A 3DOF (Degree of Freedom) customized robot model was tested with the developed mathematical model. The prototype was developed using MG995 servo motors and Processing IDE. The development was succeeded in real-time but due to the use of low cost servos, the precession of the system was not very high. In order to apply this system in surgical type applications, the system should be implemented with high precession robot servos; such as Dynamixel robot actuators.   



Bilateral Systems; Fingertip Imitating; Robot Manipulator; Matlab Robotics Tool Box; Robot Simulation; Kinematic Model; Robot Control; Leap Motion Controller; Hand Gesture

Full Text:



K. M. Tsui, "Design and evaluation of a visual control interface of a wheelchair mounted robotic arm for users with cognitive impairments," 2008.

N. Popescu, M. Poboroniuc, D. Popescu and C. D. Popescu, "Intelligent haptic roboticglove for patients diagnosed with cerebrovascular accidents," in IEEE International Conference on Rehabilitation Robotics, Iasi, Romania, 2013.

M. T. Wolf, C. Assad, M. T. Vernacchia, J. Fromm and H. L. Jethani, "Gesture-Based Robot Control with Variable Autonomy from the JPL BioSleeve," in 2013 IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany, 2013.

K. Yanagisawa, H. Sawai and H. Tsunashima, "Development of NIRS-BCI system using perception," in 2012 12th International Conference on Control, Automation and Systems, Jeju Island, Korea, 2012.

A. Harada, T. Nakakuki, M. Hikita and C. Ishii, "Robot finger design for myoelectric prosthetic hand and recognition of finger motions via surface EMG," in Proceedings of the International Conference on Automation and Logistics, Hong Kong and Macau, 2010.

M. V. Liarokapis, P. K. Artemiadis, P. T. Katsiaris and K. J. Kyriakopoulos, "Learning Task-Specific Models for Reach to Grasp Movements: Towards EMG-based Teleoperation of Robotic Arm-Hand Systems," in The Fourth IEEE RAS/EMBS International Conference, Roma, Italy, 2012.

H.-P. Huang, J.-H. Chen and H.-J. Jian, "Development of the Joint Attention with a New Face Tracking Method for Multiple People," in IEEE International Conference on Advanced Robotics and its Social Impacts, Taipei, Taiwan, 2008.

D. Guo, X. M. Yin, Y. Jin and M. Xie, "Efficient gesture interpretation for gesture-based human-service robot interaction".

M. G. Jacob, Y.-T. Li and J. P. Wachs, "Surgical Instrument Handling and Retrieval in the Operating Room with a Multimodal Robotic Assistant," in 2013 IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany, 2013.

P. D. S. H. Gunawardane, N. T. Medagedara and B. G. D. A. Madhusanka, "Studying the Competitiveness of Master Designs in Bilateral Systems for an Epidemic Environment," on 2015 IEEE International Conference on Ubi-Media Computing), Colombo, Sri Lanka, 2015.

"Kinect for Windows Sensor Components and Specifications(Library, Microsoft MSDN)," 2014. [Online]. Available: [Accessed 7 Jan. 2015].

"SR400 (Imaging, MESA)," 2014. [Online]. Available: [Accessed 7 Jan. 2015].

"Reference Design Brief CamBoard (, Leading 3D chip technology provider)," 2014. [Online]. Available: [Accessed 7 Jan. 2015].

"Bumblebee XB3 1.3 MP Color FireWire 1394b 3.8mm (Sony ICX445) Imaging, POINT GREY," 2014. [Online]. Available: [Accessed 7 Jan. 2015].

"Leap Motion Product," 2014. [Online]. Available: [Accessed 7 Jan 2015].



  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.