Nao Robot with Kinect Camera for Gesture Pattern Classification and Control

Abstract

Aldeberan Robotics’ Nao Robot is currently one of the most advanced humanoid robots in the world. This project involved developing a neural network application to recognize the gesture pattern and control Nao robot based on Microsoft’s Kinect sensor. The integrated system used artificial neural networks and backpropagation learning algorithms for gesture classification as well as the Kinect’s skeleton data for replicating the user’s body position with the robot. A WPF (Windows Presentation Foundation) application was created in Visual Studios, using C#, for interaction with the Kinect – returning data about the user’s current body position. The output signals of the trained neural network were passed to python scripts which utilize the data and control the robot action. One such script was set up to record gestures from the Kinect. This data was used to train a multilayer perceptron artificial neural network to recognize variety of Kinect skeleton gestures. The trained neural network can perform gesture classification in real time and control the robot accordingly. Further modes of operation were developed whereby the robot would imitate the user; moving to the same position and orientation as the user in its own environment.

A large proportion of the paper was dedicated to neural network application and obtaining the programming skills required for the system development. The project was successful, providing excellent research opportunities and resulting in the establishment of a system that can fully control the Nao robot through any interaction with the Kinect. This platform is ideal for further development and could easily be adapted into a powerful tool for teaching Nao new behaviours or skills via human demonstration.