Real-Time Gesture Recognition System for Game Playing
Abhilash Sunder Raj and Ameya Joshi
Introduction
Motivation
Virtual Reality (VR) is the field which deals with creating a virtual environment that the user can interact with. In recent years, the use of VR has been gaining popularity in a wide variety of applications. One of the most popular applications is the use of VR for playing video games. With the incorporation of Virtual Reality, the user can enjoy a fully immersive gaming experience. We were inspired by the idea of building a real-time system that can be used to play any existing video game in such a fully immersive gaming environment. We believe that this project is the first step towards achieving such a goal.
Problem Definition
Most computer games today have keyboard controls. In order to enjoy a fully immersive gaming experience, we must first eliminate the need for a keyboard. A natural way to do so would be to use hand gestures to play the game instead. The main aim of this project is to build a real-time system which can achieve this. The goals of such a system are three-fold:
1. Recognize the user's hand gestures.
2. Map the gestures to the keyboard controls of the game.
3. Override the keyboard controls so that the user can play the game without the help of a keyboard.
In addition, the following constraints also have to be met:
- The system must not have very high computational complexity. It should run on any ordinary laptop/desktop.
- The user should not have to wear special sensors/markers/gloves to facilitate gesture recognition.
For this project, we have chosen to implement a gesture recognition system for playing car racing games. Two major aspects common to all car racing games are steering (turning the car left/right) and acceleration. Currently, we have included these functions in our gesture recognition system.
Hardware
This project was implemented on a Macbook Pro laptop using the Intel® RealSense™ Camera (F200).
The Intel® RealSense™ F200 is a front facing, depth imaging camera. It uses structured light techniques to create a depth image of the scene. It produces three image streams:
- Depth stream: 640x480 resolution at 60fps
- IR stream: 640x480 resolution at 60fps
- RGB stream: 1080p at 30fps
Implementation
The crux of this project is designing an efficient gesture recognition system. The user's gestures are identified solely from the image stream captured by the camera.
In this project, we implement steering and acceleration functions used for playing car racing games. We have tried to make the system as intuitive as possible. In order to play the racing game, the user visualizes a steering wheel in his/her hands. He/She turns the car left or right by turning the wheel. Whenever the user wants to accelerate, he/she raises his/her thumb. The system should detect these gestures and translate it into equivalent commands in the game.
One of the major challenges in this project is the time constraint. Sophisticated image processing techniques are computationally expensive and will drastically drop the frame rate if implemented on an ordinary laptop. This will result in a very noticeable lag between the user's actions and the response on the screen.