A Leap Motion based system developed to replace the mouse
for
Human Computer Interaction and Information Visualisation
☞
For this HCI research project, my teammate and I built HandyGestures, a system that lets people browse the web using hand gestures instead of a mouse. I handled the user research side, designing studies, testing with real users, and analyzing how well our system worked compared to regular mouse navigation. My teammate took care of the technical implementation using Python to connect the Leap Motion device with web browsers.
Our main research question was simple: can hand gestures work as a practical alternative to mouse input for web browsing? We created a set of intuitive gestures like making a fist to scroll, pinching fingers to click, and using multiple fingers for tab management and navigation. The system responded in real-time, which made the interactions feel natural and immediate.
We tested our system with computer users aged 18-40 and found some interesting results. While gesture navigation was about 3 times slower than using a mouse initially, users really enjoyed the experience and found it easy to learn. The biggest challenge was getting the sensitivity right, people needed time to develop the precision for accurate clicking. But despite this learning curve, participants consistently said they found the gesture system more fun and engaging than traditional mouse input.
The evaluation showed that gesture-based web browsing is definitely feasible and users can adapt to it quickly. Even though it wasn't faster than a mouse, it opened up new possibilities for more accessible and enjoyable web navigation. This project taught me that successful HCI design isn't always about efficiency, sometimes the most valuable interfaces are those that provide novel, engaging experiences that make people rethink how they interact with computers. The key insight was finding the balance between innovation and usability.
☜
Sketch of the setup with the leap motion
☞
Here are some of the hand gestures we trained the model with.
☜
Project paper