Gesture-controlled 3D virtual koala pet that responds to your fan maneuvers, nodding and jumping interactively.
Koala Companion is an interactive installation that combines hand gesture recognition with 3D modeling using Three.js. The goal is to establish an emotional bond with a virtual pet — a stylized, charming koala — using only natural hand gestures.
Users don't need any physical controller. Just:
🖖 Hold up a ✌️ gesture to move the fan;
✊ Make a fist to start blowing wind;
👍 Or use the “Thankyou” gesture to trigger the koala's nodding + jumping animation.
Once the fan starts spinning, the koala's ears flutter in the wind, fur rustles dynamically, and it responds with adorable expressions and movements.
Inspired by childhood emotional connections with virtual pets, like Tomcat, Game Boy, I've always been interested in gestural interactions. And since I've studied machine learning, my thinking about “touch” has been redefined, as well as my longstanding interest in digital companions. My longstanding interest in digital companions forms the background for this project.
I am trying to create a virtual organism that “looks small, but acts warm”.
68104f57772325808.png_e1080.avif
The Koala was handcrafted using BoxGeometry blocks in Three.js, emphasizing a blocky and cute style, almost like digital LEGO.
The fan was constructed with four blades and animated rotation via rotation.z based on wind strength.
First version
I used the ml5.js handpose model to extract 21 keypoints of the hand. Then:
