Control Music Using Hand Gestures – Built with Three.js & MediaPipe
Imagine waving your hands in front of your webcam — and instantly creating music.
No instruments. No mouse. Just your hands. Control Music Using Hand Gestures.
That’s exactly what this new open-source project does.
Developed by @measure_plan, this project lets you generate music in your browser using nothing but hand gestures. It’s built using a powerful mix of tools like Three.js, MediaPipe, and Tone.js, and the best part? It’s completely open-source and ready for you to explore.
Here’s the GitHub link to try it out yourself: 
GitHub project link
What Exactly Does It Do?
This isn’t just a visual trick — it’s an interactive audio experience where:
- You raise your hand to change volume and pitch
- You control a drum machine and arpeggiator in real time
- You see a live waveform visualizer reacting to your gestures
- All of this happens in your web browser, with no extra software
- The message on the screen says it best:
“Raise your hands to raise the roof.”
Tools Used Behind the Scenes
This project combines computer vision, web audio, and 3D rendering:
Three.js
A JavaScript library that brings 3D to the web. It handles the smooth animations, visual effects, and camera views in the browser.
MediaPipe by Google
It tracks your hand movements in real-time using your webcam. It’s what allows the project to understand gestures like raising hands or moving fingers.
Tone.js
This is the sound engine. It generates music in the browser — chords, beats, volume, and everything.
Rosebud AI (Optional Visual Enhancer)
Used for some creative visual touches and filters.
Together, these tools create an immersive experience — all running on your device without needing any downloads.
Want to Build Something Like This? Start Here 
If you’re feeling inspired to create your own gesture-controlled projects, here’s the best way to learn Three.js — the core 3D library used in this project.
Best Three.js Course on Udemy
Course Name: Three.js Tutorials – Learn By Building Projects
Instructor: Steven Hancock
Rating: ★★★★☆ (4.5+)
Duration: 9+ hours | 50+ lessons
Why it’s recommended:
This course is perfect for beginners who want to learn Three.js by actually building cool and practical 3D projects. Instead of long theory sessions, you’ll work hands-on with real examples—learning about 3D models, lighting, animation, camera control, and more.
Best Free YouTube Video to Learn Three.js
Video Title: Three.js Crash Course – Create 3D Websites.
Channel: Chris Courses
Duration: ~40 mins
Why Watch This?
This crash course walks you through creating a basic 3D scene, setting up a camera, and rendering it interactively — perfect for beginners.
Bonus: It also teaches how to animate objects and use lighting effects, just like the open-source project above.
Final Thoughts
This project is a perfect example of how creative coding is opening up new ways to interact with technology. You don’t need a fancy music studio — just a browser, webcam, and some imagination.
Whether you’re a developer, a hobbyist, or just someone who loves cool tech, this is a fun and futuristic project to explore.
Explore the full project here: Project Link
And if you’re inspired, start learning Three.js today — you might be one project away from building your own virtual instrument.
Bonus Skill: Learn N8N Automations with AI Agents
Full AI Automation Course
Unlock the future of automation with this hands-on course covering AI Agents, tools, and practical workflows for your projects.
What you’ll learn:
- Build and deploy AI agents
- Automate workflows using advanced tools
- Real-world projects and guided walkthroughs
Special Offer:
Use coupon code REPEATLESS to get ₹300 OFF Offer valid until August 1st Click here to buy the course
