Artificial Intelligence

Google AI experiment lets you conduct an orchestra by waving at your webcam

Artificial intelligence identifies your limbs, then adjusts instruments, tempo and volume as you move

Like GearBrain on Facebook

Google' latest artificial intelligence experiment uses your webcam and a browser window to let you conduct a virtual orchestra.

Part of the 'Experiments With Google' series, the system uses AI to identify your head and limbs then turns your movements into instructions for adjusting the music. You can wave your arms around to pick sections of instruments to play, then adjust the volume and tempo.

Read More:

Buy Google Home Hub on Best Buy icon

Move your hands, and the virtual orchestra plays through the music using hundreds of tiny audio files gathered from live instrument recordings.

GearBrain had a play with the system before writing this article and, well, we're not classically-trained conductors, so the results were mixed to say the least.


While the music we created wasn't particularly good, we found the AI worked accurately — raising our arms increased the volume, pointing to the left or right changed which section of instruments was playing, and being more animated upped the tempo.

Google says: "It uses PoseNet, a machine learning library that works in the browser, to map out your movements through your webcam. An algorithm plays along to the score as you conduct, using hundreds of tiny audio files from live recorded instruments."

Buy Google Home Hub on Best Buy

Called semi-conductor (we see what Google did there), the system works entirely within your web browser, so you needn't worry about your embarrassing arm-flailing being uploaded to Google. That said, you'll need to give both the website and your web browser permission to use your webcam before the system can be launched.

You also need to stand a good distance away from the camera, so the AI can work out where your arms are in relation to your head and torso.

There's only one piece of music available for now, the first movement of Mozart's Serenade Number 13 'Eine kleine Nachtmusik,' which should be recognizable to most readers after successfully stumbling your way through the first couple of bars.

You can head to

this Google website to try the conductor AI for yourself.Check out The GearBrain, our smart home compatibility checker to see the other compatible products that work with Google Home and Home Mini

Buy Google Home Hub on Best Buy

delete Unboxing in 10 Seconds - Lenovo Smart Display with Google Assistant

Like GearBrain on Facebook
Show Comments ()