Audio from Visuals only

Amazing algorithm extracts #audio from video shot without any audio in a soundproof glass!

No audio? No problem. A team of researchers from MIT, Microsoft, and Adobe has developed an algorithm that allows them to reconstruct an audio signal even when only visual information is available. Using nothing more than a high speed camera and a special processing algorithm, the team was able to extract the audio signals in a room from 15 feet away through soundproof glass.

How? Well when you get down to it, sound waves are really just tiny disturbances in the air. Therefore, when sound waves strike something delicate –say, a piece of tin foil, a bag of chips, or the leaves of a house plant– they cause the object to vibrate ever so slightly. It’s just like how the rear-view mirror vibrates when your buddy turns cranks the subwoofer on his car stereo, just on a much more minute scale. The waves that occur when you’re just having a conversation are much weaker, and tend to cause more minute vibrations. To the naked eye, these disturbances are practically imperceptible — but with the help of high-speed photography, the team was able to capture movements as small as a tenth of a micrometer, and then use that information to guesstimate and rebuild the audio signal. Check out the video to see it in action:

As if that wasn’t incredible enough, the team also a demonstrated variation on the algorithm that allows them to extract sound from ordinary 60 frame per second video footage. Generally speaking, the sensors on most digital cameras are designed to scan images horizontally, one row at a time. Normally, that’s not a problem, but when you’re shooting fast-moving objects, this can sometimes leads to odd visual artifacts. The team was able to exploit this technological quirk to tease out information about the objects’ high-frequency vibration and, once again, use that info to reconstruct a usable (albeit murky) audio signal. It’s not quite as clear as the audio signal ripped from the high-speed camera, but even so, the fact that this kind of reconstruction is possible is mind blowing.

The researchers are presenting their work at the computer graphics conference Siggraph this month.

 

Tags: audio