3 years ago

Category: Database, artificial-intelligence

Films are generally immersive experiences, made with the aim to impress their viewers with their engaging plotlines and dazzling special effects. Now, new work from a University of Texas at San Antonio research team shows that the Foley process can be automated — using artificial intelligence that can analyze motion in a given video, and then generate its own matching artificial sound effects. Dubbed AutoFoley, the team’s system uses deep learning AI to create what they call a “deep sound synthesis network,” which can analyze, categorize and recognize what kind of action is happening in a video frame, and then produce the appropriate sound effect to enhance video that may or may not already have some sound.

AutoFoley then attempts to ensure that the sound matches the timing of the movements in each video frame.

Next, AutoFoley synthesizes a sound to correspond with the action identified from the video in the previous steps.