From Synthesizers to AI: The Journey of MIT Professor Smaragdis

Computers are shifting from being told how to hear to being taught how to listen, moving away from rigid signal processing toward machine-learning models that can understand sound. This transition allows machines to identify complex audio events—such as the difference between a violin and a cat’s meow—that traditional techniques could not achieve.

Paris Smaragdis, a professor at MIT, developed this approach after realizing that humans lack a specific vocabulary to explain the act of listening. By utilizing machine learning, Smaragdis created systems capable of understanding audio contexts, such as identifying highlights in a recorded soccer game by detecting when a crowd “goes crazy.” This technology was eventually integrated into cell phones, traffic monitoring systems, and personal video recording devices during his time at Mitsubishi Electric Research Labs. Smaragdis later joined Adobe, where he created the audio research team that built user-friendly tools to help artists, musicians, and producers manipulate sound without needing a lot of expertise.

Smaragdis’ interest in sound began in the 1980s with a synthesizer that allowed him to create original sounds. Because traditional European conservatories did not offer a path to study synthesizer-based music, he enrolled at Berklee College of Music in Boston. There, he studied the use of computers to manipulate sound and majored in bass to engage with jazz fusion ensembles, noting that the bass was a “more unexplored domain” than the guitar. He says, “I’ve always gravitated toward unexplored domains.”

Academic Evolution at MIT

While at Berklee, Smaragdis worked with professor Richard Boulanger to develop tools for music production and collaborated with the late Professor Barry Vercoe at the MIT Media Lab to build software synthesizers. This partnership led Smaragdis to pursue a master’s degree, a PhD, and a postdoc within Vercoe’s group. During this period, his focus shifted from music creation to the goal of making computers that could understand sound.

Advancements in Audio Separation

His research group there focused on incorporating modern machine-learning techniques into audio-processing systems, specifically developing neural networks capable of separating individual sounds from complex audio mixtures. This project later attracted the production team for the 2021 documentary mini-series “The Beatles: Get Back,” who hired Smaragdis to assist in cleaning archival recordings of the band. Smaragdis noted that the work was a “very different way of audio processing” compared to his academic research.

The MIT Innovation Ecosystem

Smaragdis is now part of an MIT innovation ecosystem that includes alumni, faculty, postdocs, and students. This community utilizes mentorship and classes provided by more than 80 entrepreneurship and innovation organizations on campus. Central to this is MIT InnovationHQ, situated in what is described as “the most innovative square mile on the planet,” which supports those pursuing ideas for world-changing impact.

While these systems can now identify patterns in sound, it remains unknown how humans actually listen.

Worth a look