Messa di Voce is a concert in which the performers’ speech, shouts and songs are radically augmented by custom interactive visualization software. The performance touches on themes of abstract communication, synaesthetic relationships, cartoon language, and writing and scoring systems, within the context of a sophisticated, playful, and virtuosic audiovisual narrative.

Golan Levin and Zachary Lieberman, who together form Tmema, created the software to transform every vocal nuance into correspondingly complex, subtly differentiated and highly expressive graphics. These visuals not only depict the singers’ voices, but also serve as controls for their acoustic playback. While the voice-generated graphics thus become an instrument which the singers can perform, body-based manipulations of these graphics additionally replay the sounds of the singers’ voices — thus creating a cycle of interaction that fully integrates the performers into an ambience consisting of sound, virtual objects and real-time processing.

Utterly wordless, yet profoundly verbal, Messa di Voce is designed to provoke questions about the meaning and effects of speech sounds, speech acts, and the immersive environment of language.

Messa di Voce is concerned with the poetic implications of making the human voice visible. Our core technology is a custom software system which integrates real-time computer vision and speech analysis algorithms. Specifically, a computer uses a video camera in order to track the locations of the performers’ heads. This computer also analyses the audio signals coming from the performers’ microphones. In response, the computer displays various kinds of visualizations on a projection screen behind the performers; these visualizations are synthesized in ways which are tightly coupled to the sounds spoken and sung by the performers.

Owing to the head-tracking system, moreover, these visualizations can be projected such that they appear to emerge directly from the performers’ mouths. In some of the visualizations, projected graphical elements not only represent vocal sounds visually, but also serve as a playable interactive interface by which the sounds they depict can be re-triggered and manipulated by the performers. In our concert, these techniques take shape in a series of twelve vignettes which explore different symbolic, tactile and audiovisual aspects of phonesthetic relationships.