As a follow-up to their “Das Rheingold – Visionized” pilot project, the Ars Electronica Futurelab again collaborated with artist Johannes Deutsch on a performance of Gustav Mahler’s Symphony Nr. 2 in C minor as an interactive visualization in three-dimensional space. This work, commissioned by WDR, Western Germany’s public broadcasting station, premiered on January 1, 2006 to kick off WDR’s 50th anniversary year. The Cologne Philharmonic’s performance got rave reviews for its visionary staging in a new dimension. For WDR’s live broadcast, Futurelab staffers worked out a concept in which the interactive environment is captured by four virtual cameras.

The Visionized version of the Resurrection Symphony stages a three-dimensional virtual world of visual imagery that is coordinated to the tonal imagery being produced in the concert hall.
In this largely unreconnoitered experimental terrain in which innovative artistic approaches are encountering the possibilities afforded by computer technology, Mahler’s classic work was reread and interpreted anew. What was boldly daring about this visualization was the innovative fusion of different artistic genres in a brand new staging concept that visually translates the musical composition into a three-dimensional space. The pictorial imagery, based on designs by Viennese artist Johannes Deutsch, was interpreted with the help of software developed at the Ars Electronica Futurelab and adapted for depiction in three-dimensional space. Johannes Deutsch had subjected the piece of music to a thoroughgoing analysis and executed a substantive framework for the visuals in the form of 18 objects. The design concepts for the dramaturgical transformations of the 3D elements and the overall mise en scène translate the theme of the Resurrection Symphony into an aesthetic interplay of dynamic compositions of form and color that transport their own visual language. The link-up of the latest technological innovations (audio analysis, 3D graphics, virtual camera) with the concept of the visualization of a multimedial tonal space is thus opening up new modes of staging and partaking of classical works and thus creating new artistic approaches.

Link-up of a Virtual Environment and a Tonal Space
The interaction concept for the connection of the virtual environment with the tonal space in the concert hall makes possible a computer-controlled interpretation of the musical impulses, which can flow in real time via 48 channels into the visual depiction. Audio signals are analyzed according to a number of different criteria and linked up to the visuals. They influence both individual elements as well as the overall performance and lead to an interrelationship of the modulation of the spatial imagery with the playing of the musicians and spontaneous reactions by the visualization to selected tonal elements.

“Visionized”: Performance in Actual Practice
The outcome is a visionary way of performing a musical work, one that, in the words of Cologne Mayor Fritz Schramma, brings forth “a new dimension of music and optical perception.” Through stereoscopic projection, the imagery is displayed in a three-dimensional virtual space that overlays the real space of the concert hall and envelops the audience. This gives rise to the visual impression that the design elements are moving about freely in space and that those present can partake of or perceive them in a wide variety of perspectives and dimensions—from immediate proximity to seemingly endless remoteness. The use of the Pulfrich Effect also makes it possible for the TV audience to perceive the three-dimensionality of the staged performance.

Intelligent Technologies
For the TV broadcast, the artificial environment is captured by a so-called virtual camera, one of the innovative new technologies that the Ars Electronica Futurelab came up with for this project. This device makes it possible to take leave of the audience’s fixed perspective and to navigate through the digitally construed world. As an “intelligent camera,” it adjusts automatically to the design criteria of the visualization and can be used to produce Pulfrich Effects via the televised image.
Several different technologies that had been developed by the Ars Electronica Futurelab were already being combined, modified, synchronized and enhanced for the production of “Das Rheingold – Visionized.” For instance, specialists took over a year to get a computer system to “learn” to “hear” the music and to interpret the modulations of the signals in accordance with design criteria. The pioneering work that was done in the follow-up project has led, in turn, to new technological innovations and software solutions.