The Further Adventures of Surfaces, Sounds, Pixels and Colors
3-D Music Visualizations from the Ars Electronica Futurelab

' Ars Electronica Futurelab Ars Electronica Futurelab

The virtual staging of Richard Wagner’s Das Rheingold in 2004 was the Ars Electronica Futurelab's first milestone in a series of experimental visualizations that have pioneered new approaches to presenting classical works of music featuring linkages between the music and three-dimensional visuals. Following up on this effort a year later, Futurelab staffers again worked with graphic artist Johannes Deutsch to produce Gustav Mahler’s 2nd Symphony as an interactive visualization in three-dimensional space. The treatment of Igor Stravinsky’s Le Sacre du Printemps as a blend of dance performance, visuals and tonal space in collaboration with media artist, director and composer Klaus Obermaier is this year’s sequel.

The link-up of auditory and visual sensory levels would seem to be a natural. After all, in extreme cases—certain physiological conditions or drug-induced states—image and sound do indeed merge into a single sensory experience. Thus, it’s no wonder that the resulting profusion of creative possibilities have also aroused artistic efforts to deal with certain themes. There have also been individuals who experience synaesthesia who have used this unique form of creativity to produce new and visionary works. Franz Liszt, for instance, composed pieces in which he set a particular painting to music, and Wassily Kandinsky painted music that he saw. This special gift also provided the impetus for the first devices—first and foremost, Alexander Scriabin’s “clavier à lumière”—that were designed to give users access to these subjective visions and thus to enable them to experience the music more intensively. In the 1920s, Alexander László attempted to establish his so-called color-light-music as an “art of the future,” and he successively expanded and enhanced his “Sonchromatoscope” into a regular multimedia infrastructure.

The numerous approaches to using digital means to treat a substantive theme continued all the way up to the infancy of computer graphics. Visuals for every occasion have long been a part of the standard repertoire of the culture of everyday life, and the upshot has been a broad spectrum of experimentation ranging from the more or less random play of color all the way to attempts to depict sounds and music with scientific exactitude. The Brucknerhaus’ long-term working relationship with the Ars Electronica Futurelab yielded a concrete plan to get involved in exploring this experimental field, and these joint ventures have in turn created more and more opportunities for the traditionally oriented Brucknerfest to hook up with the avant-garde Ars Electronica. Wolfgang Winkler, artistic director of the Brucknerhaus, approached the Ars Electronica Futurelab with the idea of bringing together classical performances with three-dimensional worlds of imagery. The staff of the Futurelab under the direction of Horst Hörtner then went about developing the main elements of an interaction design for this new form of performance.

The Ars Electronica Futurelab’s visualizations are neither directly related to synaesthesia, nor do they constitute the attempt to translate from one form of sensory perception into the other. Rather, the media lab brings forth a sphere of audiovisual fusion that enables interplay to occur between the composition, the conductor’s interpretation of it, the musicians, and the work of artists in many disciplines.

For Das Rheingold, several different artistic approaches and technological solutions developed by the Ars Electronica Futurelab were combined, modified, coordinated with one another and enhanced. Working together for more than a year, the project team came up with an array of tools that are able to “hear” the music and to interpret the modulations of the signals in accordance with artistic criteria. The linkage of technological development with conceptualizing a multimedia visualization of a tonal space thus opens up a new mode of configuring works of art and engenders new approaches to partaking of classical compositions.

Some of Richard Wagner’s works were infused by the mysticism of light. He had concrete conceptions of the images that would transport the audience into the world of Germanic mythology and—in what is tantamount to the performance of a ritual—obeisance is still being rendered to many of these ideas in the ways Wagner’s operas are staged today. Indeed, the visualization by Johannes Deutsch and the Ars Electronica Futurelab also followed the narrative prescribed by the author and offered the possibility of identification with the characters. Nevertheless, thanks to the stereoscopic projection and the viewer-encompassing screen, the audience, instead of just watching the stage, finds itself right in the middle of what’s happening. The performance space is filled by both images and sound. With the virtual world being linked up to the instruments and the vocalists’ voices, the visuals take on qualities of the sound, the result of which is to intensify the work’s emotional impact and to make possible a new, immersive mode of experiencing this classical piece.

The R&D effort that went into Das Rheingold constituted the basis for a subsequent music visualization project, the Resurrection Symphony by Gustav Mahler.

The occasion for this production was a gala concert to kick off the 50th anniversary celebration of WDR, the western German public broadcasting company. The piece was conducted by Semyon Bychkov in Cologne’s Philharmonic Hall and also televised live.

The libretto did provide a point of departure for virtually staging Das Rheingold, but it was necessary to try a very different tack here. The elements of the symphony are indeed pregnant with significance and plot developments are identifiable; nevertheless, it was completely out of the question to make the visual representation of the tonal space more concrete than what the music itself was saying.

The Ars Electronica Futurelab project team’s task was to immerse itself in the pictorial universe that Johannes Deutsch had invented for this piece and to utilize the possibilities of computer graphics to come up with its counterpart. Johannes Deutsch had performed a detailed analysis of the music and sketched 18 objects as a dramaturgical framework for the visualization. The audio signals are analyzed for particular characteristics and linked to a virtual representation. They influence both the individual elements, as well as the overall staging of the production, and lead to an interplay of the modulated spatial imagery synchronized with the musicians’ performance and the spontaneous reactions of the visualization to selected tonal events.

For the performance of the work, a procedure was developed that makes it possible to model the virtual world in successive steps into the tonal imagery. To achieve this, the symphony was recorded prior to the actual live concert using 48 sensors and an equal number of channels. Then, during the performance, this recording could serve as the basis for simulating those impulses that would have an impact on the visuals during the live concert. In this way, the graphic elements could be configured by an editor in step-by-step fashion in accordance with the impressions made by the music. The graphics are based, on one hand, on the 18 sculptures that were created by Johannes Deutsch and translated into 3-D models by the Ars Electronica Futurelab and, on the other hand, on functional graphic elements with which the digital spatial construction was composed. Furthermore, linking up the visuals with the music being performed in the concert hall also called for the development of an interaction vocabulary according to which the musical impulses from the computer could be interpreted and input in real time into the visual representations. Finally, with the help of software components, all the artistic elements were assembled into the grand-scale visualization that features full interaction in which the visuals are adapted to the musical activity. The foundation upon which all of this was constructed was Mahler’s composition.

Text: Pascal Maresch