Gulliver’s Box


'Adrian David Cheok Adrian David Cheok / 'Hirokazu Kato Hirokazu Kato / ' Ars Electronica Futurelab Ars Electronica Futurelab

Gulliver’s Box is a result of the Ars Electronica Futurelab’s collaboration with Prof. Hirokazu Kato (Osaka University). With the Futurelab focusing increased attention over the last two years on Mixed Reality applications—including the technology to run them and their potential areas of utilization in artistic and scientific fields—an initial meeting with Cheok and Kato was held in conjunction with Pixelspaces 2002, where they presented their work at a symposium. Then as well, they offered visitors to the OpenLab Exhibition the opportunity to try out Mixed Reality applications they had developed. Visitors’ positive experiences and reactions and the discussions triggered by their presentation ultimately gave rise to the idea of a joint project.

The developments that have been brought together in this installation represent the effort to pursue new approaches to dealing with Mixed Reality content. The challenge at the core of this project was to position an innovative medium somewhere between theater, film and installation. The result is an infrastructure that offers artists new opportunities to convey audiovisual information, and one that ought to encourage creatives in every discipline to work with these new approaches. Seen from this perspective, the platform that has been created in this way generates an experimental laboratory situation for a broad spectrum of forms of artistic expression. With it, performances by dancers, singers or actors can be recorded, transferred to avatars, and enhanced with any kind of computer animation. The application on display in the Ars Electronica Center also provides visitors with the opportunity to customize recordings of their own actions and subsequently to undertake a very special process of self-reflection. This unique aspect arises from the perspective of the viewer—just like in the world of huge Brobdignagians and tiny Lilliputians in Gulliver’s Travels, quantum dimensional leaps and the play of scale and relation are what shatter accustomed modes of seeing. Ultimately, the various approaches that go into Gulliver’s Box seem just as fantastic and horizon-expanding as the visions in Jonathan Swift’s novel. The performances rendered by this medium and the recordings of the visitors themselves are an inviting chance for viewers to fundamentally change their points of view or to reconsider them for once. The possibility of observing and manipulating the mise-en-scène from any desired position external to the action goes beyond the God-mode of computer games and seems to be unique in a media context. Interaction with characters—either those captured live or animated ones—used to be necessarily bound to monitors or projection screens, but Mixed Reality technology now gives rise to forms of artistic expression and reception in an intimate—albeit likewise projected—situation involving protagonists and viewers. In Gulliver’s Box, the processes of creative design, display and perception are brought together in a single environment.

Portions of the installation are, in turn, reminiscent of elements of the theater. These include a stage with a set, actors and a framework plot. The scene consists of animated characters, pre-produced footage and shots of the visitors that appear by means of head mounted displays on the stage—in this case, a table. The shots are made possible by the “3DLive” system (1) developed by Cheok that computer-generates a 3-D sequence out of numerous video images. The recording process is an integral part of the installation, and, in the Recording Area, visitors are given an introduction to this extremely promising form of data preservation. Then, with the help of optical markers, the video and audio information recorded in this way can be positioned on the play level however the user desires and played back.

The modular mode of scene construction allows the observer to intervene in the situation, modify it and determine the playback sequence. The tool to perform these manipulations is “MagicCup”, (2) a tangible interface that combines a number of features. The interface consists of a simple transparent cube whose position and movements are recognized and interpreted by an optical system. The user grasps the cube and places it on top of a virtual object being displayed in order to reposition it, move it to the foreground or background, copy it or delete it. Switching among the individual functional features is done by simply shaking the cube.

This ongoing project is testing a wide variety of procedures and seeking to identify the system’s full capabilities. But it will only be the process of gaining experience with the interplay of spontaneous, live 3-D recordings, artistic performances and animated characters and the public’s dealings in actual practice with the system’s technical tools for manipulation and interaction that will deliver a clearer picture of the implications of this Mixed Reality approach for future applications.

Translated from the German by Mel Greenwald
Text by Pascal Maresch / Christopher Lindinger

(1)
S.J.D. Prince, A.D. Cheok, F. Farbiz, T. Williamson, N. Johnson, M. Billinghurst and H. Kato. “3D Live: Real time captured content for mixed reality,” International Symposium on Mixed and Augmented Reality, 2002.back

(2)
H.Kato, M.Billinghurst, I.Poupyrev, K.Imamoto, K.Tachibana, “Virtual Object Manipulation on a
Table-Top AR Environment,” Proc. of IEEE and ACM International Symposium on Augmented Reality 2000, pp.111–119 (2000).back


Conception: Adrian David Cheok, Hirokazu Kato, Christopher Lindinger, Horst Hörtner, Nina Wenhart, Gerfried Stocker / Content: Li Yu, Pascal Maresch, Andreas Jalsovec, Christine Pilsl / Softwaredevelopment: Dan Borthwick, Simon Prince, Adrian David Cheok, Hirokazu Kato, Gernot Ziegler, Roland Haring, Wolfgang Ziegler, Stefan Feldler / Production: Rudolf Hanl, Martin Honzik, Gerold Hofstadler, Stefan Mittelböck-Jungwirth
Exhibition Design: Scott Ritter

Supported by the funding of DSTA Singapore
and the National Arts Council Singapore