Virtual Reality for Collective Cognitive Processing


'Derrick de Kerckhove Derrick de Kerckhove

Just as AI stands universally for Artificial Intelligence, Virtual Reality is henceforth sufficiently current as to be designated as VIZ. But Virtual Reality could just as well have been called "Artificial Imagination." If we take the concept to its natural, if not its farthest reaching conclusion, we shall soon find that the purpose of VR is to enable us to command technological simulations by thought alone. The link between imagination and technology goes back to Ancient Greece.

From the invention of the alphabet to that of the computer, western man has been groping towards the development of Virtual Realities in different guises. Ancient Greek dramatists an novelists from the Renaissance to this day have been producing virtual realities by transposing the workings of their fantasies on stage or on paper. Any novel is a virtual world and its purpose, besides entertaining the reader, is both to train the imagination and to blueprint new lifestyles. Likewise, the contents of VR experimentations are generally in line with those which make up the content of the standard literate psychology. Just as videogames do not stray very far from the folklore of epic literate traditions, action‑packed scenarios, promoting legendary skills and usually – but not always – heroic motives, the family‑room VR will undoubtedly capitalize on archetypal characters drawn from novel, theatre, film, and TV, recapping the history of our imagination from the Grail to Madonna.

However, by itself, if it is merely used to animate passive digital environments, VR is not such a big deal. We have already developed complex simulations of 3‑D objects, relationships and movements on screen. Albeit a technical feat, 3‑D television, conceptually speaking is hardly more than walk‑in television. We already do that in our shopping malls to more effect. The fundamental difference between VR and any other media experience is that it enables the user to externalize his or her imaginary skills in a dynamic process. With VR, there is an effective interaction between the dreamer and the dreamed: the dream becomes real, as if Hegel's most cherished theories of matter turning into spirit were reversed. With VR, spirit can become matter directly and have a concrete expression in the user's environment. This enables the private experience to become public and to be shared objectively with other persons. Another critical difference is touch: the VR industry is about to recap the contents of films and TV adding one more sensory response to feed our craving for complete experience.


1. Touch

In fact, our relationship to the screen is about to change radically. VR is often compared with theatre. But theatre and VR differ significantly: in the theatre, we look into a comprehensive world from which we are personally excluded. We are outside looking in (which, by the way, is the standard response of the western man to reality anyway). But with VR, we stand in the middle looking out.

Homo Theoreticus
Western man has developed a predominantly "frontal" attitude to the environment. The fact that, largely because of alphabetiditeracy, we have let our eyes dominate our cognitive psychology makes us less sensitive to everything that is not included in the focused amphitheatre of our vision, external or internal. Within and without, we project images of ourselves as mirrors would, frontal beings staring back at our frontal beams. It has been that way since the Renaissance. Indeed our relationship to the world has been dominated by the theatrical tradition, and more specifically by theory. The word in Greek was theoria and it meant "something to behold," something to look at. Theatron too meant something to watch. Control over reality was effected by the detached point of view beholding pictures models and books. The point is that if you are looking into something from the outside, that thing is open to your judgement. You can take it or leave it and psychologically speaking, you remain free of feedback control. The most that can happen from such a sight is that you will be moved or change your opinion. You might even "change your mind" about a given issue, but the structure of your mind as a protected and private environment will not be affected. However, as Eric Gullichsen puts it, (in cyberspace) "you are an active participant in (a) world which surrounds you, not a passive observer in a world which confronts you."1 Indeed, not only does VR technology reflect and emulate the new environmental sensibility introduced by the information age, but it also destroys the conventionally western illusion that space is void.

A virtual "reality" is one that you can touch and feel, as well as see and hear with your real senses, not only your internal ones such as the "mind's eye" or the "mind's ear." By penetrating into the screen with the "Dataglove," the real hand turns into a technical surrogate of a kind of "mind's hand," making "real" that which was only visible. Before the invention of VIZ, nobody ever bothered to conceive of a "mind's hand." The concept did not present itself. There seemed to be no special need for "feeling" the objects we carried in our minds. In general, the tactile sensitivity of western cultures is abysmal. Within our rather intellectual and abstract tradition, we have tended to ignore the rich learning experience derived from touch. In fact we have been terrified of touch until the sixties when television induced a collective craving for the recovery of our bodies once lost to our literate heads. After a rash of "touch-ins" and "feel‑ins" spreading from California eastward, we began to settle down to a more comfortable relationship to our bodies, but we never gave touch its honoured position among our principal sensory modes. The appearance and development of VR is about to change that.

In spite of its acknowledged lower resolution and weaker sensitivity to grain and texture, the precision of touch in simulation is much greater than that of vision because it is 3‑D. Sharing a vision is nowhere nearly as reliable when you cannot touch the object of that vision. It is worth noting that is a woman, not a man, who is presently considered as the pioneer of electronic tactility. Margaret Minsky, daughter of the well‑known MIT computer expert, is developing a leading‑edge "virtual texture simulator" which, along with movement, weight and density simulation, will eventually increase the range and the depth of our tactile appreciation of objects both in and out of VR.


From Theory to Ingestion
Indeed, the story of computer simulation is one of gradual penetration into a tactile environment: from 2‑D to 3‑D to the rapid development of tactile and force‑feedback sensations, we are being, sucked into an electronic vortex, a richly textured matrix. The call of tactile technology reminds me of the myth of Ulysses' sirens, with its intensely erotic connotations. Many silly jokes are cracked about the pornographic market available to VR, thus reflecting our atavistic puritanism, but it will soon dawn on educators' minds as it has already occurred to some artists that touch may be our most important cognitive tool. Babies learn by touching, adults learn by "grasping" a situation. "Comprehending," too, is a tactile metaphor. We develop a kind of gut relationship to the things we know or need to know. What VR brings to this tendency is a means to project outward our nervous system so that the electronic extensions of our sensory mode, and especially this new electronic extension of touch, can conquer the field of investigation by swallowing it. The next metaphor of learning will be ingestion.


2. Simultaneous Shared Cognition

Among the significant differences between an internal fantasy driven by "mind‑senses" and an external one responding to our physiological senses is that the latter can be shared objectively, if we accept the notion that objectivity is the condition of an external substance shared by two or more people at the same time. According to Jaron Lanier, "the essence of virtual reality is that it's shared." He proposes that VR is "the first new level of objectively shared reality available to humanity since the physical world."2 On the other hand, cognition, otherwise known as understanding, is the total simultaneous seizure of contextual cues relevant to a given object of thought and the interplay between a self and the content of this thought and the project consequences of this thinking. Cognition is processing Virtual Realities within the single mind, but VR technology allows many minds to collectively process cognitive material without.

"We obtain raw, direct information in the process of interacting with the situations we encounter. Rarely intensive, direct experience has the advantage of coming through the totality of our internal processes – conscious, unconscious, visceral and mental – and is most completely tested and evaluated by our nature. Processed, digested, abstracted second‑hand knowledge is often more generalized and concentrated, but usually affects us only intellectually – lacking the balance and completeness of experienced simulations . . . Although we are existing more and more in the realms of abstract, generalized concepts and principles, our roots are in direct experience on many levels, as is most of our ability to consciously and unconsciously evaluate information."3 Scott Fisher could not have found a better way to highlight how VR could improve our cognitive abilities. And indeed, he goes on to suggest that giving people instant access to "greater than one viewpoint of a given scene [VR] allows them to synthesize a strong visual percept from many points of view; the availability of multiple points of view places an object in context and thereby animates its meaning."4

This, of course, was the point of Cubism, but never before have we been in a situation where several different points of view, issuing from different people, can simultaneously interact in cognition through a direct relationship to a common object of study or investigation. In the case when two people engage in creating a common VR, such as in Jaron Lanier's primitive but impressive RB2, these agents remain operative, but the effect is cognitive, reflecting the reflections of two cognitive agents. Add to such an ability the possibility of touching the object of perception and modifying it in selected ways ruled by selected routines, and you will eventually get the most powerful thinking machine ever devised by man: a think tank where the thought is the tank.

We can readily foresee situations where VR will consist of multiple sensors built on the model of sensory projections such as seeing, hearing and feeling and beyond. At this point the VR environment will have to develop a degree of retroactive intelligence which can take any number of directions and bring them to any degree of responsiveness and complexity. Even though presently there is no way to effect other than "parallel" or "serial cognition" between users working on the same process, with the development of sophisticated groupware, VR seems to hold in store the promise of "convergent cognition." In Lanier's RB2, we can foresee a kind of mutual mind reading the product of which would be an objective result that could be recorded and replayed for further use. We can already conceive of new directions of icon and rule‑operated processing which will involve more than one person at a time. This is what I foresee as "collective cognitive processing." But, to get there, the interface technologies must come ever closer to the body and to the thought source of the users.


The Perceptual Bond
Anybody can remember the first eerie impressions of intimacy created by headphones. We have become so accustomed to our Walkmen that we may cease to notice that the sound penetrates the whole body through a space which is felt internally between the ears. Imagine a similar sensation of immediacy coming from a visual stimulation. The machine that allows for this kind of experience is already on the market: Cyberspace Corporation produces a headband‑mounted eyepiece which flashes images directly onto the user's retina. The image appears to hover in the air, full‑sized and exclusively for the benefit of the user. New devices, from eye‑tracking and image‑contact to brain‑wave interfaces are moving in the general direction of immediate processing from thought to machines. Indeed, we can predict from the kind of work going into this line of technology, as well as from the kind of tools available to increase computing capacity and speed, that we will eventually have access to any mode of interface from the twinkling of an eye to just a short bout of focused attention.


The Gradual Removal of Interfaces
Gullichsen rightly observes that VR goes a very large step beyond conventional computers which are merely "interactive": "A cyberspace system is dynamic: the virtual world changes in realtime, both autonomously and fluidly in response to the actions of the patron. Action is visceral, and there need be no veneer of symbolic 'interface,' since the objects in this 3‑D world can be directly manipulated."5 In the same publication, albeit rather uncritically, Luis Racionero suggests that the way of the future is to connect electronic pathways directly to our neural networks by some bionic engineering already under development.6

We can envisage the future of problem-solving as a VR extension of the think tank or the pow‑wow. Working out a solution to a given problem will be given much amplitude by the possibility of simulating a complete environment by thought alone, but it will be even more relevant to the combined effects of several thinking subjects given a single object under consideration. Eventually we will be creating new objects instantly by thought alone, in a collaborative manner. They will be blueprints recorded for the purpose of subsequent hardware production. The industry is already experimenting with "walk‑through" models of architectural and urban plans. Albeit primitive because the technology does not yet allow for instant interactive responses to sensory projections, the kind of work done at Autodesk is the first base of such developments.


Shared VR Models to Overcome Language and Culture Biases
Howard Rheingold reports that Cecil Patterson, the information systems director for the Port Authority of Seattle is thinking of exploiting VR to improve communications "between engineers, facilities planners, and potential clients, when it comes to discussing the actual physical configuration of future port facilities."7 According to Patterson, most of the people involved in the planning stages of "multi‑hundred‑million‑dollar plans" are Japanese, Chinese, and other who do not always fully appreciate the nuances of the English language. VR "walking‑throughs" could help to resolve a number of ambiguities. It seems as if the Almighty might have had to find a new tack to stave off the construction of Babel if it had been planned on a VR processor . . .


Shared VR Models to Overcome Complexity
Many students of VR talk about its merits in allowing "architectural walkthroughs," an obvious industrial application, but Rheingold also suggests that 3‑D visualization may be the only way to overcome the complexity of certain scientific/technological fields such as molecular innovation or, on the macro scale, the telecommunications web that has grown around the planet into what Xerox PARC researcher Bernardo Huberman calls "a computational membrane."8


3. Implications of VR for Art

VR's potential for increased creativity calls for an integrated vision even at the lowest level of simulation. "The job of the space designer is to make [the] experience [of cyberspace] seem real. Thus, the job is as artistic as it is technical, for experience is something manufactured spontaneously in the mind and senses, not something that can be built, packaged and sold like a car or a refrigerator."9 Several aspects of VR concern the arts. One is that VR is defined by its sensory values more than by any other. VR brings to the fore the fundamental nature of art which is grounded in aesthetics, that is, the exploration and manipulation of the senses. Luis Racionero predicts that "New arts and new senses are needed – or what is the same thing – a different program in the brain allowing a greater broadening of the senses. The first of these will come with new technology: there will be new arts, as was cinema in this century, based on genetic engineering, holograms, laser, space travel, nuclear transmutation."10 From such considerations alone, one might think that VR is first and foremost the artist's special preserve, were it not for early business and entertainment applications already sought by amusement parks and admen. While business stands as the watchdog, not the mother of invention, vetting and supporting only that which it feels marketable, it may be to art and to artists that we owe the most significant developments of VR. Howard Rheingold calls the effect of VR an "intuition amplifier." Even as sober a commentator as the Wall Street Journal has been led to acknowledge the special ability of the artistic mentality: "Several VPL employees are enterprising artists with a knack for electronics."11 The WSJ names Ann Lasko, Young Harvil. and Jaron Lanier as prime examples, but it could just as well have included Scott Fisher, Eric Gullichsen, Myron Krueger, Graham Smith, William Gibson, and even Tom Zimmerman, who invented the Dataglove in his spare time only to simulate a virtual hand playing a virtual guitar.

These artists either combine an art education with "learn‑as‑you‑go" engineering and computing skills, or team up with professional computer engineers to turn their institutions into realities. A good example of the latter is Vincent John Vincent. Vincent was originally a dancer who, feeling the urge to use a computer as his partner, asked Frank MacDougall to write the program for this purpose. The result was the Mandala, one of the most elegantly simple and efficient five interactive installations ever devised in Canada. Although the Mandala is not technically a "virtual reality machine" because it does not allow the user to penetrate the screen, it does the next best thing, which is to allow one to send one's image into the screen to perform increasingly complex interactions with sound and design modules according to protocols inspired by hypermedia research.
Another important Canadian artist‑engineer, Graham Smith, is working with Jaron Lanier and the VPL engineers to add total surround real video feed to the computerized articulation of VR. Smith began as a photographer but soon added robotics to his still cameras to make total surround photographies. He moved on to video when he realized that he could adapt his intuitive robotics imagination to the full motion and instant replay capabilities of the video camera. Albeit at the prototypical stage, his "horizonscan" can already offer a full 360 degrees video rendition of his environment, provided one dons the videogoggles which open the door to VR.

One of the aspects of VR which is immediately apparent to artists, but not necessarily to the public is that, as Kevin Kelly puts it, "it has deep roots which penetrate our mind's view of the world."12 We can begin to expect feedback effects of VR into consciousness itself. Since the time Marshall McLuhan suggested that "the medium is the message," we are beginning to become aware of the fact that new media affect our cognitive processes in the long run by favoring certain routines over others. Consequently, the need to educate the public to such new possibilities is urgent. Here again, because of their constitutive missionary zeal, artists tend to take the lead.

Piero Gilardi is one of this generation's representatives of a typically Italian phenomenon, an unbroken chain of artists who go back many centuries. The succession of Gilardis since the 17th century has produced architects, painters, and sculptors. And, since the time of Andrea Gilardi, who invented and produced the first backlit glass painting in the early part of the 18th century, many among them have been inspired to use the technology of their time in original and truly inventive ways. Piero Gilardi's concept is a striking, although an expensive one. Realizing some of the implications of the role of technology in changing our world view, he has proposed to erect in Turin a huge baby doll, Ixiana 2000, as an itinerant exhibit of the world's most perfected mechanical and electronic technologies, perceived and shown as direct extensions and modifiers of our body and mind (see illustration). The baby doll is a girl and her head/mind will be a theatre for virtual realities. It is by this kind of setting that the general public may quickly graduate to the understanding of the impact of media on cognition and sensation.


Art Versus Technology
However, so much of VR technology is rapidly becoming marketable that there is already a growing distinction between artists‑engineers who lean more to the art than to the technical component of the association. The deeper artistic implications are the most interesting. Years before VR became the talk of the media, Toronto composer David Rokeby had perceived that the sensory essence of electronic and computer technology was touch. His prize‑winning invention "Very Nervous System" stands among the first and the most satisfying virtual reality machines ever. Like Jaron Lanier, Rokeby began as a musician frustrated by the limitations of traditional instruments. To expand the possibilities of musical creation and make it instantly available to untrained practitioners, he assembled "video cameras, image processors, computers, synthesizers and a sound system to create a space in which the movements of one's body create sound and / or music13 (see illustration of VNS). Unlimited virtual musical pathways can thus be explored and reproduced by the movements of a dancer. Although this kind of installation has been tried by many other artists across the world, none has succeeded to the extent that Rokeby's has in allowing the user to obtain precise and controlled effects without hampering the necessary ingredient of randomness to the experience. In spite of the fact that nothing is touched except the ambient air, Rokebys perceives his invention as an extension of touch with the musical and tonal feedback as an instant environment of texture. Says Rokeby: "Within the installation, sound has a sculptural presence, both as an extension of the body, and as a physical reality which one encounters with the body." He also describes the reactions of people while they are using the system: "People seem to endow a sound that has a lot of texture with an external material reality, [they] find themselves imagining the feel of it against their body, imagining the space filled with sound particles. Rounder, deeper sounds seem to manifest themselves more as a presence within or an extension of the body, something of the stomach organs and muscles rather than the skin."14

Back in the late sixties, with his "Cortizouk," French musician Pierre Henry had already connected a synthesizer to electrodes tapping his brainwave for direct input in musical forms. The same kind of idea was proposed for brainwave control by David Rosenbaum. There are unlimited possibilities for truly artistic innovations. Jaron Lanier is fond of the idea of a virtual mirror in which the user could see metamorphoses of featuring and form. Another fascinating use for art would bean interactive painting, not merely such as Jeffrey Shaw's walkin pieces, but art that would react reflexively to the various ways by which the viewer would approach it. A kind of electronic impressionism.

To take advantage of the collective processing possibilities of VR, we can imagine, for example, the creation of an environment which would reflect immediately our thermal and postural disposition and feed into the simulation effects that would combine inputs and outputs to generate cybernetic loops. The therapeutic value of a well-tuned cybernetic environment could be considerable. Lanier, who is aware of such possibilities claims that he is sometime concerned about the proximity of the "Virtual Reality" to mind‑expanding drugs and worries about comparisons made by people who, like Timothy Leary, have used psychedelic drugs. With VR, what we will shortly be led to expect is truly psychedelic in the sense that the structure of our mind can be changed. Depending upon the dosage of sensory references, for instance, we can have an acoustically or a tactile-dominant experience of the same thing. This can alter completely our processing routines and achieve totally unexpected results. Reporting on his first experience of VR at Autodesk, John Perry Barlow, a one‑time lyricist for the Grateful Dead, registers the mild confusion it created in his sense of identity: "How can you get where you want to be when you're coming from nowhere at all? And I don't seem to have a location exactly. In this pulsating new landscape, I've been reduced to a point of view. The whole subject of "me yawns into a chasm of interesting questions."15


Pascal's Two Infinites
Such questions were once raised by French philosopher Blaise Pascal when, reflecting upon the powers of imagination, recently generalized by a rapid growth of literacy, he proposed that Man could engulf the universe by thought alone. Pascal's sensibility was surprisingly close to our time in that he was a poet of scale, drawing his most striking metaphors both from the infinitely small realm of molecular structures and the infinitely large domain of astrophysics. The most poetically minded of the cyberspace explorers, Jaron Lanier, reminds me of Pascal when he suggests that "the universe is your body and physics is your language."

A few months ago, I was asked to contribute a paper to a journal asking whether virtual technology would herald "a new Renaissance." My answer was no. Why would we start all over again the same old story when we had such interesting new ones to tell? Collective cognitive processing is bound to change the basis of Renaissance psychology. What we need is a sense of expanding, flexible self, inclusive rather than exclusive, environmental rather than frontal, collaborative rather than confrontational, conscious of its bionic extensions rather than resolutely ignoring any suggestion that we are not made of flesh alone. Still, as I was researching the literature to write this paper, I came across a better answer in one among Jaron Lanier's many fanciful projects: "a giant birth canal through which you travel and are born, only to find that you are the virtual woman giving birth to yourself and are reborn again and again."16


Notes
1. "In the Realm of the Sensors," in: Catalogue of Art Futura 1990, Barcelona, Summer 1990, p. 82.
2. Quoted by Steve Ditlea, "Inside Artificial Reality," PC Computing, November 1989, p. 97.
3. Perkins, quoted by Scott Fisher, "Virtual Environments, Personal Simulations and Telepresence," in: Catalogue of Art Futura 1990, Barcelona, Summer 1990, p. 50. Reprinted in this volume.
4. Ibid., p. 51.
5. Ibid.
6. "It is not impossible to cause a short circuit in the process [of electro‑chemical impulses] by substituting the stimulus of outer reality by a connection between the cerebral network and a computer program which sends stimuli similar to sensory perceptions." "Virtual Realities," in: Catalogue of Art Futura 1990, Barcelona, Summer 1990, p. 21.
7. "Travels in Virtual Reality," Whole Earth Review, Summer 1990, p. 85.
8. Ibid., p. 83.
9. Eric Gullichsen, "In the Realm of the Sensors," in: Catalogue of Art Futura 1990, Barcelona, Summer 1990, p. 83.
10. Ibid., p. 21.
11. G. Pascal Zachary, "Artificial Reality: Computer Sirnulations One Day May Provide Surreal Experiences," The Wall Street Journal, January 23, 1990, pp. A1 and A9.
12. Whole Earth Review, Summer 1990, p. 80.
13. David Rokeby, "The Harmonics of Interaction," Musicworks, 46, Spring 1990, pp. 24–26.
14. Ibid., p. 26.
15. "Being in Nothingness: Virtual Reality and the Pioneer of Cyberspace," Microtimes, January 22, 1990, p. 96.
16. Reported by Steve Ditlea, "Inside Artificial Reality," PC Computing, November 1989, p. 97.