The Responsive Window

2001

Gerfried Stocker (AT)
Gerold Hofstadler (AT)
Benjamin Fry (US)
Joseph Paradiso (US)
Che King Leo (US)
Kaijen Hsiao (US)
Nisha Checka (US)
Josh Lifton (US)

"The Responsive Window" turns taps on glass into an information interface. Sensors capture all the characteristics of the window knock—location, intensity of hit, and type of hit (e.g., bare hand or hard object). Interactive graphics dynamically respond to all of these parameters and are projected onto a holographic screen placed behind the glass. This glass wall is again a boundary, this time between the real and the synthetic, across which we are invited to knock.

Glass is nowadays a ubiquitous medium in modern construction. It appears in many forms, such as room dividers, windows, or in some instances exterior sidings for entire buildings. Its main function is to prevent matter from passing through whilst allowing visible photons to flow freely across.

When we look at a glass partition, we can, thus, freely gaze into the world on the other side (and potentially vice-versa), but our physical influence on that world is limited to actions at the boundary, such as knocking and tapping. As a result, people have honed their knocks and taps into practiced techniques, being able to express various levels of affect, intention, and personality depending on how hard or in what manner they strike the glass.

With the exception of relatively small touch-screen monitors, glass panes are still mainly uninstrumented and non-interactive. The basic technology to turn taps on glass into an information interface is, however, very inexpensive and already commonly employed. The glass panes enclosing urban shopfronts frequently feature an array of attached acoustic sensors. The only event these detect, however, is a catastrophic one—an impact so hard that the glass breaks and a theft or vandalism can be assumed.

These same simple sensors, with some signal processing, can be used to detect, track, and characterize softer knocks and taps, allowing very large glass surfaces to become truly interactive for much less aggressive (and more legal) degrees of expression. We can accomplish this by measuring the time it takes the bending-wave impulse created by the impact of the hand to reach piezoelectric contact microphones laminated to the glass plate at four corner locations.

The difference in arrival times of the acoustic energy at each pickup (inferred through a simple algorithm running in a digital signal processor) determines the location of the impact. The amplitude of these signals similarly reflects how hard the glass was knocked, and the frequency of the impact waveform allows one to infer the kind of object that made contact (e.g., a soft knuckle or metal ring).

The idea first struck me when contemplating interfaces for an interactive digital fish tank. Here the glass around the tank is clearly the boundary into a very different world, and few can resist the temptation to jar its denizens by tapping on the surface.

Shortly afterward, we demonstrated this acoustic time-of-flight tracking scheme in collaboration with Hiroshi Ishii's Tangible Media Group for the "PingPongPlus" project—a sensate ping pong table that used this technique to quickly locate the impacts of the ping pong ball and generate corresponding real-time graphics projected onto the table's surface.

We next turned our attention to adopting this method for locating knocks and taps on large glass surfaces, a much harder problem due to the non-repeatable characteristics of the knocks together with the faster propagation velocity and dispersive nature of bending waves in the glass.

This "Responsive Window" project represents the current culmination of our efforts. Our sensors capture all the characteristics of the window knock—location, intensity of strike, and type of strike (e.g., bare hand or hard object).