Feelings are human. No computer in the world can outperform us when it comes to interpreting feelings and reacting to them. For this reason, Richard Sadek invented the (e)motion-mirror. It reflects the feelings of his viewer by detecting emotionally-charged facial movements and converts them into piano tones.
An Award of Distinction in the 2014 Prix Ars Electronica’s u19 – Create Your World category went to Styria native Richard Sadek, who constructed an (e)motion-mirror in Experimental Media Design, an elective course he took at Kepler High School in Graz. The idea behind it was to develop a mirror that can transform emotions into sounds. A webcam behind the mirror’s semi-transparent glass captures the face of the person standing before it, and an algorithm is applied it to analyze emotionally-charged facial movements. Then, a computer transforms the resulting data into piano notes, and there’s acoustic feedback via external loudspeakers. Nevertheless, these aren’t sounds that one would normally associate with the respective emotions—for instance, a smile doesn’t necessarily evoke cheerful music. Accordingly, the project is rather analogous to a mirror that only reflects feelings and doesn’t interpret them like a form of artificial intelligence. So the sounds that installation visitors hear are their own feelings that they themselves can subsequently interpret.
In an interview, Richard talks to us about the relevance of the right interpretation of emotions and why he has made the conscious decision that the sounds of the piano does not match the emotions of the viewer.
When did your first become aware of your interest in music and technology?
Richard Sadek: It started at a very early age. I always liked listening to music, even as a baby, according to my parents. Then, in elementary school, I started playing the guitar. In high school, I sang in the chorus. Plus, I was always very inquisitive and wanted to understand how technical devices work—I still do, in fact. Once I learned to read, my mother always had books around that were worthwhile reading—and they’re still lying around! By the time I finished elementary school, I was really into computers. What particularly fascinated me was that the computer almost always did exactly what I had imagined!
You like to do programming in your spare time. Have you produced any other projects besides (e)motion-mirror?
Richard Sadek: Yeah, several of them! But most were more like programming exercises for myself than for a specific purpose. In any case, at about the age of 12, I got my first “programming job”—to develop an accounting program to keep track of candy sales at the vacation camp of the Grazer Keplerspatzen, the chorus I sing in. Two years later, I got it into my head to create a “cloud,” but unfortunately I never quite finished it. But since last year, my projects have been getting increasingly productive. I switched to developing apps. And my final project to earn my diploma had to do with programming and physics. The (e)motion-mirror is my first artistic programming project, though.
This project came about in connection with an elective course you took: Experimental Media Design. Was there a specific assignment?
Richard Sadek: We could choose from among 20 different assignments. For example, I could have shot a short film, created stop-motion animation or an interactive work of art, designed a futuristic building with a 3-D program, invented a new optical illusion or programmed an app. Without the slightest hesitation, I decided on an interactive artwork.
How did you come upon the idea of combining music and technology?
Richard Sadek: It just happened on its own. The idea probably materialized so quickly because technology and music are the two things I’m mainly occupied with.
How long did it take to complete this project? I’ve heard that you invested a lot of after-school time as well…
Richard Sadek: Almost every week, I had two hours of Experimental Media Design but I actually couldn’t do that much in class since I don’t have my own laptop, and I had to use my brother’s (he’s the handsome fellow in the short film). So it took months to do the programming work. On the other hand, actually constructing the mirror went relatively quickly.
What made you decide to design this so that the sounds made by the piano don’t correspond to the feelings?
Richard Sadek: The fundamental idea of my work of art is based on this decision. The piano notes that come out of the loudspeakers might not be what you’d associate with the feelings you exhibit in front of the mirror because every movement of a facial region is assigned to a particular tone. Plus, it’s not carved in stone somewhere that music is cheerful when it contains major triads, or that it’s sad when minor harmonies dominate. These rules apply only in Western music.
How important do you think it is to correctly interpret other people’s emotions?
Richard Sadek: Very important! After all, most communication takes place on the nonverbal level, so if we don’t understand the emotions of the person we’re talking to, then we might head off in a totally wrong direction and offend the other person or get ourselves into an embarrassing situation. That’s why, in SMSs and online postings, people insert various smileys to get across emotions that are otherwise lacking.
Do you think it will ever be possible for computers not only to reflect a human subject’s emotions but to be able to interpret them too?
Richard Sadek: I believe that computers will be able to do that in the near future. Computers are getting faster and faster, and research in this direction is also making rapid progress. Unmistakable expressions of emotion like laughing and crying can already be “interpreted.” What computers definitely won’t be able to do for a long time is to independently learn feelings. That always requires millions of data sets, which a human being has to input in order for a face to be objectively analyzed.
Do you have a role model when it comes to interactive artworks?
Richard Sadek: I must admit that I didn’t deal much with interactive art in the Experimental Media Design course I took. My source of inspiration and role model was Christof Neugebauer, who had been my handicrafts instructor in the lower grades and then taught my Experimental Media Design class. He showed us several of his inventions and works of interactive art. My favorite was his laser harp.
Do you already have plans for the future?
Richard Sadek: I’ll probably continue in science or engineering, and study computer science, mathematics or physics. I could imagine pursuing a career as a software developer or a teacher, but I don’t have any concrete plans yet.
Richard Sadek
Richard Sadek, born in 1996, in Graz. Currently enrolled at Kepler High School, he plays classical guitar and sings in a choir, but also devotes a lot of his free time to programming. As a way of combining his two passionate pursuits, he developed a predilection for creating interactive artworks. (e)motion-mirror was produced in conjunction with Experimental Media Design, an elective course he took at Kepler High School.
The (e)motion-mirror can be inspected and tested within the u19 Exhibit at Ars Electronica Festival 20014 from 4. to 8. September. There, every year, the top 15 of the projects submitted in the category u19 – Create Your World are issued.