Role
Interaction Designer
Client
MFA Thesis — Mason Gross School of the Arts, Rutgers University
Translating Gesture
Translating Gesture
OVERVIEW.
Gesture is often assumed to be universal. But the same movement can mean entirely different things depending on who is reading it, and from where.
Gesture is often assumed to be universal. But the same movement can mean entirely different things depending on who is reading it, and from where.
Translating Gesture is an interactive installation that uses real-time hand tracking to detect participants' gestures and project a rotating series of textual interpretations onto a fifteen-foot gallery wall. The system does not tell you what your gesture means. It shows you what it could mean.
CORE IDEA
We treat gesture as legible, as if a raised hand or an extended finger carries a single, shared meaning. The research behind this project told a different story. A gesture as apparently simple as the index-and-pinky extension generated at least three distinct readings depending entirely on the interpreter's background: rock culture, ASL for "I love you," or a symbol of evil. Same gesture, entirely different meaning.
The installation was built on this finding: gestures are not fixed signs. They are interpreted signs, and interpretation is always shaped by the body, the culture, and the history of the person doing the reading.
We treat gesture as legible, as if a raised hand or an extended finger carries a single, shared meaning. The research behind this project told a different story. A gesture as apparently simple as the index-and-pinky extension generated at least three distinct readings depending entirely on the interpreter's background: rock culture, ASL for "I love you," or a symbol of evil. Same gesture, entirely different meaning.
The installation was built on this finding: gestures are not fixed signs. They are interpreted signs, and interpretation is always shaped by the body, the culture, and the history of the person doing the reading.
RESEARCH.
I conducted surveys asking participants from diverse cultural and generational backgrounds a single open-ended question: What does this hand gesture mean to you?
The theoretical grounding comes from John Durham Peters' Speaking into the Air (1999), which argues that misunderstanding is not a failure of communication but a structural condition of it. The installation takes this as a generative constraint rather than a problem to solve.
"Miscommunication is the scandal that motivates the very concept of communication in the first place." — John Durham Peters
I conducted surveys asking participants from diverse cultural and generational backgrounds a single open-ended question: What does this hand gesture mean to you?
The theoretical grounding comes from John Durham Peters' Speaking into the Air (1999), which argues that misunderstanding is not a failure of communication but a structural condition of it. The installation takes this as a generative constraint rather than a problem to solve.
"Miscommunication is the scandal that motivates the very concept of communication in the first place." — John Durham Peters
THE SPACE.
The space is spare. A camera is mounted above the interaction zone. A white vinyl ring on the floor marks where participants stand; open on one side, inviting entry rather than enforcing containment. Across from the projection wall, a thermographic display renders the active participant's hand in heat-mapped color. To the right, a grid monitor loops pre-recorded gestures, functioning as both reference and archive.
No instructions are posted. Participants enter the ring, raise their hands, and discover the system through use.
The space is spare. A camera is mounted above the interaction zone. A white vinyl ring on the floor marks where participants stand; open on one side, inviting entry rather than enforcing containment. Across from the projection wall, a thermographic display renders the active participant's hand in heat-mapped color. To the right, a grid monitor loops pre-recorded gestures, functioning as both reference and archive.
No instructions are posted. Participants enter the ring, raise their hands, and discover the system through use.
THE TECHNICAL SYSTEM.
Built in TouchDesigner using Google's MediaPipe hand tracking library. The system identifies 21 skeletal landmarks across the hand in real time, distance from fingertip to palm, joint angles, finger spread. When conditions are met, a gesture is classified and the text output is triggered.
The phrases cycle and rotate, drawn from survey responses and cross-cultural gesture research. No single interpretation is presented as final. The system is designed for multiplicity, not resolution.
Built in TouchDesigner using Google's MediaPipe hand tracking library. The system identifies 21 skeletal landmarks across the hand in real time, distance from fingertip to palm, joint angles, finger spread. When conditions are met, a gesture is classified and the text output is triggered.
The phrases cycle and rotate, drawn from survey responses and cross-cultural gesture research. No single interpretation is presented as final. The system is designed for multiplicity, not resolution.
AT THE EXHIBITION.
Participants entered without prompting, began experimenting, and quickly moved from gesture recognition to something closer to performance — exaggerating their hands, holding configurations in place, moving rhythmically. The space became a site of physical play.
They became newly conscious of their own hands. What a hand does in ordinary life goes unnoticed. In the installation, that habitual movement became something to examine.
Participants entered without prompting, began experimenting, and quickly moved from gesture recognition to something closer to performance — exaggerating their hands, holding configurations in place, moving rhythmically. The space became a site of physical play.
They became newly conscious of their own hands. What a hand does in ordinary life goes unnoticed. In the installation, that habitual movement became something to examine.
