Translating Gesture

ROLE

Interaction Designer

CLIENT

MFA Thesis — Mason Gross School of the Arts

TIMELINE

2025–2026

OVERVIEW

Gesture is often assumed to be universal, yet the same movement can carry different meanings across cultures, communities, and individual experience.

Translating Gesture is an interactive installation that uses real-time hand tracking to detect participants’ gestures and project a rotating series of possible interpretations onto a fifteen-foot gallery wall.

The system does not decide what a gesture means. It makes ambiguity visible, inviting people to notice how bodies, culture, and technology shape communication.

RESULTS

21

Hand Landmarks

15 ft

Projection Wall

PROBLEM

How might an interactive system reveal that gestures are interpreted rather than universal?

People often read gestures as fixed signs, but meaning changes with culture, social context, age, and personal experience. The challenge was to reveal that multiplicity without replacing it with another authoritative answer.

RESEARCH

I surveyed participants from varied cultural and generational backgrounds, asking one open-ended question: “What does this hand gesture mean to you?” The conflicting answers became the installation’s language library.

BRAND & STYLE

A spare gallery, large projected type, a thermographic display, and an open white floor ring create a legible environment that invites participation without instructions.

LOW FIDELITY

Early spatial sketches mapped the relationship between body, camera, floor marker, projection, and gesture archive before technical development began.

HIGH FIDELITY

The final installation pairs immediate bodily feedback with shifting text, allowing participants to understand cause and effect while leaving interpretation deliberately open.

PROTOTYPE

MediaPipe tracks 21 hand landmarks in real time. TouchDesigner measures fingertip distance, joint angles, and finger spread, then triggers rotating phrases from the research archive.

IMPLEMENTATION

From skeletal data to shifting language.

I built the recognition pipeline, tuned gesture conditions, designed the spatial cues, and tested the full system in the gallery so the response felt immediate and understandable.

LAUNCH

Exhibited at Mason Gross School of the Arts in Spring 2026, the work required no written instructions. Visitors entered the ring, experimented, and turned gesture recognition into physical play.

WHAT I LEARNED

Clear interaction rules created confidence; interpretive openness created discovery. The strongest experience balanced both, using technology to intensify bodily awareness rather than abstract it.

Next project