MSigns: 3D Virtual Sign-Language Translation

By Bill Clayton

A YOUNG WOMAN STANDS IN a market looking for the price on a frozen dinner. A stock boy notices the blank look on her face. "Six for 10 dollars," he says. But she still doesn't understand - the woman is deaf and, without a sign-language interpreter, is finding it very tough to get through the day.

Aware of this sort of problem, Jason Gilbert and Judy Yu, graduate students in Michigan Engineering's Department of Atmospheric, Oceanic and Space Sciences (AOSS), asked themselves: What can we do to make life easier for people who rely on sign language to communicate?

Their answer led to the MSigns program, which aims to develop technology that would "listen" to speech and translate it into sign language that a person could see - in 3D, in real time - on a handheld computer.

Motion-capture equipment in the University of Michigan's 3D Lab records the movement
of sensors on an outfit worn by Jason Gilbert. The data is mapped onto the
MSigns skeleton (left), which is a configuration of "virtual bones" that correspond
to Gilbert's arms, hands, neck, head and torso.

AOSS and Aerospace Associate Professor Thomas Zurbuchen, the project advisor, said that Gilbert and Yu make up a highly complementary team. "Jason started learning sign language in college so he could work with people who were deaf or extremely hard of hearing. He worked as a sign interpreter for several years. Judy's an extraordinary programmer who's creating software that's crucial to the success of the program."

The first step involved the translation of voice to text. The team decided to work initially with American English because the software to translate voice to American English text was available off-the-shelf. And American English grammar is less complex than American Sign Language grammar, which includes facial expressions and a different sentence structure.

The next step was to render the text, which involved a critical choice: Would the output be complete words or smaller building blocks that other software would assemble into signs? Whereas the process using building blocks would be more flexible, it would also be far too complex for these initial stages of the program. Gilbert and Yu decided to go with a method in which motion-capture equipment records signs that translate into individual words, which they can then piece together to form sentences.

Gilbert and Yu developed a special pair of gloves with markers attached strategically on the outside at the joints. They also created a shirt and cap with markers at strategic locations.

Gilbert wears the outfit in the University of Michigan's 3D Lab. "As I sign, the motion-capture equipment records the movement of the markers. We map the data onto the MSigns skeleton, which is a configuration of 'virtual bones' in the arms, hands, neck, head and torso." Yu said that they use MotionBuilder software to process the motion-capture data. "MotionBuilder is available off-the-shelf, too. I'm using web and database technologies to write a program that converts the output from MotionBuilder into data that we use to animate the skeleton."

The movements are smooth. But the transitions between movements are a bit trickier. By using a standard pose at the beginning of each movement, Yu hopes to achieve the effect of fluidity in a 3D virtual environment.

Judy Yu is using web and database technologies to write a
program that converts raw output into data which animates a
virtual skeleton (above). Yu hopes to create fluid movements in
a 3D virtual environment.

"We can flesh out the skeleton and actually build a virtual interpreter," Yu said. "And everything plays on a web interface. In other words, there's no need to download special software to see what's on the handheld screen. All people need is a web browser."

The team's initial goal is to develop a system that can translate about 100 words into sign language. Yu noted that for words not in their current database they would use fingerspelling to fill in the gaps.

The MSigns program is proving to be a series of engineering challenges in the lab and on the computer screen. But the team knows that, ultimately, the program is all about freedom for those who are deaf and extremely hard of hearing. -E

Support for this program comes from the University of Michigan Digital Media Commons GROCS Grant Program.