Robots Get Language Lessons to Promote Speech

Robert S. Boyd
Contra Costa Times

June 18, 2002

Many parents have had the delightful, sometimes frustrating experience
of teaching a toddler to talk.

Now, computer scientists are giving language lessons to mechanical robots, enabling them to speak and to respond appropriately to what they hear.

Such language skills will be needed if robots are ever to reach their potential as humanoid helpers—cleaning house, serving cocktails, stocking grocery shelves, guiding office visitors, and other routine chores.

"It is now becoming possible to have open-ended dialogues with physically embodied robots," said Luc Steels, an artificial intelligence expert at the Sony Computer Science Laboratory in Paris.

In the past 10 years, so-called "intelligent robots" have come a long way. They walk on their own legs, climb stairs, navigate through unknown environments, even play fast-paced soccer matches.

But they are still introverts, with limited ability to communicate with humans or their fellow robots. Teaching them is extremely difficult and progress has been slow.

"It's really very hard," said Tim Oates, a robotics expert at the University of Maryland–Baltimore. "Imagine you are in a foreign country and don't know the language. You can't even tell where a word begins and ends, much less the meaning of the word."

Despite the difficulties, Oates said his current goal is to make a robot follow him around the university campus, gathering and processing information.

"I hope a year from now I can tell it: 'Look out for that trash can' or 'Let's go through that door.' I don't want to be holding the robot's hand," he said.

In five years, Oates added, "I want to be able to say things like, 'Could you go into the room we were just in and bring me the red ball?' If the robot can do that, I'll declare it a success. It's a very ambitious but not unreasonable goal."

Lessons From a Child

To teach robots to use language, researchers are experimenting with various techniques. In some experiments, instead of copying their human instructors, robots even invent their own words and grammatical structures.

One method resembles the way a parent or older sibling teaches a child to associate a spoken word with an object. A human instructor shows a toy or stuffed animal to a robot, speaks its name, and makes sure the robot remembers it.

In an experiment by Steels that resembles the way a parent teaches a child, the "pupil" was an enhanced version of AIBO, a popular robotic toy dog marketed by Sony, the Japanese electronics company, which is supporting his work.

The little robot was equipped with eyes (a video camera), ears (a microphone), a voice box, and a computer program able to recognize and pronounce human words. Steels pre-programmed AIBO to recognize a few spoken words such as "look" ... "listen" ... "what is it?" ... "good" ... "yes" ... "no."

In a report published on the Internet, Steels described a typical learning session: On a table in front of AIBO are three objects—a red ball, a yellow puppet called Smiley, and a toy dog called Poo-chi. AIBO already knows the names of Smiley and Poo-chi, but not the word "ball."

The human teacher points to the ball and speaks:

Human: "Look. Ball."

AIBO: "Ball?"

Human: "Yes."

Human: "What is it?"
AIBO: "Smiley."

Human: "No, listen. Ball."

AIBO: "Ball?"
Human: "Yes."

Human: "Is it Smiley?"

AIBO: "No, ball."

Human: "Good."

To accomplish this feat, the robot had to connect two very different electronic patterns—one from the sound waves coming into his "ears" and the other from the light waves coming into its "eyes"—and form the concept, "ball."

"We have carried out a realistic robotic experiment in the sense that the robot is not only ignorant about the words in the language, but also about the concepts underlying these words," Steels wrote.

Self-Learning Technique

In another set of experiments conducted by Oates, there is no direct instruction by a human teacher. Instead, the robot must learn the language on its own by observing what people do and say.

For example, Oates arranged a set of colored blocks in various patterns, while volunteers described what he was doing in simple sentences. The robot watched and listened.

After hearing 50 statements describing the same scene over and over again, the robot was able to associate certain sound patterns with specific objects, such as "blue block," store them in its computer memory, and repeat them when asked.

"The system successfully discovered words and their denotations in three different languages—English, German and Mandarin Chinese," Oates said.

Another approach is to let a pair of robots teach each other language, with virtually no human intervention. Steels called this "unsupervised learning."

In one test, two robots, A and B, face each other across a table covered with colored plastic chips. Robot A points to a chip and pronounces its name, for example, "red circle." If it doesn't have a name for that object already in its memory, it makes one up from a catalog of arbitrary syllables, such as "wapaku."

Robot B listens to the word. If it recognizes it, it points to it with its jointed arm. Robot A either confirms that the answer is correct, or tells Robot B it made a mistake. Robot B then points to other objects until it gets the right one.

Bit by bit, the two robots build up a common vocabulary of about 100 basic words, expressing concepts like up, down, left, right, red, green, large and small.

"We have demonstrated that a robot can learn the meanings of words," summed up Paul Cohen, a computer scientist and colleague of Oates at the University of Massachusetts in Amherst.

Copyright 2002, Contra Costa Times

Join the National Geographic Society

Join the world's largest nonprofit scientific and educational organization, and help further our mission to increase and diffuse knowledge of the world and all that is in it. Membership dues are used to fund exploration and educational projects and members also receive 12 annual issues of the Society's official journal, National Geographic. Click here for details of our latest subscription offer: Go>>