Robotic Roommates Making Pancakes

In this video we show a recent public experiment that shows two robots making pancakes using web instructions. In the experiment, the robots retrieve instructions for making pancakes from the World Wide Web and generate robot action plans from the instructions. This task is jointly performed by two autonomous robots: The first robot opens and closes cupboards and drawers, takes a pancake mix from the refrigerator, and hands it to the robot B. The second robot cooks and flips the pancakes, and then delivers them back to the first robot. AI-related in the experiment is using of the encyclopedic knowledge base for the translation of instructions into robot plans which is performed by the following sequence of steps. First, the sentences are parsed using a common natural-language parser to generate a syntax tree of the instructions. The branches of the tree are then recursively combined into more complex descriptions to create an internal representation of the instructions describing the actions, the objects involved, locations, time constraints, the amount of ingredients to be used etc. The words in the original text are resolved to concepts in the robot's knowledge base by first looking up their meanings in the WordNet lexical database, and by then exploiting mappings between WordNet and the Cyc ontology. Our system employs a simple method based on the phrase context and on information about object-action pairs obtained from Cyc to disambiguate between possible word meanings.