Toward Text-to-Picture Synthesis

It is estimated that more that 2 million people in the United States have significant communication impairments that result in them relying on methods other than natural speech alone for communication [2]. One type of commonly used augmentative and alternative communication (AAC) system is pictorial communication software such as SymWriter [8], which uses a lookup table to transliterate each word (or common phrase) in a sentence into an icon. This is an example of converting information between modalities. However, the resulting sequence of icons can be difficult to understand. We have been developing general-purpose Text-to-Picture (TTP) synthesis algorithms [10, 5] to improve understandability using machine learning techniques. Our goal is to help users with special needs, such as the elderly or those with disabilities, to rapidly browse documents through pictorial summaries (e.g., Figure 5). Our TTP system targets general English. This differs from other pictorial conversion systems that require hand-crafted narrative descriptions of a scene [1, 9], 3D models [3], or special domains [6]. Instead, we use a concatenative or “collage” approach. In this talk, we discuss how machine learning enables the key components of our TTP system.