‘Draw me a picture,’ say scientists; computer may respond

Jillian Aurisano, Ph.D. student in computer science, works with the Articulate prototype in EVL cyber-commons classroom. Seated: EVL students Yiwen Sun (left, original developer of Articulate) and Dennis Chau (right). Photo: Lance Long/EVL. Click on image to download.

Like the rest of us, scientists wish they could just ask a computer a question and have it respond with an answer presented in an easy-to-understand picture.  Today’s visualization tools can translate huge raw data sets into graphs and maps — but most scientists lack the time and training to use the tools effectively.

The National Science Foundation has awarded a $300,000 grant to the University of Illinois at Chicago and the University of Hawaii to take the first steps towards developing a computer that can take data and produce meaningful visualizations based on natural language requests, accompanied by common gestures like pointing.

With nearly a third of the human brain devoted to processing visual stimuli, vision has always been our dominant way of acquiring information. Visualization is the most effective means of converting raw data into insight that can support scientific discovery, says Andrew Johnson, director of research at UIC’s Electronic Visualization Laboratory.

“Today, with big data, you really need to be using visualizations to help you figure out what it is you’re looking at,” said Johnson, who is a co-principal investigator on the NSF grant. “Visualization should be interactive; a dynamic process. We want scientists to be able get ideas out there quickly.”

“The object is to make it more like a conversation with a person in the room,” said Barbara Di Eugenio, UIC professor of computer science and principal investigator on the grant. “’Can we recolor that? What if we moved it this way, or inverted that axis?’”

“We also don’t want to require explicit questions,” she said. “The goal is for the computer to be able to interpret even indirect questions, like ‘ It would be better to show salinity only at 10 meters’, or even statements that hint at something, and put together the visualization.”

From a simple, early effort, the researchers are now moving to the next level with visualizations that are much more complex, said Johnson.

Johnson says scientific papers often include graphs that are added after the paper is written and contribute little to the science. They are mostly created using everyday software.

“Today, [Microsoft] Excel is the most common graphing tool for scientists,” Johnson said.  “It’s easy to use, but it’s hard to use to do science. We’re dealing with an explosion of the data — the amount, and complexity, and rate is exploding so fast. Excel just can’t keep up.”

But, Johnson said, “Imagine if you had the computer behind you, helping you to see your data — it could really push scientific discovery.”

Leland Wilkinson, UIC adjunct professor of computer science, is co-principal investigator on the grant. The project is a collaboration with Jason Leigh at the University of Hawaii.