Imagine this: A person walks into a room and knocks a ball off a table. 

Did you imagine the gender of the person? The color of the ball? The position of the person relative to the ball?

Yes and no, says cognitive scientist Tomer Ullman, the Morris Kahn Associate Professor of Psychology, who with Halely Balaban recently published a paper titled “The Capacity Limits of Moving Objects in the Imagination.” If you’re like most people, you probably thought about some of these things, but not others. People build mental imagery hierarchically, starting with the ideas of “person,” “room,” “ball,” and “table,” then placing them in relation to one another in space, and only later filling in details like color.

“Our imaginations are actually patchwork and fuzzy and not filled in,” he said. His theory: Your mind’s eye might be lazier than you think. But that’s not necessarily a bad thing. “You leave things out until you need them.” 

For the latest installment of “One Word Answer,” we asked Ullman to elaborate further on the current scientific thinking behind “imagination.”

Some people very much associate imagination with the visual or the sensory: I’m creating a scene in my head that isn’t real right now. 

But there’s a different view of the imagination that’s more conceptual. When I’m imagining, say, three leprechauns jumping on a trampoline, I might not create the image in my head, but I understand the sentence. I’m creating a thought that I know is not true, but I’m considering it anyway. In that sense, imagination is very related to ideas of pretense and play. 

The argument about the role of imagery in imagination is a very old one. As far back as Plato — though he wasn’t the first to make this argument — some people have argued that imagination is like stamping a piece of clay. We perceive something and stamp the image into our minds, and we can then call up that image later, basically just as we originally saw it. While that’s an outdated way of thinking about things, it’s surprising how much parts of it have held up.

I take inspiration from a notion in computer science called lazy evaluation — lazy, not in a negative sense, but more in terms of economy of resources. I can conceptualize the calculation 17 times 63 in my head, and if you need me to figure it out, I can, but as long as you don’t need me to evaluate it, I won’t. It’s very context-specific. The same is true for our imaginations. For example, I ask you to imagine a strawberry. What color was it? You’ll probably tell me it was red. But did you really see the red strawberry? Maybe not immediately — you hadn’t gotten to that part of the rendering yet. You just know that strawberries are red. 

It helps to think about people who create video games or animated films. They create in a very hierarchical way. Say the scene is going to be a hedgehog rolling down a hill. They don’t start with coloring in the individual pixels of the hedgehog; they start with the spatial placements and the movement of the objects. Then at the end they can render it and fill in the pixels. And they can render it multiple times, changing the colors and so forth as desired, all built off the same base model. 

The debate around what happens in your head when you imagine stuff is very hot right now when it comes to aphantasia, a neurological difference where people don’t seem to experience visual images. It took a while for researchers to agree that this was a real thing, but it is.

You might imagine that aphantasia presents a challenge for some cognitive activities, but it turns out that people with aphantasia are perfectly able to complete any number of cognitive tasks that seem very visual, like rotating an object 180 degrees in their mind or counting the number of windows in their house. 

So how can that be? Halely Balaban and I suggest that aphantasia is a broken rendering operation — the underlying scene is there, but the last-mile rendering isn’t happening. 

Let’s play out one of those examples. If I ask you to count the windows in your house, you’ll likely see, in your mind’s eye, a kind of video-game visualization of your home, and you’ll pan the camera around and count. Many people report some version of that. But when you ask people with aphantasia that question, they pause for a bit, and you ask, “What are you doing?”

“I’m counting the windows.”

“Do you see anything?”

“No, just give me a second.”

The task of counting the windows doesn’t require visualizing. It’s actually us normies without aphantasia who are fooling ourselves. We see these images and get very taken in by them. We think they play a causal role in our thinking, but in fact, it’s the nonvisual framework underneath the image that does the real work.