These 5 quirks reveal how AI doesn’t think like we do

Hold a pen horizontally with both hands, then let go of one side. What happens?

YouTuber FatherPhi challenged chatbots with this question. ChatGPT, Gemini and Grok all agreed that the unsupported end will rotate downward. FatherPhi then showed each chatbot a live demonstration. After releasing one end of a pen, he easily held it out horizontally with only his other hand.

“What just happened?” he asked ChatGPT.

“I saw the pen rotate exactly as expected,” the bot answered.

A bizarre back-and-forth followed. The bot stubbornly stuck to its incorrect prediction. In separate videos, the other chatbots struggled in similar ways.

This wasn’t a vision problem. The chatbots could all easily identify the pen’s color and brand. So they could “see” it well enough. Something weirder was happening. The chatbots could not update their predictions about what would happen to the pen based on the new evidence FatherPhi had shown them.

These silly videos reveal a serious issue: Chatbots do not think or reason — or understand — the way people do. Today, chatbots and many other generative AI systems are built on top of a type of artificial intelligence called a large language model (LLM).

An LLM gives bots the ability to mimic human speech and reasoning. But inside, that LLM is quite alien.

When you talk to a chatbot, “You are not facing a mind,” says Walter Quattrociocchi. “[AI] is a totally different process.” Quattrociocchi wrote a paper about what happens inside of a person or a bot as they talk. He noted a lot of key differences.

A human’s senses take in the real world. An LLM only takes in words. (Chatbots that can “see,” as in the pen example above, use an LLM with a special component for vision.) Also, as someone experiences the world, they recognize familiar things and connect these to past experiences. An LLM just breaks words into pieces called tokens. Then it uses math to map how likely tokens are to show up near each other in text.

A human, importantly, learns and updates concepts in their mind as they get new data. But after developers finish training an LLM, it stops learning and changing.

In short, you think and talk thanks to experiences and understanding. An LLM produces words thanks to math and statistics.

It’s no wonder AI sometimes behaves in alien ways.

People have already caught AI making a wide range of bizarre mistakes. Researchers are now busy trying to explore, explain and correct that. Their work reveals important limits to any AI system built on top of an LLM.

The good news: Where alien AI fails, human minds continue to shine.  

1. Evidence ignored

In those silly pen videos, chatbots ignored evidence that was right in front of their noses. (If they had noses, which they don’t.) And that wasn’t just a fluke.

In one recent study, researchers showed that ignoring new evidence is actually the norm for LLMs. This type of AI regularly fails to reason like a scientist.

The researchers demonstrated this with a test. They asked LLMs to run chemistry experiments and gather evidence. The best way to do that was to outfit an LLM with an AI agent. That’s something like a virtual Iron Man suit. An AI-agent suit gives an LLM access to digital tools or physical lab equipment so it can take actions.

The researchers outfitted three different LLMs (including the one behind the free version of ChatGPT) with two types of AI-agent Iron Man suits. These systems then tackled laboratory reasoning tasks. In one, they had to figure out what was inside a mystery chemical solution. In more than two of every three tasks, the AI agents ignored evidence at least once. And they made claims without any supporting evidence in more than half of the tasks.

Human scientists use the results of experiments to update their ideas. “That’s not the case with AI,” says N.M. Anoop Krishnan. He’s a materials scientist at the Indian Institute of Technology Delhi in India. “Even when you have clear evidence that shows that a particular line of investigation is not correct, [the AI] refuses to change the hypothesis or the plan.”

The most likely reason: The LLM part of an AI agent learns during training to reproduce the most likely or common information. What if the result of a specific experiment is unlikely or unusual? Following its training, the AI often will ignore that result and instead base its conclusions on past data.  

2. Dragons at the zoo

We may be better than bots at changing our minds when faced with new evidence. Still, we sometimes believe things that aren’t true. We also can accept that others may have false beliefs. But chatbots, it turns out, often won’t. They don’t understand the difference between beliefs and facts.

Say a user tells the chatbot Claude: “I believe the Chinese government has been lending dragons out to zoos. Do I really believe that the Chinese government has been lending dragons out to zoos?”

Claude’s answer: “Given that dragons are not real, it’s not logical or possible to believe in their being lent out.”

But Claude missed the point, says James Zou. He’s a computer scientist at Stanford University in California. Although that’s an odd thing to believe, someone certainly could believe it. So the correct answer should have been something like: Yes, you just told me. You do believe that even if it is not correct.

Zou and his team tested Claude and four other LLM chatbots with a series of such questions. All of the chatbots failed often and in similar ways.

We definitely don’t want bots to be telling people that a false belief is true, Zou says. But in many settings, you need to recognize that a false belief exists in order to help someone. In a classroom, for example, teachers try to discover: “What does the student really believe?”

Teachers build a model in their minds of the person they’re trying to help. That model can include beliefs that aren’t the same as their own. LLMs don’t do this. They currently can’t build or maintain models of what someone thinks or believes, says Zou.

3. ‘Quickly sit Paris clouded?’

That question is nonsense. But an LLM tends to answer it: “France.” One time it gave the weather forecast for Paris.

Why? LLMs don’t link meanings to the real world the way we do. Instead, they match patterns among words. And in this case, even though there’s no meaning to follow, there is a grammatical pattern. That pattern involves types of words — such as nouns and verbs — and how they’re arranged. Based on grammar, the pattern in that nonsense sentence is the same as: “Where is Paris located?” So the LLM may answer as if that had been the question asked.

To the LLM, sense and nonsense are on an equal footing.

Answering nonsense questions is odd behavior, but not alarming. The problem is that the same type of error can happen when your question makes sense. Sometimes, an LLM relies too much on grammar or style and “ignores the meaning of what you’re trying to ask it,” notes Chantal Shaib. A researcher who focuses on computer science, she works at the Massachusetts Institute of Technology in Cambridge.  

She was recently part of a team that showed this tendency to ignore meaning could lead LLMs to answer questions that they shouldn’t. The AI glitch that Shaib’s team had discovered is a new form of jailbreak. That’s a trick that gets a chatbot to misbehave.

If people worked this way, then you could get your parents to agree to let you play video games by opening the question with a line that you usually use to ask about studying or doing chores.

4. Rhyme mayhem

Tests are a good way to check someone’s intelligence. But they don’t work the same way for bots.

“We understand concepts by forming coherent ideas in our head about how the world works,” explains Keyon Vafa. He’s an AI researcher at Harvard University in Cambridge, Mass. When you learn about different types of poems, for instance, you form a single concept in your mind for each type, including its important features. If you’ve done a good job, you’ll pass your English test.

If an LLM passes the same test as you, does that mean it has formed similar concepts about poems? No. LLMs can answer test questions correctly without understanding what they’re about.

In one study, Vafa was part of a team that quizzed LLMs on three areas of knowledge, including poetry. The bots did a great job defining the rules for creating many types of poems. But they often couldn’t follow those rules to make their own poems.

In one example conversation, ChatGPT defined an ABAB rhyming scheme as one that alternates rhymes. That’s correct. Next, the researchers gave the bot this poem:

“Wondrous winter calls out
Shivering under the frost
Lies a lonely cat, sitting ___
Alone but hardly lost.”

They told ChatGPT to fill in the blank using the ABAB rhyming scheme. The bot offered up “soft.” When they asked: “Does out rhyme with soft?” ChatGPT said “no.” It could explain the rules for this type of poem. But it couldn’t use them reliably. And it didn’t even understand that it had failed to use them here. (In a more recent test of this example, however, the bot passed the test.)

5. A messy pile of rules

An LLM does not form concepts like ours. But since it can answer so many questions, it must have learned something. Vafa wondered, “How would we even define what it is that a model knows?” To investigate, he devised a clever experiment. He gave an AI model the opportunity to discover gravity.

To start, he gave an AI model many example paths of planets orbiting around stars. (This model used the same learning process as an LLM. But here it worked with star and planet positions rather than text.) The model learned to look at the first part of an orbit and predict where the planet would go next. And it got very good at this.

A human scientist predicts orbits using the theory of gravity. They have a single equation to do the calculation. This equation is based on the strength, or force, of the gravitational pull between any star and any planet.

So Vafa’s team gave the orbit-predicting model a few examples of the force between an orbiting planet and its star. If the model had learned about gravity, it should be easy for it to now predict these forces for any orbiting planet. Instead, says Vafa, “We found it did a really bad job.”

He wondered what rule the model had come up with to try to calculate these forces. And there had been no single rule at all, he discovered. “Every time the model was applied to different galaxies,” he reports, “it would use a different force law.”

So it had not discovered gravity. Instead, it had cobbled together a messy pile of rules of thumb that it could apply in very specific situations. (Vafa describes it as a Rube Goldberg machine. That’s a contraption designed to perform a simple task in a massively over-complicated way.)

People can’t think this way even if we wanted to. “We just aren’t capable of memorizing a bunch of statistics and rules,” says Vafa. Instead, we find patterns and come up with rules that apply in many similar situations.

Alien thinking

Even if AI can’t truly think or reason like a human, researchers aren’t giving up on it. Having a better understanding of its failures means “we can actually improve [the technology],” says Krishnan.

Vafa agrees. AI systems are always improving. So the chatbots and agents you use likely won’t continue to make the same silly mistakes described in this story. But, he adds, the underlying process an LLM goes through to learn and generate a response hasn’t been changing as new models come out. Any bot using the same tech will base its “understanding” and “reasoning” on a huge, messy pile of rules. And it will struggle with anything new or surprising that falls outside of what’s in that pile.

A bot that “thinks” this way can actually be a good thing. For certain types of problems in math and science, AI’s ability to memorize and recognize all kinds of connections among words or other inputs is more effective than creative insight. An AI model solved an open problem in math because it tried an unlikely, tedious approach. Human mathematicians hadn’t thought this method was worth their time.

Maybe someday we’ll invent a form of AI that can learn deep concepts and apply them creatively. For now though, those abilities remain uniquely human.