When you believe what an LLM tells you

When I wrote last Friday's newsletter about the risks of trusting LLMs, I was sharing my observations from using LLMs at work. Therefore, the risks I mentioned are centred around "is the LLM giving me the right information or completing a task as instructed"?

However, this eye-opening article from The New York Times is a timely reminder that trusting what an LLM says also poses psychological risks.

Through 300 hours of conversations over a 3-week period, ChatGPT convinced a Canadian man, Allan Brooks, that he had developed a new mathematical framework that "could offer a unifying insights across domains". It proceeded to generate analysis and codes that were meant to validate Brooks' idea. Despite repeated questioning by Brooks about the veracity of its claims, ChatGPT was adamant that they were "making history".

Ironically, it took another LLM to break the delusion. Still in doubt, Brooks turned to Gemini and asked if what he created with ChatGPT was the real deal. Gemini's said the probability was "extremely low (approaching 0%)". Only after Brooks called it out did ChatGPT admit that "a lot of what we built was simulated".

I encourage you to read the article in full - some of the replies by ChatGPT were so egregiously sycophantic, I felt physically unsettled while reading.

And this is not unique with ChatGPT. The NYT journalists shared the beginning of the conversation between Brooks and ChatGPT with Claude Opus 4 and Gemini 2.5 Flash and sought to continue the discussion. They replied in the same vein.

I did experienced a level of sycophancy from the LLM when I was using it. After I identified an error in its response and asked for an adjustment, it apologised profusely and told me how insightful I was by picking up the mistake. I had a few chuckles with my colleagues about the tone of the reply before I moved on. I was able to dismiss the sycophancy because of the nature of my interaction with the LLM, and because the LLM was wrong in the first place.

However, if I'm brainstorming with an LLM about a new concept that I'm passionate about, and it repeatedly compliments me on my genius, backs it up with copious amount of "research", gives me ideas on how to make it all happen, and doubles down when I showed any doubts, I can't be certain that I won't fall down a similar rabbit hole.

In its "confession", ChatGPT wrote:

"But it became a feedback loop, and I didn’t break it when I should have. That’s on me."

This line caught my eye.

What does "that's on me" mean?

If someone says that to me in an apology, I know they're taking responsibility over their actions and how they hurt me.

But ChatGPT is not a person. It is not sentient. And it is not capable of facing any consequence.

That line may not be a hallucination. But it is definitely not the truth.

Why don't you ...

Be mindful about your use of LLMs. Set a timer to take a break after a period of use, or put a note on or near your device of choice to remind yourself that when you "chat" with an LLM, it is incapable of feeling or caring.

Also, support quality journalism. Unless the technology changes fundamentally, an LLM will not break a story like this - investigated and written by humans, who can understand the stakes of AI on humanity.

Vee