AIGenerated
EN · IT · ES

AI shopping bots get more confident asking to think harder — but not any smarter

A new study shows that making AI assistants reason longer before suggesting products only makes them sound sure of themselves, not actually more right.

Português O PlayStation 2 (oficialmente abreviado como PS2) foi o segundo console produzido pela empresa Sony, após o PlayStation original.
File picture · Photo: Deni Williams from São Paulo, Brasil / Brazil, CC BY 2.0, via Wikimedia Commons (licence)

What you could do with it

  • A shopper using an AI assistant to pick products online could remember that a model sounding very sure of itself does not mean its recommendation is actually better 1.
  • A developer building an AI shopping agent could skip adding extra reasoning steps just to make the system sound more confident, since the study found it does not improve real accuracy 1.
  • A researcher studying AI reliability could use the WebShop, SQID, and WANDS benchmarks from the study to test whether new models suffer from the same confidence-accuracy gap 1.

The experiment

The researchers tested ten leading AI models as autonomous shopping agents — programs that can browse product listings and make recommendations on behalf of users 1. They built three benchmarks from real e-commerce platforms: WebShop, Amazon's SQID dataset, and Wayfair's WANDS platform . Each benchmark presented the AI with multimodal product information, combining both images and text descriptions, and asked it to choose the best option.

The key question was simple: if you ask a model to think harder — spending more reasoning effort on each choice — does it actually pick better products?

Across all three benchmarks, the answer was no. Increasing thinking effort did not raise decision quality in any statistically meaningful way 1.

Confidence went up. Accuracy stayed flat.

Where the extra thinking effort did have a measurable effect was on confidence. Models that were asked to think longer reported significantly higher confidence in their choices 1. In plain terms: the AI became more convinced it was right, but it wasn't actually more right .

The researchers also measured calibration — whether a model's stated confidence matched its true accuracy. They found that most models were poorly calibrated at every level of reasoning effort, meaning their confidence numbers could not be trusted as a guide to correctness 1.

Errors travel in packs

Another striking finding was that mistakes are strongly shared among models. When one AI model got a task wrong, several others tended to err on the same task as well 1. On problems where many models failed, a large number of them picked the exact same wrong product .

What this means for everyday shoppers

As AI shopping assistants become more common, this research is a reminder that a model sounding confident does not mean it has made a good choice. The extra thinking time that some services advertise may give the impression of deeper analysis, but the study found it does not translate into better product recommendations 1.

How good is it

The researchers Adi Krishnamoorthy and X.Y. Han tested ten frontier AI models on three e-commerce benchmarks built from WebShop, Amazon's SQID dataset, and Wayfair's WANDS platform 1. They found that increasing reasoning effort did not improve decision quality in any statistically meaningful way across all three benchmarks . At the same time, models reported significantly higher confidence when asked to think longer, and most models were poorly calibrated at every reasoning level . The mistakes were also strongly shared, meaning many models tended to pick the exact same wrong product on tasks where several failed .

Key facts

  • Ten frontier AI models were tested as shopping agents across three e-commerce benchmarks
  • More reasoning effort increased reported confidence but not decision quality
  • Most models were poorly calibrated — their confidence did not match actual accuracy
  • Errors were shared across models: when one failed, others often failed on the same task with the same wrong answer

Sources

  1. Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM

If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work

Keep reading

A figure from the paper "Verifiable pXRD: Hill-climbing pXRD structure determination with LLM agents".How it worksAI agents boost crystal structure recovery from 27% to 69%, potentially speeding up new material researchHerald of AI▶Figure: Chen et al., arXiv:2611.17539, CC BY 4.0

AI agents boost crystal structure recovery from 27% to 69%, potentially speeding up new material research

A new AI system uses large language model agents to figure out the arrangement of atoms in crystals from powder diffraction data much faster, which could accelerate drug discovery and new battery development.

How it works