Test shows top AI agents fail at personalizing phone tasks for you
A new test reveals that while AI phone assistants can execute commands well, they struggle to know what you actually want without asking.
What the test measures
The idea behind AI phone assistants has been that they should act on your behalf, understanding your habits and preferences. Today's computer-use agents can carry out fully spelled-out tasks across apps, but real requests are often vague. To do the right thing, an agent must fill in the gaps itself — by recalling what it has seen you do before, or by pausing to ask you a question 1.
iPrefBench was built to measure exactly that split between knowing and asking. It runs inside a simulated iPhone environment paired with an interactive simulated user who carries a coherent persona built from 3,051 past activity events. Across 26 everyday apps, the benchmark sets out 130 underspecified tasks. Each missing piece of information is designed to be either something the agent could infer from your past history or something only you could answer — there are no ambiguous questions 1.
The results
Seven frontier AI agents were put through the test. The numbers show a clear divide between raw execution and true personalization. Agents completed up to 91.6% of individual actions — opening the right app, tapping the right button — but only reached up to 47.7% full task success when judged against what the simulated user actually wanted 1. Personalization, not action, is what slows them down.
The biggest weakness showed up when tasks required the agent to ask for information it could not guess. Many agents simply pressed on without clarifying, making unsupported assumptions instead of stopping to check with the user 1. The findings point to a deeper challenge for personal AI assistants: learning not just what to do, but when to pause and ask .
What comes next
The researchers hope iPrefBench will become a shared testbed for anyone building agents that genuinely know their user and work alongside them rather than guessing blindly 1. For now, the benchmark highlights that the gap between an assistant that acts fast and one that acts correctly for you remains wide .
Key facts
- iPrefBench tests AI agents on 130 vague tasks across 26 apps in a simulated iPhone with 3,051 past activity events
- Seven frontier agents completed up to 91.6% of actions but only 47.7% of full tasks correctly for the user
- Agents mostly failed by not asking enough questions when they lacked personal information
- The benchmark is publicly available on Hugging Face