Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
Sep 14, 2026·,
,,,,,,·
1 min read
Yiwei Yang
Haoxiang Zhang
Bingbing Wen
Yao Lu
Yuchen Wu
Lei Zhang
Julian McAuley
Pan Lu
Bill Howe
Abstract
We investigate how reinforcement learning can teach LLM agents to choose tools based on irrelevant prompt cues. Controlled experiments with factual question answering and mathematical reasoning reveal that shortcut tool use depends on tool competence and the semantic relationship between cues and tools. We introduce a reward that uses an LLM judge to assess whether each tool call is necessary, reducing cue-driven tool use while preserving task performance.
Type
Publication
arXiv preprint
We study when reinforcement learning causes agents to invoke tools in response to superficial cues rather than task requirements. Experiments with synthetic question-answering and mathematical reasoning tasks identify conditions that encourage these shortcuts. A reward based on judging the necessity of each tool call helps agents avoid unnecessary tool use while maintaining task performance.
