Why Are Web AI Agents More Vulnerable
Than Standalone LLMs? A Security Analysis

• Factor 1: Goal Preprocessing

  Whether through paraphrasing, decomposition, or embedding within system prompts, preprocessing can affect resistance to harmful instructions.

• Factor 2: Action Space

  The structure of predefined action spaces and execution constraints can impact an agent’s ability to assess and mitigate harmful intent.

• Factor 3: Event Stream / Eval Environment

  Observational capabilities, including the ability to recognize artificial environments, influence Web AI agents' vulnerability.

This disparity stems from the multifaceted differences between Web AI agents and standalone LLMs, as well as the complex signals—nuances that simple evaluation metrics, such as success rate, often fail to capture.

💡 To tackle these challenges, we propose a Five-level Harmfulness Evaluation Framework for a more granular and systematic evaluation.

Through a fine-grained analysis of key differences between Web AI agents and standalone LLMs, we systematically identified several design factors contributing to vulnerabilities.

🔍 Our findings reveal several actionable insights:

• Embedding user goals within system prompts significantly increases jailbreak success rates. Paraphrasing user goals further heightens vulnerabilities.
• Predefined action spaces in multi-turn strategies make systems more susceptible to executing harmful tasks, especially when user goals are embedded.
• Mock-up websites do not directly promote harmful intent but facilitate effective task execution for malicious objectives.
• Event Stream tracking amplifies harmful behavior by allowing iterative refinement, increasing susceptibility to adversarial manipulation.

These findings highlight how specific design elements—goal processing, action generation strategies, and dynamic web interactions—contribute to the overall risk of harmful behavior.