Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Laurence Li, Haiyang Zhang, Qi He, Toby Jia-Jun Li, Dakuo Wang

UXAgent is a framework that uses Large Language Models (LLMs) as agents to conduct usability testing in web environments. These agents simulate human-like behaviors, allowing UX researchers to:

• Perform early usability evaluations.
• Gather actionable design insights.
• Iterate without immediate reliance on human participants.

The system leverages dual-system reasoning for quick decisions and in-depth analysis, and its Universal Web Connector ensures compatibility with any web page. By offering real-time feedback, UXAgent streamlines the design process and improves testing efficiency.

1. Clone the repository:

   git clone git@github.com:neuhai/UXAgent.git

2. install uv, follow this guide

3. Set up the environment:

   uv sync

4. Install Chromium for Playwright:

   uv run playwright install chromium

5. Set API keys:

   # export AWS_ACCESS_KEY_ID=xxx123
   # export AWS_SECRET_ACCESS_KEY=xxx123
   # export OPENAI_API_KEY=sk-123
   export ANTHROPIC_API_KEY=sk-ant-123

6. Optional: Enable "headful" mode: By default, Chrome runs in headless mode (no GUI). To view the browser, set the following:

   export HEADLESS=false

uv run -m src.simulated_web_agent.main --intent "Buy a Jacket from Amazon" --start-url "https://www.amazon.com" --max-steps 20 --wait-for-login

For more options, see the help message:

uv run -m src.simulated_web_agent.main --help

uv run -m src.simulated_web_agent.main.run

runConfig.yaml defines how multiple simulated agents are launched and what behavior or survey the agents performs during a batch run. Refer to this following table for how to use the fields in runConfig.yaml.

Example runConfig.yaml you can get started with:

# Number of personas and concurrency
total_personas: 20
concurrency: 10
# Demographic sampling (weighted random)
demographics:
  - name: "Age"
    choices:
      - { name: "18-55", weight: 1 }
  - name: "Gender"
    choices:
      - { name: "male", weight: 1 }
      - { name: "female", weight: 1 }
      - { name: "non-binary", weight: 1 }
  - name: "Online Shopping Frequency"
    choices:
      - { name: "A few times per year", weight: 1 }
      - { name: "A few times per month", weight: 1 }
      - { name: "A few times per week", weight: 1 }
# The general shopping intent for all agents
general_intent: Buy the highest rated product from the meat substitute category within a budget between 100 and 200. You don't need to finish the purchase, just go to the checkout page.
# Example persona template for generation reference
example_persona: |
  Background:
    Male, age 35-44, tech professional, lives in New Jersey.
  Financial Situation:
    Stable income, careful with expenses.
  Shopping Habits:
    Shops online twice a month, prefers Amazon Prime, brand-loyal but open-minded.
  Professional Life:
    Works full-time in tech; balanced lifestyle with family time and hobbies.

# Starting point for the browser agent
start_url: https://52.91.223.130:7770/
# Maximum allowed agent actions
max_steps: 50
# Post-run questionnaire definition
questionnaire:
  questionnaire_id: web_shopping_usability_v1
  title: System Usability Survey
  questions:
    - id: q1
      type: multiple_choice
      prompt: "I think that I would like to use this system frequently. (1 = Strongly disagree, 5 = Strongly agree)"
      options: ["1", "2", "3", "4", "5"]

In addition to configuring batch runs through runConfig.yaml, you can also use the web-based quick experiment setup UI to launch and manage multiple agents visually. This interface allows you to adjust parameters such as total personas, concurrency, and intent directly from the browser without manually editing YAML files.

1. Have flask and Node.js / npm installed
2. From the project root:

   uv run -m src.simulated_web_agent.main.app

3. Boot up the interface

   cd experiment_ui
   npm install
   npm run dev

4. You can configure your multi-agent run through the experiment configuration wizard in the interface. (set participant demographics, modify task, edit survey, etc.) Click "Confirm and Run" in the UI to start the custom configured multi-agent run.

After running a simulation with UXAgent, you’ll find a structured output folder runs/<timestamp> containing logs, session traces, screenshots, and aggregate metrics.

This project is licensed under the MIT License.

@article{lu2025uxagent,
  title={UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents},
  author={Lu, Yuxuan and Yao, Bingsheng and Gu, Hansu and Huang, Jing and Wang, Jessie and Li, Yang and Gesi, Jiri and He, Qi and Li, Toby Jia-Jun and Wang, Dakuo},
  journal={arXiv preprint arXiv:2504.09407},
  year={2025}
}