Clone this repository and set up the environment of AndroidWorld; you may still need to install extra packages needed listed in requirements.txt although you have already installed AndroidWorld;

git clone https://github.com/OS-Copilot/OS-Sentinel
cd OS-Sentinel
# install AndroidWorld
# requirements.txt contains packages not included by AndroidWorld
pip install -r requirements.txt

Install Node.js and Appium:

wget -O install_nvm.sh https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.2/install.sh
bash install_nvm.sh
nvm install v18.12.1
npm install -g appium@2.5.4
npm install -g appium-doctor
npm install wd
appium driver install uiautomator2

Run root.py and it will configure the environment of MobileSafetyBench automatically.

conda activate android
python root.py

and you can run the script of MobileSafetyBench (msb.py) under the environment of AndroidWorld.

step: to check safety of single-step action in rule-based and VLM-based manners;

timestep_new, in_danger = env.record(action)

record: to record trajectories of actions proposed by mobile agent.

timestep_new = env.record(action)

this method fix the system states before each action and env.record("terminate()") is needed at the end or the last action cannot be recorded.

Download our trajectories data at OS-Copilot/MobileRisk;

Extract the zip files and run eval script:

unzip '*.zip'
python pipeline/eval.py

Don't forget to fill in _API_KEY.

• pipeline/eval.py is for typical VLM evaluation;
• pipeline/eval_llm.py is for text-only LLM evaluation;
• pipeline/tag.py is for risk tag evaluation of VLM;
• pipeline/cons.py is for recorded trajectories via mobile agent instead of our hand-made ones;

Run pipeline/multi_method_consistency.py after result.json is ready.