HTTP/2 301 server: GitHub.com content-type: text/html location: https://ai4ce.github.io/INT-ACT/ x-github-request-id: 9812:272D88:827BB1:927DA6:6951CEEA accept-ranges: bytes age: 0 date: Mon, 29 Dec 2025 00:44:27 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210071-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766969067.939767,VS0,VE200 vary: Accept-Encoding x-fastly-request-id: 9a4e5b8c2a5ec9eed40ef3fe6ca9f83c390f5a25 content-length: 162 HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Sat, 28 Jun 2025 07:33:44 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"685f9ad8-606e" expires: Mon, 29 Dec 2025 00:54:27 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: 576B:3FD64F:81CD62:91D305:6951CEEA accept-ranges: bytes age: 0 date: Mon, 29 Dec 2025 00:44:27 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210071-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766969067.153057,VS0,VE212 vary: Accept-Encoding x-fastly-request-id: 1fd1321f032781339dcce81b000c904c55a5e9b4 content-length: 5863

From Intention to Execution

Probing the Generalization Boundaries of Vision-Language-Action Models

Irving Fang*, Juexiao Zhang*†, Shengbang Tong, Chen Feng

New York University
^* Equal Contribution, ^† Project Lead

arXiv Code

TLDR

1. INT-ACT: probing suite to evaluate the generalization capability of robotic VLAs.
2. Benchmarking SOTA VLAs to understand their generalization boundaries.

---

INT-ACT Categories

INT-ACT is a probing suite to evaluate the generalization capability of robotic VLAs. It consists of three categories of tasks that probe the generalization boundaries of VLAs.

Object Diversity: Ability to handle out-of-distribution objects.
Language Complexity: Ability to understand complex language instructions.
Vision-Language Thinking: Ability to perform commonsense reasoning and visual-language thinking.

Object Diversity Language Complexity Vision-Language Thinking

Truly generalist policies require perceptual ability beyond the object distributions encountered during training or fine-tuning.

In SimplerEnv, which assume the fine-tuning dataset is BridgeV2, all manipulation tasks are Put {Source} on {Target}. Therefore, We introduce four categories of out-of-distribution objects that resemble original objects in affordances/grasping difficulty.

OOD Source: Source object not present in BridgeV2, but target object is.
OOD Target: Target object not present in BridgeV2, but source object is.
OOD Source + Target: Both source and target objects are not present in BridgeV2.
OOD Relation: Relation between objects is different from the training data. For example, if the training data has Put {Source} on {Target}, then the OOD relation can be Put {Target} on {Source}.

---

Benchmarking Results

Intention-Action Gap OOD Objects Preserving Linguistic Capability Preserving Vision-Language Thinking

All VLAs exhibit persistent intention-action gaps. They correctly interpret out-of-distribution objects or instructions, thanks to their pretrained VLM, but their execution accuracy still falls sharply.

---

BibTeX

Coming Soon
      
    

Acknowledgements

Chen Feng is the corresponding authors.