Baseline failure · Dot IR success
Prompt probe
“Reach the second largest food item on the chopping board.”
Baseline
Failure · wrong object
Oracle dot IR
Success
BEHAVIOR-Spatial Benchmark
Evaluating Spatial Understanding for Robotics
Can a model that answers a spatial question also identify the intended target and guide a robot to act on it?
BEHAVIOR-Spatial connects these evaluations across shared scenes, objects, and spatial relations.
Abstract
Robot tasks require grounding spatial language in physical scenes, but task success alone cannot establish whether spatial instructions are followed. We present BEHAVIOR-Spatial, a benchmark built on BEHAVIOR-1K that measures spatial question answering, point-based target grounding, and robot action to examine how VLM spatial understanding relates to VLA behavior. It comprises 3,256 spatial questions across 49 activities, 1,240 grounding queries across 14 manipulation tasks, and action evaluations covering those 14 tasks and 10 navigation tasks. Our manipulation protocol evaluates held-out configurations and instruction following when the requested target changes within a fixed training layout. The baseline achieves 41.6% on held-out configurations but only 16.1% under changed instructions. VLM adaptation and robot-data pretraining provide limited improvements, whereas oracle visual target cues raise these rates to 60.3% and 38.0%, respectively. Perception evaluations reveal policy-dependent correlations between VLM question-answering and grounding accuracy and VLA manipulation performance. Together, these findings motivate evaluating spatial instruction following alongside task completion and examining when spatial perception performance translates into robot behavior.
Benchmark Overview
Understand. Ground. Act.
3,256 spatial questions across 49 activities, 1,240 grounding queries, and action evaluations covering 14 manipulation and 10 navigation tasks.
Data Curation
We derive the benchmark from BEHAVIOR-1K demonstrations, combining recorded observations with simulator geometry. Spatial metadata and curated task instances produce question-answering samples, grounding labels, and action datasets, with validation for each suite.
Vision-Language-Action Model
BEHAVIOR-Spatial-VLA uses task-specific camera views, a Qwen3.5-4B backbone, and a flow-matching action expert. Manipulation variants test backbone adaptation, robot-data pretraining, and oracle visual target cues; navigation incorporates scene context.
Evaluation Results
VLM evaluation · General spatial VQA
| Model | Single-frame | Multi-frame | ||||
|---|---|---|---|---|---|---|
| All ↑ | Dir. ↑ | Dist. ↑ | Size ↑ | All ↑ | Dir. ↑ | |
| Proprietary | ||||||
| GPT-6 Astra | 82.1 | 85.8 | 77.3 | 77.7 | 74.4 | 74.6 |
| Gemini 3 | 77.8 | 80.5 | 75.2 | 72.7 | 71.1 | 69.6 |
| Fable5.1 | 70.5 | 69.3 | 70.2 | 76.3 | 68.4 | 66.8 |
| GPT-5 | 64.7 | 61.7 | 68.1 | 69.1 | 63.2 | 59.6 |
| Gemini 2.5 | 59.9 | 55.0 | 65.2 | 68.3 | 60.1 | 57.4 |
| Open-weight | ||||||
| Molmo2-8B | 51.8 | 45.0 | 57.5 | 66.9 | 36.8 | 30.7 |
| Qwen3.5-9B | 50.6 | 42.0 | 58.1 | 69.1 | 54.5 | 51.4 |
| Qwen3.5-4B | 46.8 | 44.8 | 48.1 | 51.8 | 46.6 | 44.1 |
| Molmo2-4B | 45.5 | 34.2 | 58.7 | 62.6 | 32.4 | 23.7 |
| PaliGemma | 36.2 | 32.5 | 40.7 | 41.0 | 37.9 | 34.4 |
| PaliGemma 2 | 34.8 | 29.8 | 39.2 | 45.3 | 27.3 | 23.4 |
| Gemma 3-4B | 34.4 | 35.0 | 33.6 | 33.8 | 29.2 | 21.3 |
| Qwen2.5-VL | 33.4 | 31.2 | 37.8 | 32.4 | 28.9 | 22.9 |
| DeepSeek-VL2-Tiny | 31.4 | 30.5 | 34.8 | 26.6 | 25.0 | 17.6 |
| VLM backbones extracted from released VLA models | ||||||
| OpenHelix | 9.6 | 5.8 | 14.2 | 15.1 | — | — |
| π₀.₅ | 0.9 | 0.8 | 0.3 | 2.9 | — | — |
| Chance | 25.0 | 25.0 | 25.0 | 25.0 | 25.0 | 25.0 |
Direction includes locate. “All” is pooled over questions. Higher is better; bold marks the highest value in each column. A dash indicates an unreported result.
VLM evaluation · Task-grounded perception
| Model | Task-grounded VQA | IR | ||||
|---|---|---|---|---|---|---|
| All ↑ | Dir. ↑ | Dist. ↑ | Size ↑ | Strict ↑ | Tolerant ↑ | |
| Qwen3.5-9B | 57.2 | 56.3 | 61.4 | 52.5 | 51.4 | 59.7 |
| Molmo2-4B | 53.0 | 54.1 | 56.4 | 46.8 | 27.1 | 40.5 |
| Molmo2-8B | 50.0 | 49.9 | 53.4 | 45.3 | 46.7 | 58.4 |
| Qwen3.5-4B | 49.4 | 50.9 | 48.6 | 48.6 | 43.3 | 56.2 |
| Gemma 3-4B | 33.5 | 23.0 | 42.4 | 36.0 | 36.8 | 47.9 |
VQA and IR are evaluated on shared manipulation task contexts. Higher is better; bold marks the highest value in each column.
VQA & Grounding Examples
Manipulation · policy evaluation
| Policy | Held-out | Prompt-probe | |||
|---|---|---|---|---|---|
| +Soft ↑ | S ↑ | +Soft ↑ | S ↑ | W ↓ | |
| π₀.₅-Qwen (frozen VLM) | 41.6 | 27.9 | 16.1 | 10.0 | 70.0 |
| + LoRA-VLM | 43.5 | 30.9 | 17.9 | 9.6 | 68.1 |
| + DROID pretrain init | 40.1 | 24.7 | 16.6 | 8.9 | 66.0 |
| IR-conditioned (oracle IR) | |||||
| + dot IR | 60.3 | 44.8 | 38.0 | 24.9 | 43.9 |
| + dot IR + first-frame anchor | 59.1 | 41.8 | 36.3 | 23.7 | 42.4 |
| + attn-map IR | 59.3 | 39.0 | 35.0 | 17.0 | 46.0 |
| Data scaling · dot IR recipe | |||||
| + dot IR @ 50% data | 53.7 | 33.7 | 30.6 | 19.4 | 45.7 |
| + dot IR @ 30% data | 56.1 | 34.5 | 32.6 | 19.9 | 46.6 |
+Soft: soft success, a proxy for target-selection intention. S: strict success (task completion). W: wrong-target commitment. IR-conditioned policies receive oracle target cues. Bold marks the best value in each column.
Reading the results
Separate grounding from execution.
Soft success is a behavioral proxy for target-selection intention. The baseline achieves 41.6% soft success on held-out configurations and 16.1% when instructions change within a fixed training layout. Oracle dot IR target cues raise these to 60.3% and 38.0%, respectively.
Strict success requires completing the instructed interaction. On the prompt probe, oracle dot IR achieves 24.9% strict success, compared with 38.0% soft success: identifying the target helps, but does not solve execution.
Manipulation · code-as-policy agents
| Writer | +Soft ↑ | S ↑ | W ↓ |
|---|---|---|---|
| Proprietary writers | |||
| Fable 5.1 | 66.1 | 51.4 | 23.5 |
| GPT-6 | 61.2 | 47.3 | 23.0 |
| Gemini 3 | 50.3 | 33.3 | 28.8 |
| Open-weight writers | |||
| Qwen3.5-9B | 39.0 | 5.3 | 8.7 |
| Qwen3.5-4B | 37.1 | 5.1 | 7.2 |
Writers generate robot-control programs using shared perception tools and a motion API. Programs are developed on training episodes, then frozen for held-out evaluation. +Soft: soft success. S: strict success. W: wrong-target commitment. Bold marks the highest soft and strict success rates.
Navigation VQA Examples
Success & Failure Rollouts
Manipulation rollouts
Baseline and oracle dot IR on the same task instances. Clips show a third-person view with a head-camera inset and play at 4× speed. Oracle dot IR supplies the instructed target as a visual mark.
Baseline failure · Dot IR success
Held-out
“Reach the pizza plate farthest from the black chair on the breakfast table.”
Baseline
Failure · wrong object
Oracle dot IR
Success
Baseline failure · Dot IR success
Prompt probe
“Reach the hardback book at the 2x2 grid of the bookcase.”
Baseline
Failure · wrong object
Oracle dot IR
Success
Both policies succeed
Held-out
“Reach the largest plate from the countertop.”
Baseline
Success
Oracle dot IR
Success
More examples · 8 additional comparisons
Baseline failure · Dot IR success
Baseline failure · Dot IR success
Held-out
“Reach the largest toy on the desk.”
Baseline
Failure · wrong object
Oracle dot IR
Success
Baseline failure · Dot IR success
Prompt probe
“Place the carving knife on the countertop and to the left of the cutting board.”
Baseline
Failure
Oracle dot IR
Success
Baseline failure · Dot IR success
Prompt probe
“Place the tupperware in the layer 6 of the fridge.”
Baseline
Failure
Oracle dot IR
Success
Baseline failure · Dot IR success
Prompt probe
“Reach the dice closest to the wall on the bed.”
Baseline
Failure · wrong object
Oracle dot IR
Success
Both policies succeed
Both policies succeed
Held-out
“Reach the second smallest food item from the chopping board.”
Baseline
Success
Oracle dot IR
Success
Both policies succeed
Held-out
“Reach the hardback book at the 3x2 grid of the bookcase.”
Baseline
Success
Oracle dot IR
Success
Both policies fail
Both policies fail
Prompt probe
“Reach the second largest food item on the chopping board.”
Baseline
Failure · wrong object
Oracle dot IR
Failure · wrong object
Both policies fail
Prompt probe
“Reach the hardback book at the 3x1 grid of the bookcase.”
Baseline
Failure · wrong object
Oracle dot IR
Failure · no target commitment
Perception–Action Correlation
How do spatial question answering and target grounding relate to robot performance? Across the 14 manipulation tasks, the relationship between perception accuracy and policy soft success varies by policy variant.
Task-level correlations
| Policy | VQA (ρ) | IR (ρ) |
|---|---|---|
| Baseline | 0.713 | 0.658 |
| LoRA VLM | 0.616 | 0.581 |
| IR | 0.560 | 0.701 |
| IR + first-frame IR | 0.612 | 0.618 |
| Attention map | 0.608 | 0.753 |
Spearman correlations across 14 manipulation tasks between Qwen3.5-4B VQA or tolerant IR accuracy and policy held-out soft success. Ties receive average ranks.
