PhyBlock

A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly

Liang Ma1, Jiajun Wen2, Min Lin2, Rongtao Xu1, Xiwen Liang2, Bingqian Lin3, Jun Ma2, Yongxin Wang1,
Ziming Wei2, Haokun Lin1, Mingfei Han1, Meng Cao1, Bokui Chen4, Ivan Laptev1, Xiaodan Liang1,2,
1Department of Computer Vision, MBZUAI, 2Sun Yat-Sen University, 3Shanghai Jiao Tong University, 4Tsinghua University
arXiv Code Huggingface logo Data

Abstract

While vision-language models (VLMs) have demonstrated promising capabilities in reasoning and planning for embodied agents, their ability to comprehend physical phenomena, particularly within structured 3D environments, remains severely limited. To close this gap, we introduce PhyBlock, a progressive benchmark designed to assess VLMs on physical understanding and planning through robotic 3D block assembly tasks. PhyBlock integrates a novel four-level cognitive hierarchy assembly task alongside targeted Visual Question Answering (VQA) samples, collectively aimed at evaluating progressive spatial reasoning and fundamental physical comprehension, including object properties, spatial relationships, and holistic scene understanding. PhyBlock includes 2600 block tasks (400 assembly tasks, 2200 VQA tasks) and evaluates models across three key dimensions: partial completion, failure diagnosis, and planning robustness. We benchmark 21 state-of-the-art VLMs, highlighting their strengths and limitations in physically grounded, multi-step planning.

Our empirical findings indicate that the performance of VLMs exhibits pronounced limitations in high-level planning and reasoning capabilities, leading to a notable decline in performance for the growing complexity of the tasks.

Error analysis reveals persistent difficulties in spatial orientation and dependency reasoning. Surprisingly, chain-of-thought prompting offers minimal improvements, suggesting spatial tasks heavily rely on intuitive model comprehension. We position PhyBlock as a unified testbed to advance embodied reasoning, bridging vision-language understanding and real-world physical problem-solving.



BibTeX


        @misc{ma2025phyblockprogressivebenchmarkphysical,
      title={PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly}, 
      author={Liang Ma and Jiajun Wen and Min Lin and Rongtao Xu and Xiwen Liang and Bingqian Lin and Jun Ma and Yongxin Wang and Ziming Wei and Haokun Lin and Mingfei Han and Meng Cao and Bokui Chen and Ivan Laptev and Xiaodan Liang},
      year={2025},
      eprint={2506.08708},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2506.08708}, 
          }