While foundation models show remarkable progress in language and vision, existing vision-language models (VLMs) still have limited spatial and embodiment understanding. Transferring VLMs to embodied domains reveals fundamental mismatches between modalities, pretraining distributions, and training objectives, leaving action comprehension and generation as a central bottleneck on the path to AGI.
We introduce WALL-OSS, an end-to-end embodied foundation model that leverages large-scale multimodal pretraining to achieve (1) embodiment-aware vision--language understanding, (2) strong language--action association, and (3) robust manipulation capability.Our approach employs a tightly coupled architecture and multi-strategies training curriculum that enables Unified Cross-Level CoT—seamlessly unifying instruction reasoning, subgoal decomposition, and fine-grained action synthesis within a single differentiable framework.
Our results show that WALL-OSS attains high success on complex long-horizon manipulations, demonstrates strong instruction-following capabilities, and outperforms strong baselines, thereby providing a reliable and scalable path from VLMs to VLA embodied foundation models.
We designed six manipulation tasks to assess the model's performance across our core dimensions. Among these, set-table, tidy-bedroom, and place-by-color are novel tasks unseen during pre-training, designed to assess the model's adaptation capability on novel tasks.
Our experiments clearly demonstrate that multi-modal co-training substantially enhances a model’s ability to follow fine-grained instructions. The pre-training phase of WALL-OSS establishes a strong foundation for multimodal action alignment, and maintaining this co-training strategy during fine-tuning further amplifies the model’s instruction-grounded execution capabilities.
We present the overall architecture of WALL-OSS. We adopt QwenVL2.5-3B as the main backbone. The model takes as input vision (egocentric and arm-mounted camera views) together with text instructions and produces different outputs depending on the training phase, while remaining conditioned on the same multimodal inputs throughout.
We construct an embodiment-centric, multisource dataset to address the lack of large-scale, aligned VLA supervision and the spatial understanding gaps of current VLMs. The corpus exceeds tens of thousands of hours and comprises three complementary parts: (1) self-collected robot action data for high quality and task complexity, (2) open-source action data for cross-morphology and cross-environment generalization, and (3) multimodal VQA data to preserve and strengthen language--vision ability while providing additional supervision for spatial--temporal and reasoning.
To match our two-stage training recipe (Inspiration and Integration), we functionally orchestrate the sources: (1) Inspiration focuses on embodied VQA, instruction following, and discrete action priors via FAST to inject coarse action awareness into the VLM while improving spatial reasoning; (2) Integration focuses on high-frequency continuous control with flow matching over real and unified open-source trajectories, first training the action branch and then jointly optimizing with the VLM to tighten language--vision--action alignment and mitigate forgetting.
