Flower Arrangement (Wall-OSS-0.5)

Task Objective

The robot picks up a flower from the tabletop, moves it above the vase opening, and inserts it to the specified depth. The task includes target recognition, precise grasping, spatial alignment, deformable-object manipulation, and safe release, making it suitable for validating single-arm or dual-arm manipulation, data collection, and model fine-tuning workflows.

Data Preparation:

Follow the Data Collection Guide and collect data with reference to the three videos below: left-arm view, primary view, and right-arm view.

Model Training

Code description reference URL: https://github.com/X-Square-Robot/wall-x/blob/main/workspace/TRAIN_ARRANGE_3_FLOWERS.md

Model training uses a real-robot dataset in LeRobotDataset v3 format for the dual-arm arrange 3 flowers task.

Create training configuration

For dual-arm, 448 px, three-camera scenarios, refer to:

cp workspace/example/arrange_3_flowers_wrc_red.yml \
  /path/to/arrange_3_flowers_train.yml

Core path example:

model_type: qwen2_5

model:
  backbone: qwen2_5
  config_path: /path/to/wall-oss-0.5/config.json
  processor_path: /path/to/Qwen2.5-VL-3B-Instruct
  pretrained_path: /path/to/Qwen2.5-VL-3B-Instruct
  attn_deterministic: true
  use_ema: false
  flow_loss_weight: 1.0
  ar_loss_weight: 0.01

data:
  dataset_type: lerobot
  lerobot_config:
    repo_id: /path/to/arrange_3_flowers_lerobot
    root: null
  key_mappings:
    camera:
      observation.images.faceImg: face_view
      observation.images.leftImg: left_wrist_view
      observation.images.rightImg: right_wrist_view
    state: observation.state
    action: action
  norm_stats_path: /path/to/arrange_3_flowers_norm_stats.json
  train_test_split: 0.95
  num_workers: 0
  max_length: 1024
  resolution:
    face_view: 448
    left_wrist_view: 448
    right_wrist_view: 448

checkpoint:
  save_path: /path/to/arrange_3_flowers_ckpt
  resume_from: /path/to/wall-oss-0.5/model.safetensors

Data requirements: The dataset must include observation.state, action, and the three-view image keys observation.images.faceImg / leftImg / rightImg. If state / action is 26-dimensional (including extra DOFs), the first 20-dimensional dual-arm components are automatically sliced according to dof_config during training.

Configure action and state dimensions

The following is an example of dual-arm relative actions for the flower arrangement task. Developers must modify it according to their own robot and data:

task:
  dof_config:
    follow_left_ee_cartesian_pos_relative: 3
    follow_left_ee_rotation_6D_relative: 6
    follow_left_gripper: 1
    follow_right_ee_cartesian_pos_relative: 3
    follow_right_ee_rotation_6D_relative: 6
    follow_right_gripper: 1
    action_padding: 6

  ar_dof_config:
    follow_left_ee_cartesian_pos_relative: 3
    follow_left_ee_rotation_6D_relative: 6
    follow_left_gripper: 1
    follow_right_ee_cartesian_pos_relative: 3
    follow_right_ee_rotation_6D_relative: 6
    follow_right_gripper: 1
    action_padding: 6

  agent_pos_config:
    follow_left_ee_cartesian_pos: 3
    follow_left_ee_rotation_6D: 6
    follow_left_gripper: 1
    follow_right_ee_cartesian_pos: 3
    follow_right_ee_rotation_6D: 6
    follow_right_gripper: 1
    action_padding: 6

  action_horizon: 32
  action_horizon_flow: 32
  use_state_string_representation: false

dof_config defines the action dimensions predicted by the model (20-dimensional effective dual-arm + 6-dimensional action_padding = 26 dimensions, aligned with the Wall-OSS-0.5 pre-training space), while agent_pos_config defines the observation state dimensions. The sum of all dimensions must match the normalization statistics file. Dimensions such as chassis, lift, and head are not predicted and are padded via action_padding; loss is not backpropagated to the padded portion.

Training hyperparameter example

hyperparams:
  num_epoch: 50
  batch_size_per_gpu: 4
  gradient_accumulation_steps: 4
  seed: 10222
  optimizer:
    optimizer_type: adamw
    learning_rate: 5.0e-05
    max_grad_norm: 1.0
    enable_grad_clip: true
    betas: [0.9, 0.95]
    weight_decay: 1.0e-8
    eps: 1.0e-8
  scheduler:
    scheduler_type: cosine
    num_warmup_steps: 1000
    num_training_steps: 200000
    min_lr: 1.0e-6

distributed:
  use_fsdp: true
  use_mixed_precision: true
  bf16: true

logging:
  log_name: arrange_3_flowers
  log_project: wall_oss_flowers
  log_entity: your_wandb_entity
  use_wandb: true
  log_interval: 10
  save_interval: 2000
  val_interval: 1000000
  epoch_save_interval: 1

debug:
  profile: false
  nvtx: false

The above parameters are from the flower arrangement task example and are not uniform recommended values for all tasks. They should be adjusted based on data scale, GPUs, and task performance, and the configuration actually used should be saved.

Note: data.num_workers > 0 on a network file system may trigger Errno 95; it is recommended to keep num_workers: 0 for the flower arrangement task example.

Generate normalization statistics

python scripts/compute_norm_stats.py \
  --train_config /path/to/arrange_3_flowers_train.yml \
  --data_root /path/to/arrange_3_flowers_lerobot \
  --output_path /path/to/arrange_3_flowers_norm_stats.json

The script computes statistics based on the fields and DOF transformations in the configuration, performing relative pose statistics on the _relative key consistent with the training loader. After generation, confirm that data.norm_stats_path points to the same file, and check that the effective dimension of state/action is 20 (excluding action_padding).

Start training

Smoke test (recommended before full training):

cp workspace/example/arrange_3_flowers_wrc_red_smoke.yml \
  /path/to/arrange_3_flowers_smoke.yml

CUDA_VISIBLE_DEVICES=0 \
torchrun --nproc_per_node=1 \
  wall_x/trainer/fsdp_trainer/train_fsdp.py \
  --config /path/to/arrange_3_flowers_smoke.yml

Or use the DEBUG=1 mode of the startup script:

DEBUG=1 CONFIG=/path/to/arrange_3_flowers_train.yml \
  bash workspace/example/run_oss_wandb_local.sh

Multi-GPU (4-GPU example):

CUDA_VISIBLE_DEVICES=0,1,2,3 \
torchrun --nproc_per_node=4 \
  wall_x/trainer/fsdp_trainer/train_fsdp.py \
  --config /path/to/arrange_3_flowers_train.yml \
  --log_to_file

Unified cluster/local script:

CONFIG=/path/to/arrange_3_flowers_train.yml \
  bash workspace/example/run_oss_wandb_local.sh

Single-GPU training requires at least 48 GB of GPU memory (448 resolution). FSDP is recommended for multi-GPU training. 4 GPUs × Batch 4 × grad_accum 4 = Effective Batch Size 64. Specific GPU memory requirements will be affected by resolution, number of cameras, sequence length, and batch size.

Checkpoints and resuming training

• A single .safetensors file is used to load pre-trained weights (resume_from points to wall-oss-0.5/model.safetensors);

• When fully restoring the optimizer, scheduler, and random state, checkpoint.resume_from should point to the historical checkpoint directory;

• When FSDP generates sharded weights, they need to be merged before inference:

python scripts/merge_sharded_weights.py \
  /path/to/sharded_checkpoint \
  /path/to/merged_checkpoint

Example checkpoint output directory:

/path/to/arrange_3_flowers_ckpt/
├── 3_2000/
├── 3_4000/
├── 3_50000/
│   ├── model.safetensors      # Approx. 17GB
│   ├── config.json
│   ├── config.yml             # Snapshot of training config
│   ├── norm_stats.json
│   └── preprocessor_config.json
└── ...

Check training results

• Whether training and validation loss are abnormal;

• Whether NaN, out of memory (OOM), or data loading errors occur;

• Whether checkpoints, configuration, and normalization files are saved together;

• Whether the cameras, state, and action used for training are consistent with real-robot execution;

• Whether the code version, data version, random seed, and hardware environment are recorded.

Default training parameter reference

These values come from the flower-arrangement task example configuration and are a starting point for reproducing the example, not fixed optimal parameters for all tasks.

Model Evaluation

Model evaluation takes place after training and before continuous real-robot execution, answering two questions:

1. Whether the model has learned the flower arrangement task;

2. Whether the model can run correctly and safely on the target robot.

Evaluation order

1. Checkpoint Loading Check: The model, processor, configuration, and normalization files can be fully loaded.

2. Data Replay / Offline Inference: Run on test episodes to verify that the input and output structures are correct.

3. Open-Loop Evaluation: Compare predicted actions with ground-truth actions in the dataset without executing them on the robot.

4. Input/Output Alignment: Check cameras (three views), states, actions, coordinate frames, units, and frequencies.

5. Inference Performance: Check inference latency, GPU memory usage, throughput, and action horizon (32 steps).

6. Controlled Real-Robot Rollout: Low speed, motion limits, short duration, with emergency stop available at any time; task instruction example: arrange 3 flowers.

7. Result Logging: Record statistics on successes, failures, timeouts, and human interventions.

Open-loop validation

Start the inference service first, then open a new terminal and run:

python scripts/draw_openloop_plot.py \
  --uri ws://127.0.0.1:44660 \
  --dataset-root /path/to/arrange_3_flowers_lerobot \
  --train-config /path/to/arrange_3_flowers_train.yml \
  --episode-indices 0,1,2 \
  --save-dir ./openloop_plots

The results will be saved to ./openloop_plots. The focus of the check is not complete curve overlap, but rather:

• Action dimensions, directions, and ranges are reasonable;

• Key phases such as grasping, moving, and insertion into the vase are broadly consistent with the demonstrations;

• No abnormal spikes, sustained saturation, or significant delays;

• The timing of the dual-arm gripper and end-effector actions is reasonable.

Real-robot evaluation metrics

• Task success rate / task progress: Whether all three flowers are properly placed in the vase, or to which stage it has progressed.

• Completion Time: From task start to success or stoppage.

• Failure Types: Perception, grasping, trajectory, placement, timeout, network, system, etc.

• Human Intervention: Emergency stop, takeover, reset, or manual correction.

• Inference Latency: Latency for a single inference or a single action chunk (reference ~0.2–0.3 s after warmup).

• Control Stability: Jitter, jump, out-of-bounds, and action discontinuity.

The success criterion for the flower-arrangement task can be defined by the number of flowers inserted. At this stage, pipeline acceptance still focuses on whether the data → training → inference → real-robot execution chain is correct, without mandating a uniform success rate threshold for all tasks.

Model Inference and Real-Robot Operation

Code reference: https://github.com/X-Square-Robot/wall-x/blob/main/workspace/rtx5090/DEPLOY.md

Note: Before real-robot inference, move the robot to the initial pose

1. Waist height: 0.35

2. Head: 0.25

This section treats inference and real-robot operation as a single development workflow. The current scope covers starting the inference service, connecting the client, executing actions, and completing controlled validation. It does not constitute a complete production-grade system for containerized deployment, process supervision, monitoring, rollback, and long-term operations.

Start WebSocket inference service

Method A: One-command script (recommended for RTX 5090)

# 1. Configure paths
cp workspace/rtx5090/env.example workspace/rtx5090/local/env.sh
# Edit local/env.sh to set CHECKPOINT_PATH, etc.

# 2. Start
bash workspace/rtx5090/run_server.sh

Method B: General startup command

export HOST=0.0.0.0
export ENABLE_FAST_PREPROCESS=false
export WALLX_VISION_ATTN_IMPLEMENTATION=flash_attention_2
export ENABLE_CUDA_GRAPH=1
export ENABLE_EXPERIMENTAL_ENGINE=1

bash scripts/run_serving.sh \
  --checkpoint-path /path/to/arrange_3_flowers_ckpt/3_50000 \
  --train-config-path /path/to/arrange_3_flowers_ckpt/3_50000/config.yml \
  --port 44660 \
  --cuda-id 0 \
  --robot-type ex001 \
  --serialize-actions \
  --enable-cuda-graph \
  --enable-experimental-engine \
  -- \
  --model-config.norm-key ex_normal

Default connection address:

ws://<Inference_Host_IP>:44660

Health check:

curl http://<Inference_Host_IP>:44660/healthz
# Expected output: OK

--serialize-actions returns serialized actions (follow1_pos, follow2_pos, etc.) that the robot can execute directly. For open-loop evaluation, omit this parameter to use the raw model action chunk.

Advanced startup method:

export ENABLE_CUDA_GRAPH=True
export ENABLE_EXPERIMENTAL_INFERENCE_ENGINE=True
export ENABLE_FAST_PREPROCESS=false

CKPT_PATH=/path/to/arrange_3_flowers_ckpt/3_50000

python -m wall_x._vendor.harrix.serving.launch_serving \
  --env X2ROBOT \
  --host 0.0.0.0 \
  --port 44660 \
  --serialize-actions \
  --enable-cuda-graph \
  --enable-experimental-engine \
  model-config:server-model-config \
  --model-config.checkpoint-path "$CKPT_PATH" \
  --model-config.train-config-path /path/to/arrange_3_flowers_train.yml \
  --model-config.action-horizon 32 \
  --model-config.robot-action-interpolate-multiplier 1 \
  --model-config.robot-action-end-ratio 1.0 \
  --model-config.robot-type ex001 \
  --model-config.norm-key ex_normal

The flower-arranging task uses robot-type ex001 (dual arms + three cameras). These settings must be consistent with the training data, the config.yml in the checkpoint, and the SDK Adapter, and example values from other robot models cannot be directly copied.

Client payload example (ex001 + base64 image)

{
    "state": {
        "follow1_pos": [7-dim float],   # Left arm pose + gripper
        "follow2_pos": [7-dim float],   # Right arm pose + gripper
    },
    "views": {
        "camera_front": "<base64 JPEG>",
        "camera_left": "<base64 JPEG>",
        "camera_right": "<base64 JPEG>",
    },
    "instruction": "arrange 3 flower",
}

After the connection is established, the server first sends metadata (msgpack), then the client sends an observation, and the server returns serialized actions.

Real-robot execution workflow

1. Confirm that the checkpoint, training YAML, and normalization files match;

2. Start the inference service and complete a health check;

3. Read images from the three cameras and the dual-arm state in the SDK Client;

4. Construct an observation consistent with training (camera key names and state dimensions);

5. Send the observation and task text arrange 3 flower;

6. Receive the action chunk;

7. Complete denormalization, dimension mapping, motion limits, and validity checks;

8. Issue actions via the SDK;

   cd samples/quanta_x1
   USE_MAP_NAVIGATION=1 bash scripts/start_sdk_ex001.sh 39.101.65.229 44660 \
     "arrange flower" \
     end_pose 8

   When the terminal displays "Do you want to build new map?", select n as recommended.

9. Continuously check robot state, emergency stop, network, and model latency;

10. Stop upon task completion, timeout, or exception.

Operational safety

• Use only short episodes for the initial run;

• Use low speeds and small motion ranges;

• Do not continue inference when observations are expired or missing;

• Stop immediately in case of inference timeout, disconnection, or invalid actions;

• Do not issue actions when the robot state does not permit control;

• Retain both the physical emergency stop and software stop;

• Record checkpoints, configurations, tasks, environments, and failure causes.

Real-robot deployment checklist

• A real-robot dataset in LeRobot v3 format has been created (three cameras + dual-arm state/action);

• Camera, state, and action field mappings and dimensions are consistent with the training configuration;

• The corresponding normalization statistics file has been generated and linked;

• The fine-tuning checkpoint can be fully loaded (model.safetensors is about 17 GB);

• The WebSocket inference service has been started at ws://<IP>:44660, and /healthz returns OK;

• Open-loop plotting results show no obvious action spikes, directional errors, or continuous saturation;

• The SDK client has motion limits, timeouts, stop-on-disconnect behavior, and emergency-stop handling configured;

• Initial real-robot validation uses low speed, limited motion ranges, and short episodes;

• ENABLE_FAST_PREPROCESS=false has been set on the RTX 5090 inference host.