Trash Picking (OpenPI)

This section uses the ex001 adapter for the OpenPI framework to demonstrate how to start the trash-picking model inference service and use the SDK client to drive the robot to pick up trash from the floor and place it in a waste bin.

Start the Trash Picking Model Service

Install the environment

First download and extract the adapter archive,

openpi-ex001-adapter.zip

Then install the environment from the extracted directory:

cd openpi-ex001-adapter
pip install -U uv
uv venv --python 3.11
uv sync

If network access in mainland China is slow, you can set a mirror before executing uv sync:

export UV_DEFAULT_INDEX=https://mirrors.aliyun.com/pypi/simple.

After installation, run the following command to verify that JAX detects the GPU correctly:

uv run python -c "import jax; print(jax.devices())"

The output should include a GPU device. If only CPU devices are shown, check that the GPU driver and CUDA are installed and detected correctly.

Download weights

uv run huggingface-cli download YangPaiMeng/openpi-ex001-selectwaste \
  --local-dir ./checkpoints/ex001-selectwaste

export CKPT=$PWD/checkpoints/ex001-selectwaste

If huggingface.co is inaccessible, set the mirror first: export HF_ENDPOINT=https://hf-mirror.com.

This repository pins huggingface_hub 0.32.3, which uses the huggingface-cli command. Later versions renamed the command to hf and are not applicable to this repository.

Start the service

unset JAX_PLATFORMS
export CUDA_VISIBLE_DEVICES=0
export XLA_PYTHON_CLIENT_MEM_FRACTION=0.35

uv run scripts/serve_ex001.py \
  --checkpoint "$CKPT" \
  --tokenizer "$CKPT/paligemma_tokenizer.model" \
  --port 44604

The service takes approximately 20 to 40 seconds to warm up. When the following log appears in the terminal, the service is ready and you can proceed:

[INFO] serving on 0.0.0.0:44604

Logs such as "checkpoint has 4 params ..." and "Token length (202) exceeds ..." appearing during startup are normal and can be ignored.

If the robot hardware configuration is not DVT with a G gripper, explicitly specify the corresponding hardware-configuration instruction with the --force-prompt parameter. The model was trained on only the following three instructions, which correspond to three hardware configurations rather than task stages:

Select the waste from the floor and put it into the waste bin by EVT robot.
Select the waste from the floor and put it into the waste bin by DVT robot and H gripper.
Select the waste from the floor and put it into the waste bin by DVT robot and G gripper.

By default, the service forces the third instruction, which has the largest amount of training data and the best inference results, and ignores instructions sent by the client. When changing the hardware configuration, pass the corresponding instruction exactly as written, for example:

--force-prompt "Select the waste from the floor and put it into the waste bin by EVT robot."

Self-check (without robot)

Run the following command in another terminal to check inference-service connectivity and output:

cd openpi-ex001-adapter
export CKPT=$PWD/checkpoints/ex001-selectwaste

uv run examples/ex001/demo_client.py --port 44604 --norm-stats "$CKPT/norm_stats.json"

If the terminal displays the following output, the connection to the service is successful:

OK: every stream has the expected shape and the chunk starts at the current state.

In addition, infer_ms in the timing field should be on the order of 100 ms. If it reaches the level of seconds, inference is usually running on the CPU; check that unset JAX_PLATFORMS was executed correctly when starting the service.

Inference via SDK Client

Work Area Preparation

Place the robot in an open, level area with no debris on the floor.
Place the waste bin and the trash to be picked up in front of the robot.
Confirm that sufficient space is reserved around the robot for movement and robotic arm operation, and ensure that the emergency stop is accessible at all times.

Network preparation

According to the "Local Area Network Configuration Guide", connect the development PC to the robot using an Ethernet cable, and configure the robot-side wired interface of the development PC to 192.168.10.10/24. The example in this document uses the robot SDK Server address 192.168.10.1:50051.

The development PC must also be able to access the model inference service. If using a dual-NIC setup — wired connection to the robot plus Wi-Fi/another network for the model service — ensure that the robot network segment 192.168.10.0/24 is still accessed via the robot's wired interface and that the model WebSocket address is reachable.

Software environment

Follow the SDK development section to install and configure the sdk_robot repository and its environment.

Run the Example

cd /path/to/sdk_robot

INFERENCE_SERVER_IP='REPLACE_WITH_CONFIRMED_IP_OR_HOST'
INFERENCE_SERVER_PORT='REPLACE_WITH_CONFIRMED_PORT'

bash samples/quanta_x1/scripts/start_sdk_ex001.sh \
  "$INFERENCE_SERVER_IP" \
  "$INFERENCE_SERVER_PORT" \
  "Select the waste from the floor and put it into the waste bin by DVT robot and G gripper." \
  end_pose \
  6 \
  192.168.10.1:50051

If the robot uses another IP address, explicitly specify the actual robot IP:50051 via the sixth parameter ROBOT_SDK_URL without directly modifying the start_sdk_ex001.sh source file.

Running instructions

The script connects to both the inference-service WebSocket and the robot SDK Server.
If the network disconnects, inference times out, or an SDK call fails, first confirm that the robot has stopped safely, then troubleshoot the inference service, robot status, and network connection.
Rerunning start_sdk_ex001.sh starts a new client session. The current sample does not persist or restore inference state, so it cannot guarantee resumption from the point of interruption.