If our project helps you, please give us a star ⭐ on GitHub to support us.

Spotter adds a lightweight intervention layer between a frozen embodied model (System 1) and a VLM (System 2). Instead of having the VLM plan every step, the embodied model leads and keeps executing, while a local screener watches in parallel and the VLM steps in only to confirm and repair an error. The VLM's job becomes much simpler, so the whole framework works with a locally served, open-source Qwen and no privileged information.

Real robot (Franka Research 3), stacking the blue cup onto the pink cup. Shown at 2×, with the judge's thinking time shortened; the on-screen clock is real time.
Watch the full video on the project page.

• A lightweight intervention layer. Like a person who keeps a little attention on a task and focuses only when something looks wrong, Spotter lets execution flow continuously and brings in the VLM only on a confirmed error.
• A simpler job for the VLM. The VLM no longer drives the robot. It only has to notice a failure, repair it with a short primitive plan, and check the fix. A local Qwen with no access to the task's success signal is enough.
• Continuous execution, low overhead. The embodied model never waits for the VLM on the routine path: a successful episode takes only 13–16 s longer than the policy alone.

Existing VLM-led robot agents are serial: the VLM plans a segment, the embodied model executes it, and the VLM decides what comes next. This forces a dilemma:

• Short segments mean frequent VLM calls and high latency, even on easy tasks.
• Long segments mean a deviation during execution cannot be caught in time, and later actions keep going wrong.

Spotter removes this trade-off by supervising in parallel rather than in series.

With the same local Qwen and no privileged information on RoboCasa (1,200 episodes), a representative VLM-led agent falls below the policy alone (Cosmos Policy 67.9 → 65.2, π0.5 64.3 → 60.5), while Spotter still improves it (→ 71.7 and 68.4).

With Qwen, the judge is called less than once per successful episode on average, which adds only 13–16 s over the policy alone and takes about 70% less time than a VLM-led agent using the same Qwen.

Simulators. Spotter runs on both RoboCasa and RoboTwin 2.0. RoboTwin 2.0 has a one-command evaluation, bash robotwin_eval.sh (see Evaluation).

RoboTwin 2.0 results and full logs (every episode's frames, prompts and replies): 50 tasks × 10 episodes (policy alone vs. Spotter-Qwen) and 150 episodes (policy alone, Spotter and a VLM-led agent, each with Qwen and GPT).

• File Structure
• Environment Setup
• Model Preparation
• Simulator Assets and Episode Sets
• Evaluation with Released Checkpoints: RoboCasa · RoboTwin 2.0
• Reproducing the Paper
• Building the Lesson Library and Example Bank
• Plugging In Your Own Embodied Model
• Outputs · Configuration Reference
• Acknowledgements · License · Contact · Citation

Spotter/
├── spotter.sh                        # one entry point: serve Qwen / the policy, run episodes, summarize
├── robotwin_eval.sh                  # one-command RoboTwin 2.0 evaluation (servers + Spotter arm + control + summary)
├── recovery_explore/                 # Spotter itself
│   ├── judge_driver_v7.sh            # supervision loop for one lane: screen -> judge -> repair -> verify
│   ├── env_service.py                # RoboCasa simulation service (RPC), started by launch_svc.sh
│   ├── env_service_robotwin.py       # RoboTwin 2.0 simulation service (same RPC surface, SAPIEN)
│   ├── robotwin_intervention.py      # RoboTwin repair primitives (dual arm); robodojo_intervention.py is its base
│   ├── robotwin_policy_server.py     # RoboTwin pi0.5 policy server (openpi websocket protocol)
│   ├── primitives.py                 # motion primitives the judge composes repairs from
│   ├── grasp_aim.py                  # RoboCasa `aim` step: finds the object in the wrist depth image and plans the grasp
│   ├── render_guard.py               # RoboCasa: re-renders black / noise frames from a leaked EGL context
│   ├── JUDGE_BRIEF_V7.md             # judge prompt; BRIEF_PNP.md / BRIEF_MECH.md / BRIEF_RT.md add task-family notes
│   ├── cli/                          # screener, judge channels (Qwen / GPT), harness, few-shot tools
│   ├── memory_bank/                  # frozen library of 77 general lessons (RoboCasa)
│   ├── memory_bank_robotwin_noreset/ # the RoboTwin 2.0 lessons
│   ├── episode_sets/                 # sall500.txt (paper, 1,200) · s500q96.txt (96, debugging) · rt500.txt / rt150.txt (RoboTwin 2.0)
│   ├── fewshot_bank/                 # (downloaded) worked examples for the 1-shot setting; robotwin/ for RoboTwin 2.0
│   └── runs_<RUN>_<family>/          # (generated) results.jsonl, per-lane logs, frames shown to the judge
├── rpc/                              # π0.5 policy server, RPC protocol, RoboCasa rollout and collection scripts
├── cf_bench/                         # snapshot and replay tools for the error-correction study
├── env/                              # env.sh.example, requirements-robocasa.txt, requirements-pi05.txt
├── assets/                           # logo
└── LICENSE · THIRD_PARTY_NOTICES.md · LICENSES/

Qwen screener and judge. vLLM in its own environment, and a local env/env.sh:

python3 -m venv .venvs/vllm && .venvs/vllm/bin/pip install vllm
cp env/env.sh.example env/env.sh && $EDITOR env/env.sh

RoboCasa. Spotter uses three Python environments for RoboCasa:

# 1) RoboCasa + Cosmos Policy: follow the RoboCasa setup of https://github.com/NVlabs/cosmos-policy, then
pip install -r env/requirements-robocasa.txt

# 2) π0.5
python3.12 -m venv .venvs/pi05 && .venvs/pi05/bin/pip install -r env/requirements-pi05.txt

Note. Headless MuJoCo rendering needs an EGL-capable NVIDIA GPU (MUJOCO_GL=egl, set in env/env.sh).

RoboTwin 2.0. Spotter runs on RoboTwin 2.0 (aloha-agilex dual arm, SAPIEN 3, 50 tasks, demo_randomized scenes); the family is robotwin.

RoboTwin's simulator and planner (SAPIEN 3, mplib, cuRobo) live in their own Python 3.10 environment, laid out as a portable runtime that env/env.sh points at:

<ROBOTWIN_RUNTIME>/
├── .venv/                                   # python 3.10: sapien 3.0.0b1, mplib 0.2.1, toppra, torch (CUDA build matching your GPU)
└── worktree/third_party/RoboTwin/           # the RoboTwin 2.0 tree
    ├── envs/curobo/src/                     # cuRobo sources (its CUDA kernels must match your torch)
    ├── assets/{objects,background_texture,embodiments}/   # RoboTwin 2.0 assets (TianxingChen/RoboTwin2.0 on the Hub)
    └── policy/pi0/checkpoints/pi05_robotwin2_clean_randomized/robotwin2/40000/   # the pi0.5 checkpoint

Install RoboTwin 2.0 and its assets as its README says, then set in env/env.sh:

export ROBOTWIN_RUNTIME=/path/to/robotwin-runtime            # or ROBOTWIN_ROOT + ROBOTWIN_PY
export ROBOTWIN_POLICY_PYTHON=/path/to/robotwin-policy-venv/bin/python   # a venv satisfying policy/pi05/requirements.txt (openpi, lerobot < 0.3)

The simulation renders with SAPIEN's ray tracer (Vulkan, RTX-class GPU); on RTX 5090 the optix denoiser is used because the default oidn fails there.

1) Qwen judge and screener.

huggingface-cli download Qwen/Qwen3.8-27B-FP8 --local-dir checkpoints/Qwen3.8-27B-FP8

Then set in env/env.sh:

export QWEN_MODEL=$PWD/checkpoints/Qwen3.8-27B-FP8
export VLLM_VENV=$PWD/.venvs/vllm

2) RoboCasa. Download π0.5 and Cosmos Policy:

huggingface-cli download DAVIAN-Robotics/pi05-robocasa-H50         --local-dir checkpoints/pi05-robocasa-H50
huggingface-cli download nvidia/Cosmos-Policy-RoboCasa-Predict2-2B --local-dir checkpoints/Cosmos-Policy-RoboCasa-Predict2-2B

Expected layout:

checkpoints/
├── pi05-robocasa-H50/                        # model.safetensors, config.json, policy_*processor*, tokenizer/
├── Cosmos-Policy-RoboCasa-Predict2-2B/       # Cosmos-Policy-RoboCasa-Predict2-2B.pt, robocasa_t5_embeddings.pkl, ...
└── Qwen3.8-27B-FP8/

Then set in env/env.sh:

export PI05_CHECKPOINT=$PWD/checkpoints/pi05-robocasa-H50
export COSMOS_CHECKPOINT=$PWD/checkpoints/Cosmos-Policy-RoboCasa-Predict2-2B/Cosmos-Policy-RoboCasa-Predict2-2B.pt

For the 1-shot setting, download the RoboCasa example bank (151 MB):

curl -L https://github.com/zqc3117/Spotter/releases/download/v1.0/fewshot_bank.tar.gz | tar -xz -C recovery_explore

3) RoboTwin 2.0. The policy is the pi0.5 checkpoint motus-robotics/pi0.5_robotwin2, placed at policy/pi0/checkpoints/pi05_robotwin2_clean_randomized/robotwin2/40000/ inside the RoboTwin tree (see Environment Setup). For the 1-shot setting, download the RoboTwin example bank (33 MB) into recovery_explore/fewshot_bank/robotwin/:

curl -L https://github.com/zqc3117/Spotter/releases/download/v1.0/fewshot_bank_robotwin.tar.gz | tar -xz -C recovery_explore

GPT judge (optional). Set ENGINE=api together with OPENAI_RESPONSES_URL and OPENAI_API_TOKEN_FILE; the judge becomes GPT-6 Astra while the screener stays the local Qwen.

RoboCasa assets are installed by the Cosmos Policy RoboCasa setup above.

RoboTwin 2.0 assets are installed with RoboTwin itself (Environment Setup).

Episode sets ship with the repository, one TASK:SEED:EPISODE per line:

RoboCasa lessons and worked examples were learned on seed 195, disjoint from the test seed.

Each server runs in its own terminal after source env/env.sh.

1) Serve the Qwen screener and judge (one replica per GPU, ports 8301, 8302, ...).

bash spotter.sh judge --gpus 0,1,2,3
curl -s http://127.0.0.1:8301/v1/models        # check

With fewer GPUs, pass fewer ids (e.g. --gpus 0) and export QWEN_PORTS=8301 before running.

2) Serve the embodied model: π0.5 on port 8900 or Cosmos Policy on port 8800.

bash spotter.sh policy pi05 --gpu 4             # or: bash spotter.sh policy cosmos --gpu 4
curl -s http://127.0.0.1:8900/health           # check

3) Smoke test: one episode. The driver starts the simulation service by itself.

bash spotter.sh run pi05                        # one episode, control and Spotter on the same scene

4) Full evaluation on sall500.

bash spotter.sh run pi05 sall500 --lanes 8
bash spotter.sh summary

mkdir -p recovery_explore/runs_smoke
printf 'PnPCabToCounter:195:0\n' > recovery_explore/runs_smoke/episodes_pi05.txt
ON_POD=1 RUN=smoke TARGET=1 NGPU=1 FAMILY_OVERRIDE=pi05 \
SIM_HOST=127.0.0.1 SIM_MANAGED=0 POLICY_URL=http://127.0.0.1:8900 \
ENGINE=qwen MODEL=qwen38 SCREEN=1 QWEN_KEEP_TURNS=20 \
STEP_BUDGET_SCALE=1.8 WINDOW=1 TEL_CHUNKS=10 COMPACT_AT=110000 MAX_INTERVENTIONS=8 \
ALLOW_RESET=0 MAX_RESETS=0 FEWSHOT=0 LEARN=0 CTRL_ONLY=0 SKIP_CONTROL=0 \
  bash recovery_explore/judge_driver_v7.sh 0

One command. robotwin_eval.sh starts Qwen and the pi0.5 policy if they are not already up, runs the Spotter arm and then the policy-only control on the same 500 episodes (rt500), and prints the summary. It uses the settings of the released results: 1.8× step budget, 1-shot, Qwen thinking in the repair turns with QWEN_THINK_BUDGET=5000.

bash robotwin_eval.sh --judge-gpus 0,1,2,3 --policy-gpu 4 --lanes 12

Add --dry-run to print the commands without running them. Each RoboTwin simulator takes about 8 GB of GPU memory, so plan about three lanes per 32 GB GPU. The judge's thinking dominates the time of the Spotter arm; more Qwen replicas (--judge-gpus) make it faster.

bash spotter.sh policy robotwin --gpu 4              # pi0.5 for RoboTwin, port 9100 (check: curl http://127.0.0.1:9100/healthz)
bash spotter.sh run robotwin                         # smoke: adjust_bottle, episode 35
bash spotter.sh run robotwin rt500 --lanes 12 --one-shot     # Spotter arm on the 500 evaluation episodes (1-shot)
bash spotter.sh run robotwin rt500 --lanes 12 --arm ctrl     # the policy-only baseline on the same episodes
bash spotter.sh summary

Results with this code. We re-ran the Spotter arms on RoboTwin 2.0 with this version of the code, on the 500 episodes of rt500 (1.8× step budget, 1-shot); the control is the policy alone on the same 500 episodes:

Because every run involves randomness, these numbers differ somewhat from the paper. We have uploaded the per-episode results, with every episode's frames, judge prompts and replies, to ModelScope: 50 tasks × 10 episodes.

Comparison with a VLM-led agent (150 episodes). On rt150 (50 tasks × 3 episodes, a subset of rt500), with the same settings (1.8× step budget, 1-shot, Qwen thinking in the repair turns up to 5000 tokens), we also ran Spotter with GPT and a VLM-led agent with both LLMs; every arm runs on the same episodes. The VLM-led agent drives the episode turn by turn and calls the policy as a tool; its code is not part of this release.

The complete records of all five arms, with every episode's frames, prompts and replies, are on ModelScope: 150 episodes.

bash spotter.sh run <family> [SET] [options], for both simulators:

Both arms of an episode share the same scene, instruction and step budget:

spotter.sh run <family> applies that family's column of the Configuration Reference; every setting can still be overridden through the environment.

The Spotter rows of Table 1 (RoboCasa, 1,200 episodes) are the full evaluation above with these settings; variables are exported before spotter.sh run.

The fixed-retry rows of Table 1 are not included in this release.

Supervision mode. spotter.sh runs the synchronous supervision loop (ASYNC_SCREEN=0). For the bounded asynchronous mode of Section 3.2, set ASYNC_SCREEN=2 ASYNC_K=2: the policy runs at most two supervision windows ahead of the last check (4 chunks on Cosmos Policy, 2 on π0.5).

Time. results.jsonl logs the wall-clock time of each arm and of the judge and screener calls (see Outputs); token counts are not logged by this release.

Section 3.1. rpc/run_remote_robocasa_cf_branch_collect.py collects the post-failure and normal candidates, and cf_bench/ replays them and runs the language-feedback protocol. The test-time scorers (Consistency-Consensus, GeoBoN, RCS) are not included.

The released library (77 general lessons in recovery_explore/memory_bank/global/) and the example bank are frozen and used as-is for every paper result; this section is only needed to rebuild them or to cover new tasks.

1) Run learning episodes on seed 195, never on the test seed. Only here may the judge reset and retry; each episode writes its lessons as drafts to memory_bank/inbox/<cell>/.

ENGINE=api LEARN=1 ALLOW_RESET=1 MAX_RESETS=5 bash spotter.sh run cosmos <seed-195 episode file> --run learn_cosmos

2) Review the drafts and move the general ones into memory_bank/global/, following the lesson format in recovery_explore/memory_bank/README.md. Remove anything privileged or task-specific.

3) Build the example bank for the 1-shot setting from the learning runs:

python3 recovery_explore/cli/fewshot_build.py --family cosmos --seed 195 --out recovery_explore/fewshot_bank

Spotter never trains the policy, so a new embodied model only needs to be served and registered:

1. Serve it over the same RPC interface as rpc/policy_server_pi05.py: GET /health, and POST /v1/infer taking an InferenceRequest (task description, seed, observation images and state) and returning an InferenceResponse with an action chunk (see rpc/protocol.py).
2. Register the family next to cosmos and pi05: action dimension and steps per chunk in recovery_explore/cosmos_call.py and env_service.py, service launch in launch_svc.sh, port and defaults in spotter.sh.
3. Choose the supervision granularity: WINDOW (chunks per screening check) and TEL_CHUNKS (telemetry rows shown to the judge).
4. Run bash spotter.sh run <your-family> ....

recovery_explore/runs_<RUN>_<family>/
├── results.jsonl          # one line per episode
├── lane<N>/driver.log     # per-lane progress
└── lane<N>/jd-.../w<k>/   # frames shown to the judge in window k

bash spotter.sh summary            # all runs
bash spotter.sh summary pi05_sall500
bash spotter.sh summary robotwin_rt500

spotter.sh run <family> applies the matching column, and robotwin_eval.sh the RoboTwin 2.0 one with --one-shot, as in the released results; every value can be overridden through the environment. The parameters below are shared; their paper values refer to RoboCasa, and the per-family values above take precedence.

Qwen thinks only in the repair turns of an intervention, up to QWEN_THINK_BUDGET (default 5000) tokens per call; the screener and the window verdict never think, and QWEN_REPAIR_THINK=0 turns thinking off everywhere, as in the paper.

Choosing the thinking budget. Set QWEN_THINK_BUDGET to what your Qwen servers can afford. A larger budget gives the repair plan more room but makes every repair turn slower; a smaller one is faster but cuts the thinking off before it finishes. Use the largest budget that keeps repair turns within ACT_DEADLINE (raise that too on slow GPUs or with many lanes per replica), and check after a run that most repair turns were not cut off. Each repair reply is saved as act_w*_a*_r*.json in the episode folder, with think_budget_hit: true when the thinking ran into the budget:

python3 - <<'EOF'
import glob, json
files = glob.glob("recovery_explore/runs_<RUN>_<family>/lane*/jd-*/act_w*.json")
hit = sum(bool(json.load(open(f)).get("think_budget_hit")) for f in files)
print(f"{hit}/{len(files)} repair turns hit the thinking budget")
EOF

If more than about half are cut, raise the budget (or add Qwen replicas so the larger budget stays fast). On RoboTwin 2.0, a budget of 2000 cut off the thinking in about 65% of the repair turns, and 5000 in about 44%.

How the judge is paced has its own switches, all off or conservative by default:

RoboCasa repair steps. Besides the motion primitives, the RoboCasa judge has an aim step: the robot looks down with its wrist depth camera, finds the object near the last missed grasp (or near a point the judge names), grasps it across its short side, lifts slightly to check the hold, and reports held, closed_empty, slipped, no_object, too_wide, blocked or step_cap. A hand-back right after a held grasp keeps the gripper closed ({"op": "policy", "hold": true} does the same explicitly). Pixel targets are [row, col] normalized to 0–1000 within one camera image (PIXEL_FRAME=1000; 512 takes raw pixels), and Qwen gets each camera as its own image for pointing (QWEN_POINT_TILES=1). Everything the judge uses comes from the cameras and the robot's own joint and gripper readings.

Changing STEP_BUDGET_SCALE, WINDOW or the policy family changes the experiment. Compare runs only on the same episode set; because the policies' rollouts are not bit-reproducible, compare aggregate success rates rather than single episodes.

• RoboCasa: the simulation benchmark and assets we evaluate on.
• Cosmos Policy: the released RoboCasa checkpoint and its policy server.
• π0.5 and LeRobot: the RoboCasa π0.5 checkpoint and the library that loads it.
• RoboTwin 2.0 and pi0.5 for RoboTwin 2.0: the dual-arm benchmark and the policy checkpoint we evaluate with.
• Qwen and vLLM: the local screener and judge.
• RPent: parts of recovery_explore/ are adapted from it (Apache-2.0, see THIRD_PARTY_NOTICES.md).

This project is released under the MIT License. Some files in recovery_explore/ are adapted from RPent and remain under the Apache License 2.0; see THIRD_PARTY_NOTICES.md.

Have a question, want to share results, or just follow development? Join us:

WeChat Group Scan to join

Discord discord.gg/dcsEJHE4Pk

You can also ask questions and share ideas in GitHub Discussions, report bugs in Issues, or email Qichao Zhao at [email protected].

@article{li2026spotter,
  title   = {Spotter: Let the Embodied Model Lead, and the VLM Reflect for It},
  author  = {Li, Long and Zhao, Qichao and Yang, Yue and Xu, Fan and Wang, Zhe and Liew, Alan Wee-Chung and Qu, Chao and Shen, Heng Tao and Pan, Shirui},
  journal = {arXiv preprint arXiv:2609.36808},
  year    = {2026}
}