DexUMI
real-stanford
A wearable interface that records dexterous demonstrations and adapts their visual appearance for robot learning.
What you can explore
Explore exoskeleton capture, force sensing, visual hand replacement and the documented XHand and Inspire examples.
Before you use it
The system needs a calibrated exoskeleton, sensors and capture hardware. Recording human motion alone does not produce a deployable robot policy.
Original contributors
Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso and Shuran Song
License / access: Code: MIT · data, models and dependencies may have separate terms
Capture, calibration and processing
DexUMI: understand capture, replay and data generation
- Environment
- Ubuntu 22.04 in the authors’ documented setup
- Embodiment
- Documented XHand and Inspire exoskeleton configurations
What you need
Follow environment.yml and the linked hardware guide. The processing pipeline uses SAM2, ProPainter and Record3D; it is more than a single Python package. Confirm the matching hand configuration and capture hardware first.
First useful check
Inspect the provided sample data and its generation workflow before building a complete capture rig.
A small first experiment
Check the episode stage before processing
Save the read-only checker below as check_dexumi_episode.py. Run python check_dexumi_episode.py replay /path/to/extracted/episode for a replay sample, or choose capture or training for those stages. Use the episode folder, not the archive or dataset parent. Exit 0 means the selected stage’s listed paths exist; exit 1 lists missing paths. It reads no sensor values and runs no robot commands. Missing paths may mean the wrong stage or an incomplete extraction. It does not infer the stage automatically or require force data.
Official episode layouts and sample download →from pathlib import Path import argparse parser = argparse.ArgumentParser() parser.add_argument("stage", choices=("capture", "replay", "training")) parser.add_argument("episode", type=Path) args = parser.parse_args() if not args.episode.is_dir(): parser.error("Expected an extracted episode directory") required = { "capture": ("camera_0.mp4", "camera_1.mp4", "numeric_0", "numeric_1", "numeric_2", "numeric_3"), "replay": ("dex_camera_0.mp4", "exo_camera_0.mp4", "hand_motor_value", "joint_angles_interp", "pose_interp", "valid_indices"), "training": ("camera_0", "hand_action", "pose", "proprioception"), } missing = [] for name in required[args.stage]: present = (args.episode / name).exists() print(("FOUND " if present else "MISSING ") + name) if not present: missing.append(name) print("Presence check only: not synchronization or data-quality validation") raise SystemExit(1 if missing else 0)Inspect the batch script without running hardware
Save this as inspect_dexumi.py in the DexUMI repository root, then run python inspect_dexumi.py. It prints only the settings and stages that need review; it does not run the shell script, import a robot driver or connect to hardware. At the reviewed revision, process.sh spans episodes 0–99, enables FSR processing and selects xhand. Review all three choices for your rig before commissioning one episode.
Reviewed batch-script defaults →from pathlib import Path pipeline = Path("real_script/data_generation_pipeline/process.sh") if not pipeline.is_file(): raise SystemExit("Run from the DexUMI repository root") markers = ("DATA_DIR=", "TARGET_DIR=", "REFERENCE_DIR=", "for i in", "0_interpolation.py", "1_replay_hand.py") for number, line in enumerate(pipeline.read_text().splitlines(), 1): text = line.strip() if not text.startswith("#") and any(m in text for m in markers): print(f"{number}: {text}") print("Review only: no pipeline or robot command was executed")Distinguish replay from image processing
The documented process.sh includes robot-hand replay; do not treat it as an offline-only command. Review the stages before execution. After replay data exists, inspect segmentation, inpainting and the final composite for hand/object boundary errors.
Capture and processing walkthrough →Validate alignment before scaling collection
Check synchronization, encoder-to-motor mapping and wrist-camera/end-effector frames for the exact configuration. Force sensing is optional in the recording instructions; do not label absent force channels as measurements.
Audit a demonstration episode →
What to record
- Save a one-episode manifest, configuration and commit, with units and frame definitions.
- Record visible processing artifacts, missing frames and calibration limitations before adding more episodes.
- For one episode, review first/last timestamps, frame alignment, motor units, object occlusion and compositing artifacts. Preserve a failure clip, not only a successful composite.
Common problems and what to check
- Replay files are missing from the sample
- Do not create empty placeholders to pass this checker. Confirm the downloaded archive is extracted and point to a single episode. A capture episode must go through the documented replay stage before it has replay outputs; process.sh can move hardware. A successful path check still needs timestamp, array-shape and video decoding checks.
- Human motion is treated as robot actions
- The embodiment mapping and calibration must be explicit before using recordings as control labels.
- A clean composite is mistaken for a good policy
- Image appearance and temporal alignment require separate checks; task performance still needs a trained-policy evaluation.
- Inspire data is processed as XHand
- Set the hand choice consistently in replay and final dataset generation. The reviewed batch script hardcodes xhand; the final generator also defaults to xhand. The capture flag alone does not configure later stages.
- Final generation rejects its force option
- At the reviewed revision, --force-process declares flatten/total but its default is xhand. Inspect the parser and pass an allowed mode explicitly when force data is present. Do not manufacture a force channel when your recording has none.
Where it stops
Processed demonstrations are training inputs, not a trained policy. New embodiments require their own mapping, calibration and validation. Published XHand/Inspire examples do not establish Prima 1 or another hand’s compatibility.
Keep exploring
DexWild
Portable human demonstrations and robot data combined to train dexterous manipulation policies.
DexCap
Hand motion capture and scene observations for learning dexterous manipulation from human demonstrations.
Dex Retargeting
Software for mapping human hand motion to the joint configurations of robotic hands.
Editorial reference · checked 2026-10-24