DEXTEROUS MANIPULATION

The world of dexterous hands.

Compare hardware. Discover applications. Share what works. An open reference and community for dexterous manipulation.

DexUMI

real-stanford

A wearable interface that records dexterous demonstrations and adapts their visual appearance for robot learning.

What you can explore

Explore exoskeleton capture, force sensing, visual hand replacement and the documented XHand and Inspire examples.

Before you use it

The system needs a calibrated exoskeleton, sensors and capture hardware. Recording human motion alone does not produce a deployable robot policy.

Original contributors

Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso and Shuran Song

License / access: Code: MIT · data, models and dependencies may have separate terms

Original code & documentation

Hardware assembly guide

Capture, calibration and processing

FROM REFERENCE TO FIRST EXPERIMENT

DexUMI: understand capture, replay and data generation

INPUTWearable demonstrations, camera streams and encoder readings
OUTPUTProcessed robot-hand observations and aligned action data
Environment
Ubuntu 22.04 in the authors’ documented setup
Embodiment
Documented XHand and Inspire exoskeleton configurations

What you need

Follow environment.yml and the linked hardware guide. The processing pipeline uses SAM2, ProPainter and Record3D; it is more than a single Python package. Confirm the matching hand configuration and capture hardware first.

First useful check

Inspect the provided sample data and its generation workflow before building a complete capture rig.

A small first experiment

  1. Check the episode stage before processing

    Save the read-only checker below as check_dexumi_episode.py. Run python check_dexumi_episode.py replay /path/to/extracted/episode for a replay sample, or choose capture or training for those stages. Use the episode folder, not the archive or dataset parent. Exit 0 means the selected stage’s listed paths exist; exit 1 lists missing paths. It reads no sensor values and runs no robot commands. Missing paths may mean the wrong stage or an incomplete extraction. It does not infer the stage automatically or require force data.

    from pathlib import Path
    import argparse
    
    parser = argparse.ArgumentParser()
    parser.add_argument("stage", choices=("capture", "replay", "training"))
    parser.add_argument("episode", type=Path)
    args = parser.parse_args()
    if not args.episode.is_dir():
        parser.error("Expected an extracted episode directory")
    required = {
        "capture": ("camera_0.mp4", "camera_1.mp4", "numeric_0", "numeric_1", "numeric_2", "numeric_3"),
        "replay": ("dex_camera_0.mp4", "exo_camera_0.mp4", "hand_motor_value", "joint_angles_interp", "pose_interp", "valid_indices"),
        "training": ("camera_0", "hand_action", "pose", "proprioception"),
    }
    missing = []
    for name in required[args.stage]:
        present = (args.episode / name).exists()
        print(("FOUND   " if present else "MISSING ") + name)
        if not present:
            missing.append(name)
    print("Presence check only: not synchronization or data-quality validation")
    raise SystemExit(1 if missing else 0)
    Official episode layouts and sample download →
  2. Inspect the batch script without running hardware

    Save this as inspect_dexumi.py in the DexUMI repository root, then run python inspect_dexumi.py. It prints only the settings and stages that need review; it does not run the shell script, import a robot driver or connect to hardware. At the reviewed revision, process.sh spans episodes 0–99, enables FSR processing and selects xhand. Review all three choices for your rig before commissioning one episode.

    from pathlib import Path
    
    pipeline = Path("real_script/data_generation_pipeline/process.sh")
    if not pipeline.is_file():
        raise SystemExit("Run from the DexUMI repository root")
    markers = ("DATA_DIR=", "TARGET_DIR=", "REFERENCE_DIR=",
               "for i in", "0_interpolation.py", "1_replay_hand.py")
    for number, line in enumerate(pipeline.read_text().splitlines(), 1):
        text = line.strip()
        if not text.startswith("#") and any(m in text for m in markers):
            print(f"{number}: {text}")
    print("Review only: no pipeline or robot command was executed")
    Reviewed batch-script defaults →
  3. Distinguish replay from image processing

    The documented process.sh includes robot-hand replay; do not treat it as an offline-only command. Review the stages before execution. After replay data exists, inspect segmentation, inpainting and the final composite for hand/object boundary errors.

    Capture and processing walkthrough →
  4. Validate alignment before scaling collection

    Check synchronization, encoder-to-motor mapping and wrist-camera/end-effector frames for the exact configuration. Force sensing is optional in the recording instructions; do not label absent force channels as measurements.

    Audit a demonstration episode →

What to record

  • Save a one-episode manifest, configuration and commit, with units and frame definitions.
  • Record visible processing artifacts, missing frames and calibration limitations before adding more episodes.
  • For one episode, review first/last timestamps, frame alignment, motor units, object occlusion and compositing artifacts. Preserve a failure clip, not only a successful composite.
Common problems and what to check
Replay files are missing from the sample
Do not create empty placeholders to pass this checker. Confirm the downloaded archive is extracted and point to a single episode. A capture episode must go through the documented replay stage before it has replay outputs; process.sh can move hardware. A successful path check still needs timestamp, array-shape and video decoding checks.
Human motion is treated as robot actions
The embodiment mapping and calibration must be explicit before using recordings as control labels.
A clean composite is mistaken for a good policy
Image appearance and temporal alignment require separate checks; task performance still needs a trained-policy evaluation.
Inspire data is processed as XHand
Set the hand choice consistently in replay and final dataset generation. The reviewed batch script hardcodes xhand; the final generator also defaults to xhand. The capture flag alone does not configure later stages.
Final generation rejects its force option
At the reviewed revision, --force-process declares flatten/total but its default is xhand. Inspect the parser and pass an allowed mode explicitly when force data is present. Do not manufacture a force channel when your recording has none.

Where it stops

Processed demonstrations are training inputs, not a trained policy. New embodiments require their own mapping, calibration and validation. Published XHand/Inspire examples do not establish Prima 1 or another hand’s compatibility.

Public documentation reviewed; not independently reproduced · reviewed 2026-10-27. Record the exact source revision when reproducing it.

Keep exploring

DexWild

Portable human demonstrations and robot data combined to train dexterous manipulation policies.

DexCap

Hand motion capture and scene observations for learning dexterous manipulation from human demonstrations.

Dex Retargeting

Software for mapping human hand motion to the joint configurations of robotic hands.

Editorial reference · checked 2026-10-24

Start a discussion

Published posts are public. First contributions are reviewed. Only share material you have permission to disclose.

Report a post