DEXTEROUS MANIPULATION

The world of dexterous hands.

Compare hardware. Discover applications. Share what works. An open reference and community for dexterous manipulation.

DexMV

DexMV research team

Simulation and retargeting tools for learning dexterous manipulation from human videos.

What you can explore

Explore demonstration processing and simulated relocation, pouring and placement tasks.

Before you use it

This is the simulation/retargeting repository; the learning code is separate. Its documented MuJoCo environment is an older release.

Original contributors

Yuzhe Qin, Yueh-Hua Wu and collaborators

License / access: Apache-2.0 · check model, data and dependency terms separately

Original code & documentation

FROM REFERENCE TO FIRST EXPERIMENT

DexMV: from human poses to a checked simulation demonstration

INPUTEstimated human hand/object poses or author-provided processed demonstrations
OUTPUTRetargeted robot poses, translated demonstrations and learned simulation policies
Environment
Legacy MuJoCo 2.0 / mujoco-py; separate simulation and learning repositories
Embodiment
Released Adroit simulation tasks: relocate, pour and place inside

What you need

DexMV separates dexmv-sim from dexmv-learn, including its mjrl subdirectory. The released environment.yml is a Linux/Python 3.7-era snapshot with mujoco-py 2.0.2.13. Treat it as a legacy reproduction environment; it is not a tested Apple Silicon or current MuJoCo installation recipe.

First useful check

Choose either pretrained-policy playback or the supplied retargeting sequence. These answer different questions. First confirm files and paths; do not begin by collecting a new video or launching a full training run.

A small first experiment

  1. Keep the two repositories and their versions together

    Follow the author setup for dexmv-sim, dexmv-learn and dexmv-learn/mjrl. Record both commits. The released environment includes old Linux-specific packages, so resolve platform compatibility in isolation before adding your robot software. Installing only the simulation package is not sufficient for every saved-policy class.

    Separate learning repository →
  2. Replay one policy from the examples directory

    From dexmv-sim/examples, the visualizer resolves ../pretrained_model/relocate-tomato_soup_can.pickle for this task. Verify that trusted file exists, then run the command below. The upstream README’s DEXMV_PACKAGE_PATH/examples line omits the repository directory; use the real path. The visualizer reports accumulated reward, not a universal physical success rate.

    cd dexmv-sim/examples
    python visualize_policy.py --env_name=relocate --object_name=tomato_soup_can
    Policy path, task settings and replay loop →
  3. Check paired pose frames before retargeting

    The supplied hand_pose folder uses joints_<frame>.npy and results_global_<frame>.npy. The retargeter sorts two lists separately and takes their minimum length, so missing or mismatched frames can silently pair different times. Save this original checker as check_frames.py and run it on that folder. It reads filenames only; it does not load pose data or modify files.

    from pathlib import Path
    import re
    import sys
    
    # Usage: python check_frames.py /path/to/hand_pose
    folder = Path(sys.argv[1])
    def frame_ids(prefix):
        ids = set()
        for path in folder.glob(prefix + "*.npy"):
            match = re.fullmatch(re.escape(prefix) + r"(\d+)\.npy", path.name)
            if match:
                frame = int(match.group(1))
                if frame in ids:
                    raise ValueError("Duplicate numeric frame ID: " + str(frame))
                ids.add(frame)
        return ids
    
    joints = frame_ids("joints_")
    global_poses = frame_ids("results_global_")
    print("Joint frames:", len(joints), "Global poses:", len(global_poses))
    print("Missing global poses:", sorted(joints - global_poses))
    print("Missing joint frames:", sorted(global_poses - joints))
    if not joints or joints != global_poses:
        raise SystemExit("STOP: repair missing or mismatched frames first")
    print("Frame IDs match; pose contents, timing and calibration still need review.")
    Frame loading and retargeting implementation →
  4. Keep the generated filename identical during playback

    Run these commands from dexmv-sim/examples after the frame check passes. The README’s viewer example uses a different filename extension from its generation command; the commands here consistently use .pkl. The viewer contains calibration values for the supplied sequence and skips its first ten frames. Reusing those values with a new recording is not a valid calibration.

    python retarget_human_hand.py --hand_dir=./retargeting_source/relocate_mustard_example_seq/hand_pose --output_file=example_retargeting.pkl
    python visualize_retargeting.py --retargeting_result=example_retargeting.pkl --object_dir=./retargeting_source/relocate_mustard_example_seq/object_pose
    Viewer calibration, alignment and trusted-file loading →
  5. Translate poses into the environment’s state/action convention

    Retargeting alone supplies robot joint motion. A learning demonstration also needs camera-to-world transforms, aligned timestamps, joint velocities/accelerations and inverse-dynamics actions. DexMV additionally sets a hindsight target for relocation. Validate each stage with a short sequence; a plausible hand animation is not yet a training-ready trajectory.

    Author demonstration-translation stages →
  6. Audit one training configuration before committing compute

    The released mug DAPG example expects ../demonstrations/relocate-mug.pkl and ENV_NAME relocate-mug-0.8. Check object identity, scale, demonstration provenance and USE_DAPG together. The trainer reads the demonstration dictionary’s values; a retargeted joint array cannot simply replace that file. Run training only after the environment and data checks pass.

    # From dexmv-sim/examples, after setup and data checks:
    python train.py --cfg configs/dapg-mug-example.yaml
    Exact demonstration path and DAPG configuration →

What to record

  • Record both repository commits, platform, legacy simulator environment and trusted file origins.
  • Keep frame IDs, timestamps, units and camera/world transforms with each demonstration.
  • Separate pose alignment, simulated rollout reward and real-task success in your report.
Common problems and what to check
The policy file cannot be found
Run from dexmv-sim/examples and check the matching file in ../pretrained_model. Task and object names determine the filename.
Retargeting succeeds but playback looks misaligned
Check frame pairs and the viewer’s fixed calibration/initial-frame skip. Recalculate transforms for your capture; do not copy the supplied sequence’s extrinsics.
Training rejects a retargeted file
The trainer expects processed demonstrations, not a plain joint-position sequence. Complete translation and inspect the trainer’s expected schema.
A new MuJoCo installation does not run the examples
This release uses the older mujoco-py stack. Isolate that environment or explicitly port and revalidate it; changing only the package name does not establish compatibility.

Where it stops

This page does not reproduce DexMV’s learning results or certify real-hand transfer. The demonstration-generation documentation assumes estimated hand and object poses already exist; it is not a complete raw-video pose-estimation package. Only open trusted author or self-generated pickle files.

Public source reviewed; filename checker tested on release file listings and synthetic cases, not a learning or hardware reproduction · reviewed 2026-10-25. Record the exact source revision when reproducing it.

Keep exploring

Dex Retargeting

Software for mapping human hand motion to the joint configurations of robotic hands.

DexPoint

Point-cloud reinforcement learning for generalizable dexterous manipulation.

DexImit

A pipeline for learning bimanual dexterous manipulation from monocular human videos.

Editorial reference · checked 2026-10-25

Start a discussion

Published posts are public. First contributions are reviewed. Only share material you have permission to disclose.

Report a post