DexMV
DexMV research team
Simulation and retargeting tools for learning dexterous manipulation from human videos.
What you can explore
Explore demonstration processing and simulated relocation, pouring and placement tasks.
Before you use it
This is the simulation/retargeting repository; the learning code is separate. Its documented MuJoCo environment is an older release.
Original contributors
Yuzhe Qin, Yueh-Hua Wu and collaborators
License / access: Apache-2.0 · check model, data and dependency terms separately
DexMV: from human poses to a checked simulation demonstration
- Environment
- Legacy MuJoCo 2.0 / mujoco-py; separate simulation and learning repositories
- Embodiment
- Released Adroit simulation tasks: relocate, pour and place inside
What you need
DexMV separates dexmv-sim from dexmv-learn, including its mjrl subdirectory. The released environment.yml is a Linux/Python 3.7-era snapshot with mujoco-py 2.0.2.13. Treat it as a legacy reproduction environment; it is not a tested Apple Silicon or current MuJoCo installation recipe.
First useful check
Choose either pretrained-policy playback or the supplied retargeting sequence. These answer different questions. First confirm files and paths; do not begin by collecting a new video or launching a full training run.
A small first experiment
Keep the two repositories and their versions together
Follow the author setup for dexmv-sim, dexmv-learn and dexmv-learn/mjrl. Record both commits. The released environment includes old Linux-specific packages, so resolve platform compatibility in isolation before adding your robot software. Installing only the simulation package is not sufficient for every saved-policy class.
Separate learning repository →Replay one policy from the examples directory
From dexmv-sim/examples, the visualizer resolves ../pretrained_model/relocate-tomato_soup_can.pickle for this task. Verify that trusted file exists, then run the command below. The upstream README’s DEXMV_PACKAGE_PATH/examples line omits the repository directory; use the real path. The visualizer reports accumulated reward, not a universal physical success rate.
Policy path, task settings and replay loop →cd dexmv-sim/examples python visualize_policy.py --env_name=relocate --object_name=tomato_soup_canCheck paired pose frames before retargeting
The supplied hand_pose folder uses joints_<frame>.npy and results_global_<frame>.npy. The retargeter sorts two lists separately and takes their minimum length, so missing or mismatched frames can silently pair different times. Save this original checker as check_frames.py and run it on that folder. It reads filenames only; it does not load pose data or modify files.
Frame loading and retargeting implementation →from pathlib import Path import re import sys # Usage: python check_frames.py /path/to/hand_pose folder = Path(sys.argv[1]) def frame_ids(prefix): ids = set() for path in folder.glob(prefix + "*.npy"): match = re.fullmatch(re.escape(prefix) + r"(\d+)\.npy", path.name) if match: frame = int(match.group(1)) if frame in ids: raise ValueError("Duplicate numeric frame ID: " + str(frame)) ids.add(frame) return ids joints = frame_ids("joints_") global_poses = frame_ids("results_global_") print("Joint frames:", len(joints), "Global poses:", len(global_poses)) print("Missing global poses:", sorted(joints - global_poses)) print("Missing joint frames:", sorted(global_poses - joints)) if not joints or joints != global_poses: raise SystemExit("STOP: repair missing or mismatched frames first") print("Frame IDs match; pose contents, timing and calibration still need review.")Keep the generated filename identical during playback
Run these commands from dexmv-sim/examples after the frame check passes. The README’s viewer example uses a different filename extension from its generation command; the commands here consistently use .pkl. The viewer contains calibration values for the supplied sequence and skips its first ten frames. Reusing those values with a new recording is not a valid calibration.
Viewer calibration, alignment and trusted-file loading →python retarget_human_hand.py --hand_dir=./retargeting_source/relocate_mustard_example_seq/hand_pose --output_file=example_retargeting.pkl python visualize_retargeting.py --retargeting_result=example_retargeting.pkl --object_dir=./retargeting_source/relocate_mustard_example_seq/object_poseTranslate poses into the environment’s state/action convention
Retargeting alone supplies robot joint motion. A learning demonstration also needs camera-to-world transforms, aligned timestamps, joint velocities/accelerations and inverse-dynamics actions. DexMV additionally sets a hindsight target for relocation. Validate each stage with a short sequence; a plausible hand animation is not yet a training-ready trajectory.
Author demonstration-translation stages →Audit one training configuration before committing compute
The released mug DAPG example expects ../demonstrations/relocate-mug.pkl and ENV_NAME relocate-mug-0.8. Check object identity, scale, demonstration provenance and USE_DAPG together. The trainer reads the demonstration dictionary’s values; a retargeted joint array cannot simply replace that file. Run training only after the environment and data checks pass.
Exact demonstration path and DAPG configuration →# From dexmv-sim/examples, after setup and data checks: python train.py --cfg configs/dapg-mug-example.yaml
What to record
- Record both repository commits, platform, legacy simulator environment and trusted file origins.
- Keep frame IDs, timestamps, units and camera/world transforms with each demonstration.
- Separate pose alignment, simulated rollout reward and real-task success in your report.
Common problems and what to check
- The policy file cannot be found
- Run from dexmv-sim/examples and check the matching file in ../pretrained_model. Task and object names determine the filename.
- Retargeting succeeds but playback looks misaligned
- Check frame pairs and the viewer’s fixed calibration/initial-frame skip. Recalculate transforms for your capture; do not copy the supplied sequence’s extrinsics.
- Training rejects a retargeted file
- The trainer expects processed demonstrations, not a plain joint-position sequence. Complete translation and inspect the trainer’s expected schema.
- A new MuJoCo installation does not run the examples
- This release uses the older mujoco-py stack. Isolate that environment or explicitly port and revalidate it; changing only the package name does not establish compatibility.
Where it stops
This page does not reproduce DexMV’s learning results or certify real-hand transfer. The demonstration-generation documentation assumes estimated hand and object poses already exist; it is not a complete raw-video pose-estimation package. Only open trusted author or self-generated pickle files.
Keep exploring
Dex Retargeting
Software for mapping human hand motion to the joint configurations of robotic hands.
DexPoint
Point-cloud reinforcement learning for generalizable dexterous manipulation.
DexImit
A pipeline for learning bimanual dexterous manipulation from monocular human videos.
Editorial reference · checked 2026-10-25