Dex Retargeting
DexSuite contributors
Software for mapping human hand motion to the joint configurations of robotic hands.
What you can explore
Explore retargeting objectives and supported hand models before connecting a robot.
Before you use it
Joint ordering can differ between a model, simulator and robot driver. Map joints by name and validate motion limits before hardware use.
Original contributors
Yuzhe Qin and contributors
License / access: MIT · check dependency and model licenses separately
Dex Retargeting: from a recorded hand video to robot joint motion
- Environment
- Python / SAPIEN examples
- Embodiment
- A hand configuration supplied by the project
What you need
Install the repository with its example dependencies in a separate environment. The core package alone does not include everything needed for rendering and hand detection. Record the commit, NumPy version and selected hand configuration.
First useful check
Use a prerecorded clip and visualization before attempting a live camera or robot connection.
A small first experiment
Confirm the example environment
From the repository root, follow the upstream example installation. Inspect the command options before running a conversion. Use the hand and laterality supported by that example.
Video retargeting tutorial →cd example/vector_retargeting python3 detect_from_video.py --helpConvert and render one short clip
After installing the documented dependencies, run these commands from example/vector_retargeting. This is the author’s right-hand Allegro example, not a configuration for every hand. Use a new output filename if you need to preserve an earlier run. Inspect the rendered clip against the input; the pickle contains metadata and per-frame joint vectors, not a hardware-ready command stream. Load only your own or trusted pickle files.
Conversion and rendering examples →python3 detect_from_video.py --robot-name allegro --hand-type right --retargeting-type dexpilot --video-path data/human_hand_video.mp4 --output-path data/allegro_joints.pkl python3 render_robot_hand.py --pickle-path data/allegro_joints.pkl --output-video-path data/allegro.mp4 --headlessAudit the interface by name
Write a joint-name mapping between retargeter and simulator. Equal vector lengths do not guarantee equal ordering. Check left/right selection, motion range, discontinuities and missing input frames before moving to a driver. Use the hand directory to verify the exact model and SDK before choosing a target configuration.
Audit the action interface →Test joint mapping without a robot
Run this standalone Python exercise. The expected output is [0.3, 0.1, 0.2]. It deliberately uses a different target order. In your application, replace the example names with the saved trajectory metadata and simulator active-joint names; never infer order from array length. This tests indexing only, not units, limits, collisions or a hardware driver.
Upstream renderer: joint names and saved metadata →from math import isfinite def reorder(source, target, values): if len(source) != len(values): raise ValueError("Joint/value count mismatch") if len(set(source)) != len(source) or len(set(target)) != len(target): raise ValueError("Duplicate joint names") if not all(isfinite(v) for v in values): raise ValueError("Non-finite joint value") by_name = dict(zip(source, values)) if any(name not in by_name for name in target): raise ValueError("Target joint missing in source") return [by_name[name] for name in target] print(reorder(["thumb", "index", "middle"], ["middle", "thumb", "index"], [0.1, 0.2, 0.3]))
What to record
- Save the input clip, configuration, commit and rendered result together.
- Record which joint names map to which outputs, and examples of tracking or pose failures.
- For the same short input, inspect thumb opposition, fingertip placement and tracking gaps before and after mapping. Keep hand side and configuration fixed; a smoother-looking render is not a measured grasping improvement.
- Offline indexing exercise tested with reordered names, missing names, duplicate names, wrong vector length and non-finite values. Full upstream detection/rendering and hardware motion were not reproduced.
Common problems and what to check
- The wrong finger moves
- Inspect the joint-name mapping; do not fix this by swapping array entries without documenting their names.
- Example dependencies fail
- Check the selected release and its NumPy requirements. Keep rendering/detection dependencies separate from a working robot-control environment.
- Robot assets are missing
- Check git submodule status and the assets/robots/hands directory. The repository stores robot assets in a submodule; follow the upstream checkout instructions and preserve its revision.
- Headless rendering fails
- The reviewed renderer selects ray-tracing shaders when headless is enabled. Headless does not mean CPU-only or no graphics dependencies. Check the SAPIEN rendering environment before changing retargeting settings.
- Rendered speed differs from the recording
- The reviewed renderer writes at 30 fps. A rendered video is a visual inspection aid; do not assume it preserves source timestamps or is a real-time control trajectory.
- The example renders, but my hand is different
- Select a released configuration for your exact target and side, then recheck joint names and limits. Do not rename the Allegro output and feed it to another hand. Use the hand directory to check model, SDK and simulation evidence before integration.
Where it stops
This maps motion; it is not a collision-safe hardware controller. Joint-name mapping, limits and a unit-specific commissioning procedure remain necessary.
Keep exploring
AnyTeleop
Vision-based hand–arm teleoperation research with links to retargeting and visualization code.
LEAP Hand
A research hand with design files, APIs and examples for teleoperation and learned manipulation.
MuJoCo Menagerie
A collection of robot models for MuJoCo, including LEAP, Allegro, Shadow Hand and DEX-EE models.
Editorial reference · checked 2026-09-22