DEXTEROUS MANIPULATION

The world of dexterous hands.

Compare hardware. Discover applications. Share what works. An open reference and community for dexterous manipulation.

DexScrew

x-robotics-lab

Dexterous tool manipulation using a learned rotation prior and real-world sensing adaptation.

What you can explore

Inspect the simulation training and documented XHand deployment and fine-tuning pipeline.

Before you use it

Real deployment requires tactile sensing and task-specific data. Simulation rotation success alone does not establish reliable tool use.

Original contributors

Elvis Hsieh, Wen-Han Hsieh, Yen-Jen Wang, Toru Lin, Jitendra Malik, Koushil Sreenath and Haozhi Qi

License / access: Check the original software license and third-party data/model terms

Original code & documentation

FROM REFERENCE TO FIRST EXPERIMENT

DexScrew: evaluate a rotation policy before planning real tool use

INPUTTask assets, teacher/student checkpoints and later real demonstrations
OUTPUTSimulation rollouts, a serialized student policy and a separate real-data adaptation workflow
Environment
Isaac Gym Preview 4 / PyTorch; separate XHand deployment environment
Embodiment
Released XHand screwdriver task

What you need

Pin the repository and follow its Isaac Gym installation guide. The training environment uses Python 3.8, whereas the included XHand SDK instructions describe Python 3.10/3.12. Keep these environments distinct. The install guide contains a malformed pip install -r gdown line; -r expects a requirements file. Use the actual requirements.txt and inspect dependency compatibility rather than copying that line unchanged.

First useful check

Inspect one existing teacher checkpoint in simulation, with explicit device and checkpoint overrides, before starting a new training run.

A small first experiment

  1. Map the four stages and their artifacts

    Keep teacher training, student distillation, policy export and real-data adaptation as separate experiment stages. Before evaluation, obtain or train the matching task checkpoint and record its origin. Do not substitute a nut-bolt checkpoint for a screwdriver policy just because both scripts launch successfully.

    Author method and training entry points →
  2. Fix the viewer’s device and checkpoint inputs

    The reviewed viewer script sets graphics_device_id=7 and checkpoint=last.pth; its third argument is assigned to CACHE but is not used to choose a checkpoint. After completing the documented installation, replace the path below with a checkpoint you actually have. The trailing Hydra overrides select GPU 0 and the checkpoint explicitly. This launches simulation, not the hand.

    bash scripts/vis_screwdriver_teacher.sh 0 42 inspect \
      graphics_device_id=0 \
      checkpoint=/absolute/path/to/your/teacher_checkpoint.pth
    Reviewed viewer script →
  3. Keep teacher and student checkpoints distinct

    The student training script also defaults to checkpoint=last.pth. Pass the intended teacher path explicitly when you start distillation, and keep the student output directory separate. Record the task, seed, observation configuration and file used by each stage. If a checkpoint produces a shape mismatch, compare the policy architecture and observation dimensions before changing code.

    Student distillation settings →
  4. Inspect export settings before producing an artifact

    The conversion script takes GPU, seed and a third positional argument, plus an optional output name. The third argument is not used as the input path: train.load_path defaults to model_best.ckpt. Review and override that setting with the matching student file. Save the resulting JIT artifact alongside its input/output specification, normalization and joint order; successful serialization alone does not validate deployment.

    Student export script →
  5. Treat real-world adaptation as a separate integration

    The XHand deployment folder requires model-path and task settings, an appropriate SDK build and the physical interface. Review these with the hardware operator before any motion. The real-data stage links to skill-teleop; it is not included simply by running the simulator. Compare tactile availability, timing, units and the action map for the exact hardware before applying a learned policy.

    XHand deployment boundary and SDK requirements →
  6. Record a meaningful evaluation

    For simulation, save the checkpoint hash, task settings and several complete rollouts, including resets and failures. For later physical work, define tool engagement, contact, completion and failure criteria separately from rotation. The action-interface guide helps document the hand-specific mapping without assuming a policy trained for XHand supports another hand.

    Document observations and actions →

What to record

  • The viewer uses an available graphics device and the intended existing checkpoint.
  • Teacher, student and exported artifacts have separate provenance and matching task settings.
  • Rotation, physical tool use and real-data adaptation have distinct completion criteria.
Common problems and what to check
The third shell argument does not load my model
The reviewed viewer stores that argument but uses a literal checkpoint path. Set checkpoint= explicitly.
Invalid graphics device
The viewer defaults to graphics device 7. Override it for your actual machine and visible devices.
Package installation fails immediately
The documented -r gdown command treats a package name as a file. Inspect the requirements file and install instructions instead of repeatedly rerunning it.
JIT export works but the robot does not
Verify observation normalization, joint order, SDK revision and control timing with the hardware integration team. Export is not a commissioning test.

Where it stops

The method combines a simulation rotation prior with real demonstrations and additional observations. A simulated spinning tool is not evidence of reliable screw engagement or fastening. This page does not connect to hardware or verify the trained model’s performance.

Public source and scripts reviewed; no GPU training, policy evaluation or hardware deployment reproduced · reviewed 2026-10-23. Record the exact source revision when reproducing it.

Keep exploring

HORA

In-hand object rotation using reinforcement learning and rapid motor adaptation.

DexJoCo

A MuJoCo benchmark for coordinated dexterous manipulation and tool use.

Editorial reference · checked 2026-09-22

Start a discussion

Published posts are public. First contributions are reviewed. Only share material you have permission to disclose.

Report a post