DexScrew
x-robotics-lab
Dexterous tool manipulation using a learned rotation prior and real-world sensing adaptation.
What you can explore
Inspect the simulation training and documented XHand deployment and fine-tuning pipeline.
Before you use it
Real deployment requires tactile sensing and task-specific data. Simulation rotation success alone does not establish reliable tool use.
Original contributors
Elvis Hsieh, Wen-Han Hsieh, Yen-Jen Wang, Toru Lin, Jitendra Malik, Koushil Sreenath and Haozhi Qi
License / access: Check the original software license and third-party data/model terms
DexScrew: evaluate a rotation policy before planning real tool use
- Environment
- Isaac Gym Preview 4 / PyTorch; separate XHand deployment environment
- Embodiment
- Released XHand screwdriver task
What you need
Pin the repository and follow its Isaac Gym installation guide. The training environment uses Python 3.8, whereas the included XHand SDK instructions describe Python 3.10/3.12. Keep these environments distinct. The install guide contains a malformed pip install -r gdown line; -r expects a requirements file. Use the actual requirements.txt and inspect dependency compatibility rather than copying that line unchanged.
First useful check
Inspect one existing teacher checkpoint in simulation, with explicit device and checkpoint overrides, before starting a new training run.
A small first experiment
Map the four stages and their artifacts
Keep teacher training, student distillation, policy export and real-data adaptation as separate experiment stages. Before evaluation, obtain or train the matching task checkpoint and record its origin. Do not substitute a nut-bolt checkpoint for a screwdriver policy just because both scripts launch successfully.
Author method and training entry points →Fix the viewer’s device and checkpoint inputs
The reviewed viewer script sets graphics_device_id=7 and checkpoint=last.pth; its third argument is assigned to CACHE but is not used to choose a checkpoint. After completing the documented installation, replace the path below with a checkpoint you actually have. The trailing Hydra overrides select GPU 0 and the checkpoint explicitly. This launches simulation, not the hand.
Reviewed viewer script →bash scripts/vis_screwdriver_teacher.sh 0 42 inspect \ graphics_device_id=0 \ checkpoint=/absolute/path/to/your/teacher_checkpoint.pthKeep teacher and student checkpoints distinct
The student training script also defaults to checkpoint=last.pth. Pass the intended teacher path explicitly when you start distillation, and keep the student output directory separate. Record the task, seed, observation configuration and file used by each stage. If a checkpoint produces a shape mismatch, compare the policy architecture and observation dimensions before changing code.
Student distillation settings →Inspect export settings before producing an artifact
The conversion script takes GPU, seed and a third positional argument, plus an optional output name. The third argument is not used as the input path: train.load_path defaults to model_best.ckpt. Review and override that setting with the matching student file. Save the resulting JIT artifact alongside its input/output specification, normalization and joint order; successful serialization alone does not validate deployment.
Student export script →Treat real-world adaptation as a separate integration
The XHand deployment folder requires model-path and task settings, an appropriate SDK build and the physical interface. Review these with the hardware operator before any motion. The real-data stage links to skill-teleop; it is not included simply by running the simulator. Compare tactile availability, timing, units and the action map for the exact hardware before applying a learned policy.
XHand deployment boundary and SDK requirements →Record a meaningful evaluation
For simulation, save the checkpoint hash, task settings and several complete rollouts, including resets and failures. For later physical work, define tool engagement, contact, completion and failure criteria separately from rotation. The action-interface guide helps document the hand-specific mapping without assuming a policy trained for XHand supports another hand.
Document observations and actions →
What to record
- The viewer uses an available graphics device and the intended existing checkpoint.
- Teacher, student and exported artifacts have separate provenance and matching task settings.
- Rotation, physical tool use and real-data adaptation have distinct completion criteria.
Common problems and what to check
- The third shell argument does not load my model
- The reviewed viewer stores that argument but uses a literal checkpoint path. Set checkpoint= explicitly.
- Invalid graphics device
- The viewer defaults to graphics device 7. Override it for your actual machine and visible devices.
- Package installation fails immediately
- The documented -r gdown command treats a package name as a file. Inspect the requirements file and install instructions instead of repeatedly rerunning it.
- JIT export works but the robot does not
- Verify observation normalization, joint order, SDK revision and control timing with the hardware integration team. Export is not a commissioning test.
Where it stops
The method combines a simulation rotation prior with real demonstrations and additional observations. A simulated spinning tool is not evidence of reliable screw engagement or fastening. This page does not connect to hardware or verify the trained model’s performance.
Keep exploring
HORA
In-hand object rotation using reinforcement learning and rapid motor adaptation.
T-Dex / Dexterity from Touch
Learning tactile representations from exploratory interaction and using them for dexterous manipulation.
DexJoCo
A MuJoCo benchmark for coordinated dexterous manipulation and tool use.
Editorial reference · checked 2026-09-22