D(R,O) Grasp
NUS / SJTU research team
A shared representation of robot–object geometry for predicting grasps across different robotic hands.
What you can explore
Inspect the hand descriptions, point-cloud inputs and grasp validation used for cross-hand learning.
Before you use it
The documented evaluation depends on Isaac Gym and downloaded checkpoints and data. Generalization claims apply to the authors’ tested embodiments, not every hand in this catalog.
Original contributors
Zhenyu Wei, Zhixuan Xu, Jingxiang Guo, Yiwen Hou, Chongkai Gao, Zhehao Cai, Jiayu Luo and Lin Shao
License / access: Code: MIT · upstream data and robot assets have separate terms
D(R,O) Grasp: from point-cloud matching to a grasp evaluation
- Environment
- Python / PyTorch; Isaac Gym for grasp validation
- Embodiment
- Released Barrett, Allegro and Shadow Hand examples
What you need
The upstream installation uses Python 3.8 and PyTorch 2.3 or later. Keep this legacy environment separate from current Isaac Lab work. Record the repository commit and use the linked release for the checkpoints and data. Isaac Gym is optional for pretraining and training, but required for the released grasp-validation route.
First useful check
Check a released correspondence example before spending time on a full grasp evaluation. Treat them as two different tests with two different success criteria.
A small first experiment
Match the checkpoint, hand and point-cloud mode
Download the author-linked release and inspect its contents before running the examples. Keep pretrained encoders under ckpt/pretrain and grasp networks under ckpt/model. The example scripts distinguish a full-object checkpoint from the partial-point-cloud checkpoint. Record which one you use; do not mix a partial observation with a full-object experiment and call the results comparable.
Released checkpoints and hand assets →Check correspondence before contact physics
After installing the documented dependencies and release assets, run the upstream pretraining example from the repository root. It evaluates the released Barrett, Allegro and Shadow Hand inputs and reports Mean Order. Lower values indicate better correspondence ranking in this test; they are not percentages of objects successfully lifted. Read the script before execution and load only trusted checkpoints.
Read the exact correspondence example →python scripts/example_pretrain.pyMove to the separate grasp-validation route
The released simulation example needs Isaac Gym and a CUDA-capable setup. Inspect its gpu, ckpt_name and batch_size before running it. For a trained model, validate.py instead uses configs/validate.yaml: dataset.batch_size is the total candidates per hand–object combination, while split_batch_size chunks network inference and pose optimization. The script then concatenates those candidates and passes the complete batch to validate_isaac. Reducing split_batch_size does not itself shrink the physics-validation batch.
Inspect the simulation example →# Requires the documented Isaac Gym setup and released assets. python scripts/example_isaac.pyKeep the evaluation settings with the result
For every result, retain the hand, object set, checkpoint, point-cloud mode, candidate count, simulator settings and failures. When using validate.py, compare its overrides against the dataset configuration; the folder is configs/, even where the README prose says config/. Compare success counts only under the same protocol, then use the experiment guide to document what changed.
Validation configuration and overrides →Plan a new-hand integration as a separate experiment
A new URDF alone is insufficient. The upstream route also needs link geometry, sampled points, link-direction definitions and suitable grasp data. First document joint names, frames, limits and the mapping to any eventual driver. Keep model adaptation separate from permission to command a physical device.
Audit the hand action interface →
What to record
- The repository commit, asset release and checkpoint path are recorded.
- Correspondence output and contact-physics results are reported separately.
- Full versus partial point clouds match the chosen checkpoint and evaluation.
- The report includes failed trials and the denominator, not only successful examples.
Common problems and what to check
- How do I record a memory experiment?
- The reviewed config uses 100 candidates and split_batch_size 25: four inference/optimization chunks, followed by validation of the combined batch. A split of 10 makes ten chunks, still 100 candidates. Record the failing stage, both settings, GPU and peak memory. This arithmetic is not a measured memory or speed claim.
- Weights or geometry not found
- Check extraction depth and the ckpt/ and data/ directories; a downloaded archive is not necessarily an installed asset tree.
- GPU memory exhausted
- Locate the failing stage first. For inference or optimization, try a smaller positive split_batch_size while keeping dataset.batch_size fixed. For an error inside validate_isaac, inspect the physics-validation setup: this script passes the concatenated candidate batch to it. Reducing dataset.batch_size changes the number of attempted grasps and must be reported; it is not an equivalent evaluation.
- Unexpectedly good matching score
- Verify which script produced it. The pretraining score does not test object lift, contact stability or hardware transfer.
- A new hand renders but grasps fail
- Inspect link frames, sampled geometry and controller directions before attributing failures to the network. Use the author’s new-hand instructions.
Where it stops
A correspondence score is not a grasp success rate. Released cross-embodiment results do not establish support for an arbitrary hand, and this source review is not a reproduced simulation or a physical-hand test.
Keep exploring
GenDexGrasp
Contact-map-based grasp synthesis designed to generalize across robotic hand embodiments.
DexGraspNet 2.0
Generative dexterous grasping tools for large synthetic cluttered scenes.
CordViP
A correspondence-based policy implementation with data collection, training and real-robot evaluation code.
Editorial reference · checked 2026-09-22