DexYCB
NVIDIA / University of Washington
Human hand–object RGB-D data for grasp perception and handover research. Explore downloads, annotation fields, official splits and a first Python inspection.
What you can explore
Evaluate hand/object perception; inspect one sample and select a benchmark split before training.
Before you use it
Human observations are not robot action trajectories. Retargeting and robot validation are separate work.
Original contributors
Yu-Wei Chao and collaborators
License / access: Data: CC BY-NC 4.0 · toolkit: GPL-3.0 · MANO terms separate
DexYCB dataset: from download to a trustworthy first sample
- Environment
- Python toolkit; upstream tested Python 3.7 on Linux
- Embodiment
- Human hands with YCB objects; no robot-hand driver
What you need
Start with a data question: detecting a hand, estimating an object pose, or studying handover. The authors provide downloads and benchmark tools. Their legacy environment needs an isolated setup; a current Python install is not automatically compatible. Downloaded data, toolkit code and MANO models have different terms.
First useful check
Inspect one annotation file and its matching image before training. Success here means that you can explain the fields, identify absent labels and reproduce a pose overlay—not that you have trained a policy.
A small first experiment
Choose the files and record their provenance
Use the official project download page. It offers one archive (listed as 119G compressed) and separate subject, calibration and model archives. Allow additional extraction space and keep the directory structure intact. Review the data’s CC BY-NC 4.0 terms, the toolkit’s GPL-3.0 license and MANO’s separate access conditions. Record the download date and toolkit commit in your experiment notes; do not assume one license covers everything.
Official data downloads and terms →Load one sample in the documented environment
After following the upstream installation, set DEX_YCB_DIR to the extracted dataset root and run the author example below from the toolkit directory. It enumerates setups and splits and prints sample paths and metadata. This is a dataset-loader check, not a training command. A partial subject download may not satisfy the full split loader.
Read the exact dataset-loading example →python examples/create_dataset.pyInspect labels before rendering
Save this as check_dexycb_labels.py and run python check_dexycb_labels.py /path/to/labels_000053.npz in an environment with NumPy. It checks joint and MANO shapes, finite values and missing-hand sentinels without writing files or loading pickle objects. Exit 0 means these checks passed, including a correctly marked absent hand; exit 1 means malformed or unreadable input. This is not a full dataset validator: images, object poses, calibration, annotation accuracy and split membership still need separate checks.
Annotation definitions in the toolkit →import sys from pathlib import Path import numpy as np def check(path): with np.load(path, allow_pickle=False) as labels: arrays = {} for name, shape in (("joint_3d", (1, 21, 3)), ("joint_2d", (1, 21, 2)), ("pose_m", (1, 51))): value = labels[name] if value.shape != shape or value.dtype.kind not in "fi": raise ValueError(f"{name}: expected numeric array {shape}") if not np.isfinite(value).all(): raise ValueError(f"{name}: NaN or infinity") arrays[name] = value absent = [bool((arrays[k] == -1).all()) for k in ("joint_3d", "joint_2d")] absent.append(bool((arrays["pose_m"] == 0).all())) if any(absent) and not all(absent): raise ValueError("Missing-hand markers disagree; inspect this sample") if all(absent): print("PASS: hand annotation absent; exclude from hand-pose error") else: print("PASS: finite arrays with expected shapes; verify the overlay next") print("Presence/shape checks do not establish annotation correctness") if __name__ == "__main__": if len(sys.argv) != 2: raise SystemExit("Usage: python check_dexycb_labels.py LABEL_FILE.npz") try: check(Path(sys.argv[1])) except (OSError, ValueError, KeyError, EOFError) as error: raise SystemExit(f"FAIL: {error}")Choose the generalization question, then the split
The official loader defines s0 for seen subjects, camera views and grasped objects; s1 holds out subjects; s2 holds out camera views; s3 holds out grasped objects. Select train, val or test within that setup. Use the released grouping when comparing published results. Randomly mixing frames from the same recording can turn recognition of a familiar scene into an apparently strong generalization result.
Inspect the released split definitions →Overlay first; benchmark second
Run the author’s visualization example after its dependencies and models are available. Check that the hand and object align with the image, using the provided camera intrinsics and coordinate conversions. The renderer converts MANO mesh scale and camera axes; do not remove those steps to make a misaligned mesh look plausible. For evaluation, choose COCO detection, BOP object pose, HPE hand pose or Grasp handover metrics that match your question.
Read the original pose-overlay implementation →python examples/visualize_pose.pyMove from human pose labels to a robot experiment
First audit a demonstration: identify observations, missing annotations and the action labels your task would require. Then inspect retargeting joint order and coordinate frames before connecting any controller. A passing NPZ check does not turn a human pose into a safe robot command.
Retargeting setup and joint-mapping checks →
What to record
- Write down the dataset source, toolkit commit, environment, setup (s0–s3), split and sample identifier.
- Save the matching image and your overlay locally; report missing annotations and any frame or scale mismatch.
- State the chosen metric, units and evaluation protocol. A successful loader or overlay is not a benchmark score.
- If your goal is robot action learning, list the missing action labels and embodiment mapping before choosing a model.
Common problems and what to check
- The loader cannot find data
- Check DEX_YCB_DIR, extraction layout, calibration/models and every subject required by the chosen split. Pointing it at one subject directory is not the same as the dataset root.
- The mesh floats away from the RGB image
- Inspect camera intrinsics, handedness, frame conversion and scale against the original renderer. Do not tune an arbitrary offset before identifying the mismatch.
- The reported result looks unusually strong
- Check whether views or frames from the same recording crossed your split boundary. Report the official setup and metric; a custom split needs a separate label.
- I want to train a VLA directly
- Human keypoints are observations or pose labels, not motor commands. Define the target robot’s action interface and data mapping first; the dataset name alone does not establish policy compatibility.
- The script passes but the image overlay is wrong
- Passing checks only establish a narrow input contract. Match color, depth and labels from the same camera directory and frame number; keep the sample intrinsics and MANO side. Use the linked author renderer to inspect alignment before computing errors.
Where it stops
DexYCB observes human grasping. It does not provide a ready-made action interface for Prima 1, LEAP, Allegro or another robot hand. A retargeting or policy project still needs joint mapping, control semantics and separate validation. No dataset files or author checkpoints are hosted by Dexhands.
Understand the data before downloading.
Human hand–object observations · RGB-D images + NPZ labels
color_file / depth_file
Image paths
Synchronized visual observations, not actions.
- Timing and units
- Check the capture metadata; image samples alone do not define a robot control rate.
- Permissions
- Data: CC BY-NC 4.0. Toolkit: GPL-3.0. MANO assets require their own access terms.
First inspection
- Obtain the dataset and separately licensed MANO assets from the original providers.
- Use the toolkit’s create_dataset example to inspect paths and labels.
- Visualize a sample and confirm missing-label handling before evaluation.
Human joints are not robot joint targets. Check the subject/object split and coordinate conventions before training.
Keep exploring
GRAB
Whole-body human motion and object interactions, with articulated hand poses and derived contact.
DexCap
Hand motion capture and scene observations for learning dexterous manipulation from human demonstrations.
Dex Retargeting
Software for mapping human hand motion to the joint configurations of robotic hands.
Editorial reference · checked 2026-10-23