DEXTEROUS MANIPULATION

The world of dexterous hands.

Compare hardware. Discover applications. Share what works. An open reference and community for dexterous manipulation.

DexYCB

NVIDIA / University of Washington

Human hand–object RGB-D data for grasp perception and handover research. Explore downloads, annotation fields, official splits and a first Python inspection.

What you can explore

Evaluate hand/object perception; inspect one sample and select a benchmark split before training.

Before you use it

Human observations are not robot action trajectories. Retargeting and robot validation are separate work.

Original contributors

Yu-Wei Chao and collaborators

License / access: Data: CC BY-NC 4.0 · toolkit: GPL-3.0 · MANO terms separate

Project & data

Toolkit & examples

Research paper

FROM REFERENCE TO FIRST EXPERIMENT

DexYCB dataset: from download to a trustworthy first sample

INPUTRGB-D observations, calibration and pose annotations
OUTPUTA checked sample, chosen split and reproducible evaluation plan
Environment
Python toolkit; upstream tested Python 3.7 on Linux
Embodiment
Human hands with YCB objects; no robot-hand driver

What you need

Start with a data question: detecting a hand, estimating an object pose, or studying handover. The authors provide downloads and benchmark tools. Their legacy environment needs an isolated setup; a current Python install is not automatically compatible. Downloaded data, toolkit code and MANO models have different terms.

First useful check

Inspect one annotation file and its matching image before training. Success here means that you can explain the fields, identify absent labels and reproduce a pose overlay—not that you have trained a policy.

A small first experiment

  1. Choose the files and record their provenance

    Use the official project download page. It offers one archive (listed as 119G compressed) and separate subject, calibration and model archives. Allow additional extraction space and keep the directory structure intact. Review the data’s CC BY-NC 4.0 terms, the toolkit’s GPL-3.0 license and MANO’s separate access conditions. Record the download date and toolkit commit in your experiment notes; do not assume one license covers everything.

    Official data downloads and terms →
  2. Load one sample in the documented environment

    After following the upstream installation, set DEX_YCB_DIR to the extracted dataset root and run the author example below from the toolkit directory. It enumerates setups and splits and prints sample paths and metadata. This is a dataset-loader check, not a training command. A partial subject download may not satisfy the full split loader.

    python examples/create_dataset.py
    Read the exact dataset-loading example →
  3. Inspect labels before rendering

    Save this as check_dexycb_labels.py and run python check_dexycb_labels.py /path/to/labels_000053.npz in an environment with NumPy. It checks joint and MANO shapes, finite values and missing-hand sentinels without writing files or loading pickle objects. Exit 0 means these checks passed, including a correctly marked absent hand; exit 1 means malformed or unreadable input. This is not a full dataset validator: images, object poses, calibration, annotation accuracy and split membership still need separate checks.

    import sys
    from pathlib import Path
    import numpy as np
    
    
    def check(path):
        with np.load(path, allow_pickle=False) as labels:
            arrays = {}
            for name, shape in (("joint_3d", (1, 21, 3)),
                                ("joint_2d", (1, 21, 2)),
                                ("pose_m", (1, 51))):
                value = labels[name]
                if value.shape != shape or value.dtype.kind not in "fi":
                    raise ValueError(f"{name}: expected numeric array {shape}")
                if not np.isfinite(value).all():
                    raise ValueError(f"{name}: NaN or infinity")
                arrays[name] = value
            absent = [bool((arrays[k] == -1).all())
                      for k in ("joint_3d", "joint_2d")]
            absent.append(bool((arrays["pose_m"] == 0).all()))
            if any(absent) and not all(absent):
                raise ValueError("Missing-hand markers disagree; inspect this sample")
            if all(absent):
                print("PASS: hand annotation absent; exclude from hand-pose error")
            else:
                print("PASS: finite arrays with expected shapes; verify the overlay next")
            print("Presence/shape checks do not establish annotation correctness")
    
    
    if __name__ == "__main__":
        if len(sys.argv) != 2:
            raise SystemExit("Usage: python check_dexycb_labels.py LABEL_FILE.npz")
        try:
            check(Path(sys.argv[1]))
        except (OSError, ValueError, KeyError, EOFError) as error:
            raise SystemExit(f"FAIL: {error}")
    Annotation definitions in the toolkit →
  4. Choose the generalization question, then the split

    The official loader defines s0 for seen subjects, camera views and grasped objects; s1 holds out subjects; s2 holds out camera views; s3 holds out grasped objects. Select train, val or test within that setup. Use the released grouping when comparing published results. Randomly mixing frames from the same recording can turn recognition of a familiar scene into an apparently strong generalization result.

    Inspect the released split definitions →
  5. Overlay first; benchmark second

    Run the author’s visualization example after its dependencies and models are available. Check that the hand and object align with the image, using the provided camera intrinsics and coordinate conversions. The renderer converts MANO mesh scale and camera axes; do not remove those steps to make a misaligned mesh look plausible. For evaluation, choose COCO detection, BOP object pose, HPE hand pose or Grasp handover metrics that match your question.

    python examples/visualize_pose.py
    Read the original pose-overlay implementation →
  6. Move from human pose labels to a robot experiment

    First audit a demonstration: identify observations, missing annotations and the action labels your task would require. Then inspect retargeting joint order and coordinate frames before connecting any controller. A passing NPZ check does not turn a human pose into a safe robot command.

    Retargeting setup and joint-mapping checks →

What to record

  • Write down the dataset source, toolkit commit, environment, setup (s0–s3), split and sample identifier.
  • Save the matching image and your overlay locally; report missing annotations and any frame or scale mismatch.
  • State the chosen metric, units and evaluation protocol. A successful loader or overlay is not a benchmark score.
  • If your goal is robot action learning, list the missing action labels and embodiment mapping before choosing a model.
Common problems and what to check
The loader cannot find data
Check DEX_YCB_DIR, extraction layout, calibration/models and every subject required by the chosen split. Pointing it at one subject directory is not the same as the dataset root.
The mesh floats away from the RGB image
Inspect camera intrinsics, handedness, frame conversion and scale against the original renderer. Do not tune an arbitrary offset before identifying the mismatch.
The reported result looks unusually strong
Check whether views or frames from the same recording crossed your split boundary. Report the official setup and metric; a custom split needs a separate label.
I want to train a VLA directly
Human keypoints are observations or pose labels, not motor commands. Define the target robot’s action interface and data mapping first; the dataset name alone does not establish policy compatibility.
The script passes but the image overlay is wrong
Passing checks only establish a narrow input contract. Match color, depth and labels from the same camera directory and frame number; keep the sample intrinsics and MANO side. Use the linked author renderer to inspect alignment before computing errors.

Where it stops

DexYCB observes human grasping. It does not provide a ready-made action interface for Prima 1, LEAP, Allegro or another robot hand. A retargeting or policy project still needs joint mapping, control semantics and separate validation. No dataset files or author checkpoints are hosted by Dexhands.

Public documentation and source code reviewed; label checker tested on synthetic inputs, not independently reproduced on the full dataset · reviewed 2026-11-06. Record the exact source revision when reproducing it.

LOOK INSIDE THE FORMAT

Understand the data before downloading.

Human hand–object observations · RGB-D images + NPZ labels

Interactive schema guide, not a hosted dataset sample. The original files remain with their authors.

SELECTED FIELD

color_file / depth_file

Image paths

Synchronized visual observations, not actions.

Timing and units
Check the capture metadata; image samples alone do not define a robot control rate.
Permissions
Data: CC BY-NC 4.0. Toolkit: GPL-3.0. MANO assets require their own access terms.

First inspection

  1. Obtain the dataset and separately licensed MANO assets from the original providers.
  2. Use the toolkit’s create_dataset example to inspect paths and labels.
  3. Visualize a sample and confirm missing-label handling before evaluation.

Human joints are not robot joint targets. Check the subject/object split and coordinate conventions before training.

Schema guide reviewed 2026-11-06.

Keep exploring

GRAB

Whole-body human motion and object interactions, with articulated hand poses and derived contact.

DexCap

Hand motion capture and scene observations for learning dexterous manipulation from human demonstrations.

Dex Retargeting

Software for mapping human hand motion to the joint configurations of robotic hands.

Editorial reference · checked 2026-10-23

Start a discussion

Published posts are public. First contributions are reviewed. Only share material you have permission to disclose.

Report a post