DEXTEROUS MANIPULATION

The world of dexterous hands.

Compare hardware. Discover applications. Share what works. An open reference and community for dexterous manipulation.

DexImit

DexImit research team

A pipeline for learning bimanual dexterous manipulation from monocular human videos.

What you can explore

Inspect video annotation, reconstruction, hand alignment and trajectory-generation stages.

Before you use it

Substantial vision dependencies and compute are required. Third-party models have separate terms; availability of code is not a reproduced result.

Original contributors

Juncheng Mu, Sizhe Yang and collaborators

License / access: MIT per project README · bundled models and dependencies have separate terms

Original code & documentation

FROM REFERENCE TO FIRST EXPERIMENT

DexImit: inspect a human-video task before generating robot trajectories

INPUTA monocular video, hand–object task labels and correction settings
OUTPUTIntermediate masks, geometry and poses; generated robot trajectories
Environment
Linux / CUDA research pipeline with multiple vision and planning dependencies
Embodiment
Human hand reconstruction followed by the released robot configuration

What you need

Use the upstream environment recipe, which specifies Python 3.11 and a PyTorch/CUDA combination plus native dependencies. Begin with supplied manual labels. Automatic Qwen annotation is optional and has additional compute and model requirements. Inspect each dependency and weight license; the main repository license does not grant rights to every model or input video.

First useful check

Run the read-only input check below on the supplied apple task. It needs Python only and does not start inference, download weights or connect to a robot.

A small first experiment

  1. Check task inputs before downloading model weights

    The supplied task separates input.mp4, label_hand_centric.json and rectify.json from generated output. This original Dexhands check catches missing files, invalid object references, hand-side mistakes and unordered frame indices. It does not decode the video or validate the complete upstream schema. After extraction, compare annotation indices against the sampled frames rather than assuming they refer directly to the original video timeline.

    # Run from the DexImit repository root. Reads files; starts no models.
    import json
    from pathlib import Path
    
    root = Path("demo/apple")
    for name in ("input.mp4", "label_hand_centric.json", "rectify.json"):
        path = root / name
        if not path.is_file() or path.stat().st_size == 0:
            raise ValueError(f"Missing or empty input: {path}")
    labels = json.loads((root / "label_hand_centric.json").read_text())
    rectify = json.loads((root / "rectify.json").read_text())
    if not isinstance(rectify, dict):
        raise ValueError("rectify.json must contain an object")
    objects = labels.get("obj_names", {})
    tasks = labels.get("tasks", [])
    if not isinstance(objects, dict) or not objects or not isinstance(tasks, list) or not tasks:
        raise ValueError("Expected named objects and at least one task")
    for task in tasks:
        if task.get("hand") not in ("left", "right") or task.get("object") not in objects:
            raise ValueError("Check hand side and object-name references")
        actions = task.get("subactions", [])
        frames = [action.get("frame_index") for action in actions]
        if not frames or any(type(n) is not int or n < 0 for n in frames):
            raise ValueError("Expected non-negative integer frame indices")
        if frames != sorted(frames):
            raise ValueError("Task frames must be in chronological order")
        print(task["hand"], task["object"], "frames:", frames)
    print("Input structure checked. Video decoding and frame alignment are NOT checked.")
    Compare the supplied task labels →
  2. Prepare the environment and model inventory

    Use the upstream installation order and record your GPU, driver, environment and repository revision. Some weights download explicitly; others arrive on first use. Confirm storage and access before a long run. A manual-annotation trial avoids the optional Qwen model but still needs the other vision and planning dependencies. A Mac running this input check is not evidence that it can run the CUDA pipeline.

    Original installation and weight instructions →
  3. Use one task and a separate output directory

    Once the documented environment and weights are ready, run the author’s pipeline with an explicit output folder from the repository root. The driver requires the deximit environment and handles stage order. Running an individual stage without output_dir may write into its input directory; keep originals separate and preserve a copy of your labels.

    conda activate deximit
    bash pipeline.sh demo/apple output/apple_review
    Inspect the pipeline and its output handling →
  4. Inspect reconstruction before trusting the trajectory

    Review masks, hand alignment, object pose tracking and the world-frame reconstruction before inspecting the final rollout. In the released layout, visual_debug/align.png, visual_debug/hand_align.mp4 and final_render/ expose different stages. A wrong hand side, scale or occluded object can carry through to plausible-looking motion. Compare the beginning, contact transition and release against the source clip.

    Stage outputs and inspection files →
  5. Audit the generated data before training or deployment

    Keep human reconstruction, planned robot motion and physical observations as distinct data sources. Record action names, units, timing, camera frames and task outcomes. A data_record.npz file is an output artifact, not a universal driver command. Use the data-audit lesson before converting a successful example into a training set.

    Audit demonstrations and action semantics →

What to record

  • The task inputs and outputs live in separate directories.
  • Label object names and left/right assignments match the inspected clip.
  • Annotation frames are checked against the actual extracted sequence.
  • Intermediate geometry and final motion are reviewed separately; hardware execution remains untested unless demonstrated.
Common problems and what to check
Missing model on the first run
An installed Python package does not imply its weights are cached. Review the upstream checkpoint inventory before switching to offline mode.
Environment mismatch
Native rendering and planning dependencies depend on the stated Python, PyTorch and CUDA combination. Replacing one component can break another.
Good render, poor contact
Inspect depth scale, object masks, hand alignment and scene coordinates; visual plausibility is not a physical success measure.
Overwriting original inputs
Always supply an output directory when running a single stage. Save input labels separately so a retry remains traceable.

Where it stops

Valid JSON and a rendered preview do not demonstrate accurate depth, contact or task success. We have not run the full reconstruction/planning pipeline or tested physical execution. Changing the robot requires a matching planning and action interface.

Public pipeline and sample labels reviewed at commit c574980; the input checker is a Dexhands utility, not a reproduced DexImit result. · reviewed 2026-10-23. Record the exact source revision when reproducing it.

Keep exploring

DexCap

Hand motion capture and scene observations for learning dexterous manipulation from human demonstrations.

BODex

GPU-based dexterous grasp synthesis and approach-trajectory planning using bilevel optimization.

Editorial reference · checked 2026-09-22

Start a discussion

Published posts are public. First contributions are reviewed. Only share material you have permission to disclose.

Report a post