DEXTEROUS MANIPULATION

The world of dexterous hands.

Compare hardware. Discover applications. Share what works. An open reference and community for dexterous manipulation.

DexVLA

midea-ai

A vision-language model with a diffusion-policy component for visuomotor robot learning.

What you can explore

Explore example data preparation, pretrained model setup, training and evaluation.

Before you use it

The name does not establish compatibility with every dexterous hand. Match the robot action space and data; the original license text restricts commercial use.

Original contributors

Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen and Feifei Feng

License / access: Attribution and non-commercial restrictions in source LICENSE · check pretrained model terms

Original code & documentation

Research paper

FROM REFERENCE TO FIRST EXPERIMENT

DexVLA: check the data contract before training

INPUTCamera observations, robot state, language and action labels
OUTPUTValidated episode metadata, then a configured policy experiment
Environment
Python 3.10 training stack; begin with an offline HDF5 audit
Embodiment
The selected dataset and action head; no universal hand driver

What you need

DexVLA combines a vision-language backbone with a diffusion action expert. Start by writing down your observation and action contract, then check one episode. Training requires the upstream dependencies, pretrained components and a task configuration. The project name does not mean a checkpoint can directly control every dexterous hand.

First useful check

Produce a one-episode report: frame count, camera names, image layout, state/action dimensions and missing fields. This inexpensive check catches format mistakes before a long GPU run.

A small first experiment

  1. Audit one local episode without a robot

    Use the original example-data link in the repository. Save this read-only snippet as inspect_episode.py and run it with an HDF5 file path in a Python environment containing h5py and NumPy. It checks nonempty numeric state/action arrays and camera lengths, reading numerical values in batches of 1,024 frames. Explicit validation remains active under python -O. Camera pixels are not loaded or decoded; passing does not prove synchronization, action meaning or task success. Missing language/reasoning labels are reported for separate task-specific review.

    import argparse
    import h5py
    import numpy as np
    
    parser = argparse.ArgumentParser()
    parser.add_argument("episode")
    args = parser.parse_args()
    
    def require(condition, message):
        if not condition:
            raise SystemExit(message)
    
    with h5py.File(args.episode, "r") as episode:
        def dataset(key):
            require(key in episode, "Missing dataset: " + key)
            value = episode[key]
            require(isinstance(value, h5py.Dataset), "Expected dataset: " + key)
            return value
    
        action = dataset("action")
        require(action.ndim == 2 and all(action.shape), "Empty/invalid action array")
        frames = action.shape[0]
        for key in ("action", "observations/qpos"):
            values = dataset(key)
            require(values.ndim == 2 and values.shape[0] == frames and values.shape[1] > 0,
                    key + " shape mismatch")
            require(values.dtype.kind in "iuf", key + " must contain real numeric values")
            for start in range(0, frames, 1024):
                require(np.isfinite(values[start:start + 1024]).all(), key + " contains NaN/Inf")
            print(key, values.shape, values.dtype)
        key = "observations/images"
        require(key in episode and isinstance(episode[key], h5py.Group), "Missing camera group")
        require(len(episode[key]) > 0, "No camera streams")
        for name in episode[key]:
            camera = dataset(key + "/" + name)
            require(camera.ndim > 0 and camera.shape[0] == frames, name + " length mismatch")
            print("camera", name, camera.shape, camera.dtype)
        for key in ("language_raw", "substep_reasonings"):
            print(key, episode[key].shape if key in episode and isinstance(episode[key], h5py.Dataset) else "MISSING")
        print("PASS: structural checks only; image decoding, timing and action meaning not verified")
    Original episode format and sample data →
  2. Map the task by meaning, not array size

    Match camera names to the task configuration in aloha_scripts/constants.py. For every action slot, document its joint or end-effector meaning, units, reference frame and absolute-versus-delta convention. State and action dimensions may differ. Check timestamp alignment separately; equal frame counts cannot prove that an image and action happened together.

    Write the state-to-action contract →
  3. Keep backbone, action expert and processor together

    Follow the repository’s staged setup rather than mixing weights from unrelated examples. Record the code revision, selected backbone and diffusion-head configuration. Check that preprocessor_config.json and chat_template.json accompany the checkpoint you actually load, not only its parent directory. Preserve the original downloads when adapting a configuration.

    Backbone and training instructions →
  4. Separate offline checks from physical evaluation

    Compare predicted actions with a held-out recording before connecting a robot. Report per-channel ranges, units and obvious discontinuities, while noting that offline agreement does not establish closed-loop success. The repository’s robot evaluation scripts are deployment references; they need the matching controller and a separately commissioned setup.

    Check splits, alignment and data leakage →

What to record

  • Keep an episode manifest, camera/action map and source revision with the experiment.
  • Record which observations and reasoning labels exist; do not silently fabricate missing training fields.
  • Log configuration, checkpoint, dataset split, random seed and errors. Keep offline checks distinct from closed-loop robot trials.
Common problems and what to check
Unexpected images argument in the processor
Check the processor and chat-template files in the loaded checkpoint directory against the original instructions before changing library code.
CUDA memory runs out
Treat GPU use as configuration-dependent. The authors suggest options including LoRA, offload and a smaller action expert; first make the data audit pass, then change one setting at a time.
Shapes pass but actions look wrong
Check normalization, action ordering, units and checkpoint/config pairing. Stop before hardware execution; a correctly shaped tensor can still encode the wrong command.

Where it stops

The example action shape is not a dexterous-hand DOF specification. A new hand needs an explicit joint map, units, limits and timing. The linked repository’s license restricts commercial use; check model and dataset terms separately. We have not trained DexVLA or deployed its policy on hardware.

Public documentation reviewed; audit snippet checked on synthetic data, not a reproduced training run · reviewed 2026-10-28. Record the exact source revision when reproducing it.

Keep exploring

LingBot-VLA 2.0

A vision-language-action release with an action representation that includes dexterous hands and other robot subsystems.

Diffusion Policy

A visuomotor policy-learning framework that generates action sequences through diffusion.

DeCAL

A dexterous vision-language-action model combining contact-aware multimodal prediction and action generation.

Editorial reference · checked 2026-09-22

Start a discussion

Published posts are public. First contributions are reviewed. Only share material you have permission to disclose.

Report a post