DexVLA
midea-ai
A vision-language model with a diffusion-policy component for visuomotor robot learning.
What you can explore
Explore example data preparation, pretrained model setup, training and evaluation.
Before you use it
The name does not establish compatibility with every dexterous hand. Match the robot action space and data; the original license text restricts commercial use.
Original contributors
Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen and Feifei Feng
License / access: Attribution and non-commercial restrictions in source LICENSE · check pretrained model terms
DexVLA: check the data contract before training
- Environment
- Python 3.10 training stack; begin with an offline HDF5 audit
- Embodiment
- The selected dataset and action head; no universal hand driver
What you need
DexVLA combines a vision-language backbone with a diffusion action expert. Start by writing down your observation and action contract, then check one episode. Training requires the upstream dependencies, pretrained components and a task configuration. The project name does not mean a checkpoint can directly control every dexterous hand.
First useful check
Produce a one-episode report: frame count, camera names, image layout, state/action dimensions and missing fields. This inexpensive check catches format mistakes before a long GPU run.
A small first experiment
Audit one local episode without a robot
Use the original example-data link in the repository. Save this read-only snippet as inspect_episode.py and run it with an HDF5 file path in a Python environment containing h5py and NumPy. It checks nonempty numeric state/action arrays and camera lengths, reading numerical values in batches of 1,024 frames. Explicit validation remains active under python -O. Camera pixels are not loaded or decoded; passing does not prove synchronization, action meaning or task success. Missing language/reasoning labels are reported for separate task-specific review.
Original episode format and sample data →import argparse import h5py import numpy as np parser = argparse.ArgumentParser() parser.add_argument("episode") args = parser.parse_args() def require(condition, message): if not condition: raise SystemExit(message) with h5py.File(args.episode, "r") as episode: def dataset(key): require(key in episode, "Missing dataset: " + key) value = episode[key] require(isinstance(value, h5py.Dataset), "Expected dataset: " + key) return value action = dataset("action") require(action.ndim == 2 and all(action.shape), "Empty/invalid action array") frames = action.shape[0] for key in ("action", "observations/qpos"): values = dataset(key) require(values.ndim == 2 and values.shape[0] == frames and values.shape[1] > 0, key + " shape mismatch") require(values.dtype.kind in "iuf", key + " must contain real numeric values") for start in range(0, frames, 1024): require(np.isfinite(values[start:start + 1024]).all(), key + " contains NaN/Inf") print(key, values.shape, values.dtype) key = "observations/images" require(key in episode and isinstance(episode[key], h5py.Group), "Missing camera group") require(len(episode[key]) > 0, "No camera streams") for name in episode[key]: camera = dataset(key + "/" + name) require(camera.ndim > 0 and camera.shape[0] == frames, name + " length mismatch") print("camera", name, camera.shape, camera.dtype) for key in ("language_raw", "substep_reasonings"): print(key, episode[key].shape if key in episode and isinstance(episode[key], h5py.Dataset) else "MISSING") print("PASS: structural checks only; image decoding, timing and action meaning not verified")Map the task by meaning, not array size
Match camera names to the task configuration in aloha_scripts/constants.py. For every action slot, document its joint or end-effector meaning, units, reference frame and absolute-versus-delta convention. State and action dimensions may differ. Check timestamp alignment separately; equal frame counts cannot prove that an image and action happened together.
Write the state-to-action contract →Keep backbone, action expert and processor together
Follow the repository’s staged setup rather than mixing weights from unrelated examples. Record the code revision, selected backbone and diffusion-head configuration. Check that preprocessor_config.json and chat_template.json accompany the checkpoint you actually load, not only its parent directory. Preserve the original downloads when adapting a configuration.
Backbone and training instructions →Separate offline checks from physical evaluation
Compare predicted actions with a held-out recording before connecting a robot. Report per-channel ranges, units and obvious discontinuities, while noting that offline agreement does not establish closed-loop success. The repository’s robot evaluation scripts are deployment references; they need the matching controller and a separately commissioned setup.
Check splits, alignment and data leakage →
What to record
- Keep an episode manifest, camera/action map and source revision with the experiment.
- Record which observations and reasoning labels exist; do not silently fabricate missing training fields.
- Log configuration, checkpoint, dataset split, random seed and errors. Keep offline checks distinct from closed-loop robot trials.
Common problems and what to check
- Unexpected images argument in the processor
- Check the processor and chat-template files in the loaded checkpoint directory against the original instructions before changing library code.
- CUDA memory runs out
- Treat GPU use as configuration-dependent. The authors suggest options including LoRA, offload and a smaller action expert; first make the data audit pass, then change one setting at a time.
- Shapes pass but actions look wrong
- Check normalization, action ordering, units and checkpoint/config pairing. Stop before hardware execution; a correctly shaped tensor can still encode the wrong command.
Where it stops
The example action shape is not a dexterous-hand DOF specification. A new hand needs an explicit joint map, units, limits and timing. The linked repository’s license restricts commercial use; check model and dataset terms separately. We have not trained DexVLA or deployed its policy on hardware.
Keep exploring
LingBot-VLA 2.0
A vision-language-action release with an action representation that includes dexterous hands and other robot subsystems.
Diffusion Policy
A visuomotor policy-learning framework that generates action sequences through diffusion.
DeCAL
A dexterous vision-language-action model combining contact-aware multimodal prediction and action generation.
Editorial reference · checked 2026-09-22