Dexterity Intelligence
From a model’s promise
to a hand that can act.
Learn how vision, language, prediction and contact fit together. Build the understanding to choose an experiment—and the evidence to trust its result.
Can a checkpoint command my fingers? Does prediction help planning? What evidence would prove improvement? You can read everything without an account. Exercises run locally in your browser; no robot or model is connected.
Understand the model. Build the experiment.
Nine technical classes · worked examples, interactive calculations and assignments. Start with hand motion and control interfaces.
Core course
Six connected classes: from the model interface to a result you can defend.
- 01
Understand a VLA
Separate language understanding from executable hand actions.
Action tensors · objectives · language tests
Workshop, assignment & worked answer - 02
Trace the action
Make observations, coordinates and controller commands explicit.
Schemas · normalization · retargeting
Workshop, assignment & worked answer - 03
Understand a world model
Separate prediction quality from successful control.
Latent dynamics · rollouts · planning
Workshop, assignment & worked answer - 04
Choose an experiment
Compare learning approaches against the evidence you have.
Splits · baselines · controlled comparisons
Workshop, assignment & worked answer - 05
Close the loop
Explore timing, sensing and the cost of stale observations.
Timing traces · chunking · stale actions
Workshop, assignment & worked answer - 06
Make the result credible
Design held-out tests, ablations and a reproducible evidence record.
Uncertainty · held-out groups · result cards
Workshop, assignment & worked answer
Applied research workshops
Go deeper into contact, training data and adapting behavior to another hand. Complete the relevant core classes first.
- 07
Learn from contact
Choose a tactile representation, synchronize it and test what it adds.
Sparsh · causal sensor joins · tactile ablations
Workshop, assignment & worked answer - 08
Build data worth learning from
Turn recordings into auditable episodes, useful splits and valid labels.
EgoDex · DexUMI · provenance & data leakage
Workshop, assignment & worked answer - 09
Adapt to another hand
Separate geometry, action meaning, dynamics and the transfer experiment.
Joint mapping · sim-to-real · transfer evidence
Workshop, assignment & worked answer
Read the model. Inspect the boundary.
8 selected references · checked 2026-09-25
Author-reported work; not reproduced by Dexhands.
VLA, imitation learning and world models describe overlapping ideas. Families here are navigation labels, not a ranking. Broader manipulation methods are included to explain what a dexterous-hand adaptation would require.
8 profiles · expand any reference
VLAOpenVLAAn instruction-conditioned action model. A useful example for understanding how a vision-language backbone becomes a robot policy.
Scope: Original openvla-7b release; OFT is a separate adaptation recipe. · OpenVLA research team
- Input
- Camera image and language instruction.
- Output
- Tokenized actions decoded with embodiment-specific normalization; the original example uses a 7-dimensional action.
- Evidence & embodiment
- Author-reported arm-and-gripper experiments, including WidowX and Google Robot. This is not a verified multi-finger hand controller.
- Data / model
- Original model: 7B parameters; 970K robot trajectories from an Open X-Embodiment mixture.
- Compute
- Inference and LoRA recipes are provided. Measure memory and end-to-end latency for the exact precision and hardware.
- Release availability
- Code and model checkpoints linked by the authors. Pin the checkpoint and action normalization key.
- License boundary
- MIT code; released Llama-2-derived weights also carry base-model terms. Dataset terms remain separate.
Which finger action fields would need to be added, and where would their training labels come from?
VLAπ₀ / openpiA foundation-policy reference for studying adaptation and continuous action chunks.
Scope: π₀ flow-based policy through openpi. π₀-FAST and π₀.₅ are distinct variants. · Physical Intelligence
- Input
- Images, robot state and task prompt, through a platform-specific adapter.
- Output
- Continuous action chunks for π₀; do not conflate them with π₀-FAST tokenization.
- Evidence & embodiment
- The release includes ALOHA and DROID configurations. The maintainers explicitly caution that checkpoints may not transfer to another setup.
- Data / model
- Base and task-specific checkpoints are available; your own task requires the corresponding data schema and normalization.
- Compute
- The README separates inference, LoRA and full-training GPU requirements. Check the selected model/configuration before allocating hardware.
- Release availability
- Code, base weights and selected expert checkpoints. Follow the exact platform example and checkpoint path.
- License boundary
- Repository code lists Apache-2.0; inspect checkpoint and bundled base-model terms separately, including LICENSE_GEMMA.
Can your arm and individual fingers be represented by the released adapter, or is new data and adaptation required?
VLASmolVLAA smaller VLA reference with a documented fine-tuning workflow.
Scope: lerobot/smolvla_base and the official LeRobot documentation. · Hugging Face / LeRobot
- Input
- Multiple camera views, robot state and language conditioning.
- Output
- Continuous action chunks, learned with flow matching.
- Evidence & embodiment
- The documentation illustrates SO100/SO101 arm tasks. Those examples do not establish independent finger control.
- Data / model
- A 450M base model; task demonstrations must cover the variations you intend to test.
- Compute
- The docs provide a single-GPU training example. Its time and episode count are examples, not a guarantee for another task.
- Release availability
- Base checkpoint, model card and LeRobot implementation. Match the processor, configuration and library revision.
- License boundary
- Base model card lists Apache-2.0. Check each dataset and dependency independently.
Can you first reproduce one documented example, then define a separate multi-finger adaptation?
ImitationDiffusion PolicyA demonstration-learning baseline that generates action sequences by conditional denoising.
Scope: Original RSS 2023 method and official implementation. · Diffusion Policy research team
- Input
- Observation history: visual or low-dimensional state, depending on the task configuration.
- Output
- An action sequence, of which a configured portion is executed before replanning.
- Evidence & embodiment
- Author-reported simulation benchmarks and physical manipulation tasks. The original release is not a universal dexterous-hand driver.
- Data / model
- Task demonstrations; the release provides experiment configurations, data links and checkpoints.
- Compute
- Official simulation reproduction uses a Linux/NVIDIA environment; configuration and inference steps affect runtime.
- Release availability
- Code, logged experiments and selected checkpoints are publicly linked.
- License boundary
- MIT repository code. Check licenses of each dataset, checkpoint and dependency separately.
Does a task-specific policy with the same demonstrations and sensors already solve your problem?
ImitationACT / ALOHAAn imitation-learning reference for predicting a sequence of actions rather than a single next command.
Scope: Action Chunking with Transformers; original ALOHA study. · Tony Zhao and ALOHA research collaborators
- Input
- Multi-view images and joint state.
- Output
- Joint-position action chunks; temporal aggregation can combine overlapping predictions.
- Evidence & embodiment
- The original work uses bimanual arms with grippers. It is a method reference, not evidence for a particular five-finger hand.
- Data / model
- Demonstrations for a defined task; the repository includes simulated transfer and insertion examples.
- Compute
- Use the release environment and task configuration; training loss alone is not a physical-success metric.
- Release availability
- ACT training/evaluation code and simulation examples; physical use also needs the ALOHA stack.
- License boundary
- MIT repository code; check data and hardware asset terms separately.
Would chunked joint targets fit your controller, and what changes when finger contact requires rapid correction?
World modelDreamerV3A model-based reinforcement-learning reference: learn dynamics, then train behavior in imagined trajectories.
Scope: Nature study and the author-maintained reimplementation. · Danijar Hafner and collaborators
- Input
- Experience containing observations, actions, rewards and episode boundaries.
- Output
- Predicted latent states and rewards; an actor-critic policy supplies actions.
- Evidence & embodiment
- Diverse control benchmarks, not a demonstrated driver for every dexterous hand.
- Data / model
- Environment interaction and a defined reward. A hand task still needs an appropriate environment and action interface.
- Compute
- JAX implementation with configurable model sizes. The debug configuration is for debugging, not benchmark-quality learning.
- Release availability
- Code and environment configurations. Reproducing a task requires its own training run and setup.
- License boundary
- MIT repository code; environment assets may have other terms.
Can your simulator reproduce contact well enough that optimizing its reward teaches the behavior you want?
World modelDexWMA hand–object world-model study with an explicit separation between rollout prediction and robot-task evaluation.
Scope: Official World Models for Learning Dexterous Hand-Object Interactions release. · Meta FAIR / New York University researchers
- Input
- Human/robot interaction observations and action-conditioned sequences in the released setup.
- Output
- Predicted latent states; a separate keypoint model supports rollout evaluation.
- Evidence & embodiment
- EgoDex keypoint-rollout evaluation and RoboCasa MURP simulation task evaluation. This review does not establish real-hand transfer.
- Data / model
- EgoDex and DROID pretraining; exploratory RoboCasa sequences for adaptation.
- Compute
- The README provides distributed training configurations and a single-GPU debug path; debug is not equivalent to full pretraining.
- Release availability
- Code and data links; evaluation requires checkpoints and separately trained evaluation components. A ready-to-use weight package was not verified here.
- License boundary
- Repository states CC-BY-NC. Commercial permission must not be inferred from public access.
Does a lower prediction error actually help a planner complete your held-out manipulation task?
World modelDexterous World ModelsA scene-action-conditioned video model for visualizing hand–scene interaction.
Scope: DWM, CVPR 2026; distinct from DexWM. · Byungjun Kim, Taeksoo Kim, Junyoung Lee and Hanbyul Joo
- Input
- Static scene renderings and egocentric hand mesh/motion renderings.
- Output
- Predicted videos of interaction; not executable joint commands or measured forces.
- Evidence & embodiment
- The paper evaluates generated interactions. Visual plausibility is not a physical contact or control guarantee.
- Data / model
- The paper combines synthetic interactions and real-world video; the repository documents processing and split files.
- Compute
- The default 5B training path is described as requiring an 80GB-class GPU. Other configurations need their own measurement.
- Release availability
- Code, preprocessing, training and inference instructions. A complete pretrained weight download was not verified here.
- License boundary
- Apache-2.0 repository code; base video models and data require separate license checks.
Which errors at first contact or release would make the generated future unusable for your planning objective?
No matching profiles. Try a broader term or reset the filters.
A useful reference states what it does not establish.
Primary sources. Each profile links the original research or author-maintained release. Code availability, downloadable weights, data access and hardware support are checked separately.
Explicit evidence. Author-reported results, local reproductions and illustrative exercises are different. This release contains no new benchmark measurements or hardware tests.
Scoped coverage. This is an initial curated course, not all VLA or world-model research. Source review is dated; upstream releases can change. Contributions should include a source and the exact claim being corrected.