The question a world model answers

An action-conditioned world model asks what may happen after an action. Its output could be a future image, hand pose or a learned representation of the scene. A controller or planner must still decide what to do with that prediction. For example, two candidate finger motions might predict different object rotations. Selecting a motion needs a task objective and a way to reject unsuitable predictions.

Different outputs serve different purposes

DexWM studies future latent representations of hand–object interaction; its repository documents training and rollout evaluation. Dexterous World Models (DWM) provides scene- and hand-motion-conditioned video generation. These are useful research references with different representations and evaluation paths. A realistic-looking video is not a force estimate, a collision guarantee or an executable sequence of joint commands. Inspect the output rather than assuming the name “world model” defines a complete control system.

Make contact the test, not the decoration

A hand approaching an object can be easy to predict until contact changes the motion. Test the transitions that matter to your task: first touch, slip, occlusion, regrasp and release. State the prediction horizon and what is observed versus inferred. If tactile signals are unavailable, record that limitation. A model that appears convincing before contact may still give poor predictions during the part of the task that decides success.

Use two separate evaluations

First test prediction on held-out sequences, including errors at multiple horizons and failures around contact. Then ask whether using the model improves completed tasks compared with a policy or planner that does not use it. Keep the available observations and action budget comparable. Improvement in image similarity alone does not establish better manipulation, and task success alone does not explain whether the predictions were physically meaningful.

Avoid evaluation leakage

Neighboring frames of the same interaction share object appearance, lighting and motion. Randomly splitting frames can make a test easier than a new deployment. Choose an episode, session or object split that matches the claim you want to make. Record the source data and its permissions, the software revision, checkpoint and preprocessing. When only simulation is evaluated, label the conclusion as simulation evidence.

A first experiment without hardware

Select a documented public evaluation, inspect one permitted sequence and reproduce its stated output if you have the required assets and compute. Build a small failure ledger: object drift, incorrect hand shape, missed contact, unstable rollout and timing mismatch. Keep examples that fail. Only after the prediction baseline is understood should you propose a separate planning or control experiment with a compatible embodiment.

Try it yourself

Check your understanding