From a promising demo to a defensible result
Build a test that can fail, retain the failures and state precisely what the evidence supports.
Define a reproducible evaluation that separates prediction, autonomous execution and assistance.
Turn the claim into a testable sentence
Replace “general dexterity” with a claim such as “reorients a held-out object from a defined start distribution within a fixed time, without assistance.” Specify the hand revision, arm, objects, observations, action interface and controller. Decide what counts as failure, timeout, reset and intervention before running the evaluation. A changed camera or calibration can change the result.
Split at the level of the claim
Adjacent video frames are correlated; random frame splitting can leak nearly identical scenes into training and test sets. Keep complete episodes together. For object generalization, hold out object identities; for deployment generalization, consider sessions, operators or environments. Record which split was used to tune hyperparameters. An untouched final test should not become an informal development set through repeated inspection.
Audit data before optimizing the network
Inspect the observation-to-action alignment, units, missing sensor values, reset boundaries and any clipping. Report collection conditions and permissions. Human-hand poses, robot targets, measured joint states and generated videos are different labels. Avoid mixing them into one “demonstration count” without describing their roles. Dataset size alone does not establish useful coverage for a multi-finger task.
Use a baseline and a targeted ablation
Keep task conditions, available sensors and evaluation budgets comparable. Change one intended factor at a time when attributing an effect: touch, language, world-model planning or action representation. If adding touch also changes the training set and controller, report the bundle as a system change. Do not claim the sensor caused an improvement from that comparison alone.
Report denominators and uncertainty
Show successes over all eligible attempts, separately with and without assistance. Keep a failure ledger, reset time and latency distribution. Report variation across training seeds when learning randomness matters. For proportions, use an appropriate confidence interval and describe how trials were sampled; repeated attempts on one object may not be independent. Do not compare percentages from different tasks as a universal ranking.
Evaluate prediction and control separately
For a world model, measure prediction error on held-out sequences at multiple horizons and inspect contact transitions. Then run a separate downstream experiment to test whether the prediction improves task completion. Keypoint accuracy, image similarity and success rate answer different questions. State whether evidence comes from authors’ results, your reproduction, simulation or physical trials.
Package enough to reproduce and correct
Record source commit, checkpoint identifier, configuration, dependency versions, dataset version/split, seed, hand/controller revisions and evaluation script. Preserve licensed logs and representative failures, not only the best video. If an artifact cannot be shared, explain the resulting reproducibility limit. Dexhands can host a source-linked experiment record; it does not certify that a model is compatible or independently reproduce every paper.
A result record worth sharing
| Record | Required evidence | Common omission |
|---|---|---|
| Claim | Task, start distribution and success rule | Unstated operator assistance |
| Training | Data/split, seed, checkpoint and config | Test scenes used while tuning |
| Prediction | Metric and rollout horizons | Video quality treated as control |
| Execution | All trials, interventions, timing and failures | Only successful clips retained |
| Reuse | Code, assets, licenses and exact embodiment | Code license treated as data permission |
Work through the mechanics
Produce a result card with honest uncertainty and reproducible evidence.
Prerequisites: Fractions, a fixed success definition and held-out trial conditions.
Keep the denominator visible
Write the success rule before evaluation: initial state distribution, allowed duration, completion condition and intervention policy. Count every eligible attempt, including timeouts and failures. Report assisted outcomes separately from unassisted outcomes. A command accepted by the controller is not task success.
For an action predictor, an offline error measures agreement with recorded targets. For a world model, a rollout error measures prediction under a particular horizon and conditioning. For a deployed system, success measures task completion. These metrics may disagree; each belongs to its own claim.
Use the calculator to see why 18 successes in 20 trials and 90 in 100 trials both give 90% yet convey different uncertainty. The Wilson interval uses a binomial model with independent trials and a common success probability. Repeated near-identical trials, shared resets or mixed difficulties can break that interpretation.
p̂ = successes / trials 95% Wilson interval, z = 1.96: center = (p̂ + z²/(2n)) / (1 + z²/n) half-width = z√(p̂(1−p̂)/n + z²/(4n²)) / (1 + z²/n)
Compare evidence strength
Synthetic teaching example · calculations only. No inference, simulation or hardware execution.
18 / 20 trials · approximate 95% Wilson interval: 69.9–97.2%.
Assumes independent trials with a common success probability. It does not account for object/session clustering or distribution shift.
Read the primary work: NIST: confidence intervals for proportions ↗
Use the right unit of independence
If 100 trials use one object in one session, the dataset says little about variation across objects or sessions. Keep a per-condition table and report the number of independent groups. For a claim about new objects, each object’s result matters; a pooled percentage can be dominated by many easy trials.
For matched starting conditions, compare methods on paired tasks and examine their paired differences. Confidence-interval overlap by itself is not a significance test of the difference. For clustered data, consider group-level resampling with a justified grouping scheme. Very few groups cannot establish a stable generalization estimate.
Report both micro-average (all successes divided by all trials) and, when appropriate, macro-average (mean of per-condition rates), stating which is primary. In an invented example, 90/100 on an easy object and 0/10 on a hard one yield 81.8% micro-average but 45% equal-object macro-average. Neither number should hide the failed object.
| Synthetic condition | Successes / attempts | Rate |
|---|---|---|
| Easy object | 90 / 100 | 90% |
| Hard object | 0 / 10 | 0% |
| All trials weighted equally | 90 / 110 | 81.8% |
| Objects weighted equally | (90% + 0%) / 2 | 45% |
Read the primary work: Diffusion Policy: per-task and multi-seed evaluation ↗
Build a result another lab can audit
Save the source commit, checkpoint identifier, dependencies, configuration, dataset split and preprocessing statistics. Add hardware/controller revisions, camera calibration and timing traces for physical work. Record exact launch commands in your own repository after validating them; do not imply a generic tutorial command was run.
Keep representative failures and a complete outcome ledger, subject to the dataset’s permissions. Separate author-reported results from your reproduction. For a learned world model, attach both the prediction experiment and the downstream task experiment, including the no-model baseline.
The six core classes and three applied workshops can become one capstone: define a task, specify its observation/action contract, select a baseline, design a split, measure timing and write an evaluation card. A complete plan is valuable before training, but mark “not run” fields explicitly. Completion of reading is not a certified skill or a validated hardware integration.
claim: <one measurable manipulation claim>
evidence: not_run | simulation | physical_trial
source_commit: <exact revision>
checkpoint: <identifier and terms>
data_split: <manifest and grouping unit>
embodiment: <hand, controller, sensors, calibration>
success_rule: <condition, timeout, intervention policy>
results: <all trials, per-condition counts, uncertainty>
failures: <logs and representative examples>
limitations: <untested conditions and missing artifacts>Read the primary work: DexWM: distinct prediction and task evaluations ↗
Write the conclusion the data support
Your toy study obtains 18 unassisted successes from 20 attempts on one object in one session. Write a defensible conclusion and identify the next experiment.
Reveal the worked answer
“Under the documented single-object, single-session conditions, 18/20 attempts succeeded without intervention. The approximate 95% Wilson interval is 69.9–97.2% under an independent common-probability trial model. This does not establish generalization across objects or sessions.” Next, add predeclared held-out groups, a matched baseline and failure analysis. If trials are correlated, the binomial interval alone is insufficient.
Self-review checklist
- Reports successes and all attempts, with intervention policy.
- States the statistical assumptions and narrow evidence scope.
- Does not infer broad reliability from one object/session.
- Specifies new held-out groups and a reproducible baseline.
Original Dexhands teaching examples. This is a self-study rubric, not automated grading or certification. Research sources reviewed September 25, 2026; examples do not report experiments run by Dexhands.
Try it yourself
Your practical task
- Choose one precise manipulation claim and a held-out split.
- Define success, timeout, assistance and the baseline before evaluating.
- Create a record containing revisions, all trial outcomes, failures and permission to share artifacts.
What to produce: A reproducibility brief and a blank trial ledger ready for real results.
Check your understanding
Original sources & next steps
Diffusion Policy: configurations, logs and multiple seeds ↗DexWM: separate rollout and task evaluation ↗DWM: released train/test split files ↗Original Dexhands teaching material. Lesson and linked references reviewed 2026-09-25. Research links are not endorsements or evidence of hardware compatibility.