Build a demonstration dataset that survives an audit
More video is not automatically more learning signal. Design the episode, labels, coverage and evaluation split before collecting at scale.
Define provenance, collection coverage and a leakage-resistant split.
Start with the intended claim
Specify the task and the variation the policy should handle: objects, operators, sessions, camera placements and hardware revisions. A claim about unseen objects requires held-out objects. A claim about later sessions requires a temporal or session split. Random frame splits answer neither question.
Distinguish human motion from robot action
EgoDex provides egocentric video and estimated body/hand poses; its annotations are not robot torque labels. DexCap and DexUMI include explicit steps for transferring human demonstration information into robot learning. Decide whether each field is measured, estimated, commanded or transformed.
Preserve raw and derived provenance
Keep the source episode identifier, task, hand revision, calibration, timestamps, preprocessing version and permissions. Derivatives such as crops, retargeted trajectories or inpainted frames should point back to the source episode. This lets you reproduce a failure and prevents derivatives of one scene from leaking into another split.
Keep missingness and failures visible
A missing confidence field is not perfect confidence. A tracking failure is not a zero joint angle. Store validity masks and a reason for rejecting an episode. Preserve failures in the outcome ledger even if a training recipe uses only successful demonstrations; otherwise the published success denominator changes silently.
Validate labels, not just file formats
Check transforms using known poses, pair video and annotations, and inspect actions against measured states. EgoDex documents a session-dependent world frame and potentially imperfect automated language annotations. Semantic errors can remain even when every file opens and every array has the expected shape.
Separate training growth from test independence
Overlapping action windows and generated variations increase training examples, not the count of independent operators or real-world sessions. DexMimicGen is a concrete example of generated simulation trajectories. Preserve the seed lineage and keep evaluation scenes and seed families separate where the claim requires independence.
Work through the mechanics
Create a dataset card, provenance tree and split that can support a specific generalization claim.
Prerequisites: Episode boundaries, action windows and the evaluation lesson.
Read the fields before choosing the dataset
EgoDex is useful for studying egocentric manipulation and human motion. Its release pairs video with estimated poses and documents the coordinate frame and optional confidence fields. The session origin can change across recordings. Those poses need interpretation and potentially retargeting before they become robot learning targets; they are not measured robot motor commands.
DexUMI demonstrates a pipeline that addresses human-to-robot kinematic and visual differences. DexCap likewise combines wearable capture with robot imitation. These methods help explain how human demonstrations become training data, but the processing and supported hardware are part of the result. Treat inpainted images, inferred poses and replayed actions as derived fields with explicit provenance.
DexMimicGen releases simulation environments and generated demonstration data. The research also studies a real-to-simulation-to-real workflow; distinguish a paper experiment from the contents of the downloadable dataset. Availability of code, data and weights should be checked separately for every project.
| Case study | Useful learning signal | Boundary to preserve |
|---|---|---|
| EgoDex | Video, estimated hand/body poses and language annotations | Human observations; automated labels and session frames need checking |
| DexUMI / DexCap | A documented bridge from human demonstration to robot learning | Embodiment and preprocessing affect the action labels |
| DexMimicGen | Generated simulation trajectories with reproducible environments | Seed-derived examples are not independent real-world collections |
Read the primary work: EgoDex: fields, coordinate frames and dataset terms ↗ · DexUMI: hardware, data processing and training ↗ · DexCap: wearable capture and robot imitation ↗ · DexMimicGen: simulation data and reproduction artifacts ↗
Count windows without inflating the evidence
Suppose every episode contains T aligned timesteps. For this teaching convention, a sample takes K observations ending at t and H action targets starting at t. Valid start indices run from K − 1 through T − H. With stride s, the number of windows is max(0, floor((T − K − H + 1)/s) + 1). Other libraries may use different offsets, padding or terminal masks; record their exact convention.
For 20 invented episodes with 100 timesteps each, K = 2, H = 8 and stride 1, each episode yields 92 windows and the set yields 1,840. That is still 20 episodes. If those episodes came from four sessions, the independent session coverage is still four. The calculator exposes why multiplying windows cannot manufacture new experimental conditions.
Create the split using the intended independent groups before deriving overlapping windows or augmentations. Keep every child of a source episode in its assigned split. If you promise unseen-object evaluation, prevent that object identity from crossing splits even when it appears in multiple sessions. Some claims require multiple held-out axes rather than one convenient group key.
w = max(0, floor((T − K − H + 1) / s) + 1) Total training windows = episodes × w More windows ≠ more independent episodes
Training examples versus independent evidence
Synthetic teaching example · calculations only. No inference, simulation or hardware execution.
92 windows per episode × 20 episodes. Split the source groups before windowing; this number is not an independent trial count.
Equal-length, complete episodes only. Does not infer the number of objects, sessions or independent experimental conditions.
Read the primary work: DexGraspVLA: the released action and observation schema ↗ · DexMimicGen: simulation data and reproduction artifacts ↗
Build an auditable release and a coverage plan
Write a manifest with source_episode_id, session_id, object_id, operator_id where permitted, hand_revision, split, calibration_id, preprocessing_commit and derived_from. Add an outcome ledger with success rules and exclusion reasons. A quality filter should produce a documented decision, not quietly delete inconvenient evaluation failures.
Prioritize collection by missing task conditions, not video duration alone. For example, a dataset can have many easy grasps yet no releases, no recoveries and no occluded contacts. Define a table of required conditions, count independent episodes and groups, then collect the gaps. There is no universal minimum episode count that guarantees a dexterous policy will work.
Access is not the same as unrestricted reuse. At review, the EgoDex README identifies CC-BY-NC-ND dataset terms; DexMimicGen distinguishes NVIDIA code terms from CC-BY 4.0 data. Read the actual terms for your intended use, including derivatives and commercial work. A code license does not automatically cover recordings, model weights, people or third-party assets.
claim: <unseen objects, later sessions, or another specific axis>
raw_episode: <immutable source ID>
lineage: <source → retargeting → crop/inpainting → windows>
fields: <measured | commanded | estimated | transformed>
clock_and_frames: <synchronization, origin and calibration>
split: <group rule and frozen manifest>
quality: <validity masks, label checks and exclusions>
coverage: <conditions, episodes, sessions and objects>
permissions: <code, data, weights and asset terms separately>
known_gaps: <unrecorded conditions and unresolved errors>Read the primary work: EgoDex: fields, coordinate frames and dataset terms ↗ · DexMimicGen: simulation data and reproduction artifacts ↗ · DexUMI: hardware, data processing and training ↗
Fix a dataset that looks larger than it is
A team takes 20 episodes from four sessions, derives 1,840 overlapping windows and randomly puts windows into train and test. It calls this 1,840 independent demonstrations and claims new-object generalization. What should change before training?
Reveal the worked answer
Report 20 episodes, four sessions and 1,840 derived windows separately. Split the source groups before windowing and keep all derivatives together. Define and hold out object identities for the new-object claim; a session split alone is insufficient if the same objects appear in both sets. Fit preprocessing statistics on training data only and freeze the test manifest. All counts here are synthetic.
Self-review checklist
- Reports raw collection and derived window counts separately.
- Keeps source episodes and their derivatives within one split.
- Matches the held-out axis to the new-object claim.
- Preserves a frozen manifest, training-only statistics and exclusions.
Original Dexhands teaching examples. This is a self-study rubric, not automated grading or certification. Research sources reviewed September 25, 2026; examples do not report experiments run by Dexhands.
Try it yourself
Your practical task
- Try different history and horizon settings in the window calculator.
- Write the group split before deriving windows.
- List measured, inferred and transformed fields and their permissions.
What to produce: A dataset card with a frozen split rule and provenance map.
Check your understanding
Original sources & next steps
EgoDex: fields, coordinate frames and dataset terms ↗DexCap: wearable capture and robot imitation ↗DexUMI: hardware, data processing and training ↗DexMimicGen: simulation data and reproduction artifacts ↗Original Dexhands teaching material. Lesson and linked references reviewed 2026-09-25. Research links are not endorsements or evidence of hardware compatibility.