Teach a policy to use touch
A camera shows an object. Touch can reveal contact, slip and load—but only if the measurements, timing and learning target agree.
Specify the tactile observation, causal join and matched ablation.
Choose the signal before the model
A binary contact bit, a calibrated force measurement and an image of a deforming sensor surface are different observations. Describe what is measured, where on the hand it is measured, and which quantities are inferred. A tactile image is not a force vector until an appropriate model and calibration support that interpretation.
Separate a representation from a controller
Sparsh learns representations for vision-based tactile sensing. A downstream decoder can use those features for a labeled task such as force estimation or slip detection. This is useful building material for a manipulation policy; downloading an encoder does not provide a complete closed-loop hand controller.
Keep history when contact changes
A single frame may show deformation but not whether the object is beginning to move. Short histories can expose temporal change, at the cost of extra delay and data requirements. Contact transitions, loss of contact and re-contact deserve their own evaluation slices. Do not report only steady grasps.
Align by availability, not convenience
Training must reproduce what was available when the policy chose an action. Record acquisition and receive times, clock synchronization, valid-data masks and preprocessing. The nearest sample in time might be a future sample. A sensor that arrives late cannot explain a decision already made.
Measure the value of touch fairly
Compare vision-plus-state with vision-plus-state-plus-touch under matched splits, task conditions and training budgets. Add a masked-touch diagnostic to the tactile model, while recognizing that masking changes its input distribution. Include conditions where touch is expected to matter, such as visual occlusion, and conditions where it may add little.
Begin with a recorded-data experiment
Inspect one permitted recording, mark contact transitions and design a causal alignment rule. Fit a simple supervised task before making claims about manipulation. Evaluate the learned representation and the final control task separately. This lesson produces an experiment plan and timing audit, not a sensor driver.
Work through the mechanics
Design a causal tactile input pipeline and an ablation that measures its contribution.
Prerequisites: Sensor timestamps, basic supervised learning and the action-interface lesson.
From sensor pixels to a learning target
A tactile learning pipeline has at least three layers: the sensor reading x, an encoder z = f(x), and a task-specific decoder or policy. An optical tactile sensor records appearance changes caused by surface deformation. A force-sensing resistor can provide a load-related electrical response; thresholding it creates a contact bit. Neither has the same information content or calibration as a six-axis force/torque sensor.
Sparsh is a useful encoder case study. Its release includes pretrained representations and downstream task data, including paired tactile/force observations. Self-supervised pretraining and supervised downstream calibration are separate stages. If you use a force decoder, test its units and error on held-out loads, contact locations and sensor conditions; representation quality alone does not establish calibrated force accuracy.
Rotating without Seeing studies a different route: a hand with distributed binary touch and proprioception learns a particular in-hand rotation behavior. This evidence does not mean sparse binary contact can replace optical touch for every task. Choose the sensing modality by the decision you need to make, then compare within that task.
| Observation | Can support | Still needs evidence |
|---|---|---|
| Binary contact | Contact/no-contact patterns over time | Load magnitude and direction are not encoded by one bit |
| Calibrated force signal | Load estimation in defined axes and range | Calibration drift, cross-talk and contact location |
| Optical tactile frames | Learned deformation, slip or contact features | Sensor-domain transfer and labeled downstream validation |
Read the primary work: Sparsh: tactile representations and downstream tasks ↗ · Sparsh: code, labeled tasks and released checkpoints ↗ · Rotating without Seeing: touch-based in-hand rotation ↗
Build a causal, age-aware sensor join
At policy decision time t, select only sensor samples whose acquisition time and receive time are both no later than t, expressed in a common clock. From those valid samples, choose the latest acquisition time. Define age as t minus acquisition time. A sample can be causally available yet too old for your chosen task budget.
In the invented trace below, contact samples were acquired at 80, 95 and 108 ms. With a 10 ms transport delay and a decision at 100 ms, only the 80 ms sample has arrived. Nearest-time alignment would choose 95 ms even though it arrives at 105 ms. Using it during training would give the policy information that online execution did not have.
The calculator assumes constant delay, a common clock and no packet loss; real systems need measured per-sample receive times and validity masks. A freshness threshold is a task-specific hypothesis, not a certified safety limit. Interpolating with a sample that arrives later also violates online causality, even if it makes an offline plot smoother.
Eligible: captureᵢ ≤ t and receiveᵢ ≤ t Select j = argmax eligible captureᵢ Age = t − captureⱼ
Which sample was actually available?
Synthetic teaching example · calculations only. No inference, simulation or hardware execution.
Causally available, but stale under your selected budget.
| Capture (ms) | Arrival (ms) | At decision time |
|---|---|---|
| 80 | 90 | Selected |
| 95 | 105 | Not yet received |
| 108 | 118 | Not yet captured |
Fixed synthetic acquisition times: 80, 95 and 108 ms. Constant delay and synchronized clocks are teaching assumptions.
Read the primary work: DexUMI: hardware, data processing and training ↗ · Sparsh: code, labeled tasks and released checkpoints ↗
Prove the tactile signal changes the result
Design a small matrix before training: visible versus occluded contact, familiar versus held-out objects, and nominal versus changed sensor conditions. Keep the action/controller stack fixed. Train a vision-and-state baseline and a vision-state-touch policy on the same episode split. Report the amount of data and compute used by each.
Evaluate both the downstream representation task and the physical or simulated manipulation task. Low force-prediction error does not establish low drop rate. A shuffled-touch or masked-touch test can expose reliance on spurious correlations, but it is a distribution-shift diagnostic rather than a replacement for training a fair baseline.
For a tactile world-model extension, ask it to predict contact changes conditional on an action, then test whether that prediction changes planning decisions. Compare against persistence and a no-touch model. Counterfactual predictions require coverage of relevant actions; logged outcomes alone do not reveal what every untried action would have done.
| Test | What to record | What it can establish |
|---|---|---|
| Signal prediction | Force/slip label definition, calibration and error | A scoped sensing or prediction result |
| Policy ablation | All attempts, failures and matched task conditions | Whether touch improves the selected task |
| Contact prediction → planning | Action-conditioned prediction and task outcome | Whether a predictor helps decisions, not merely renders plausible contact |
Read the primary work: Sparsh: code, labeled tasks and released checkpoints ↗ · Rotating without Seeing: touch-based in-hand rotation ↗
Repair a future-leaking tactile dataset
A policy decides at 100 ms. The loader uses the 95 ms tactile frame, which arrives at 105 ms. The next experiment reports excellent slip prediction but never measures object drops. Identify the two independent gaps and propose a minimal correction.
Reveal the worked answer
The 95 ms frame is unavailable at decision time. Choose the latest valid sample already received, record its age and mask or handle staleness according to the contract. Slip prediction is a sensing result, not a manipulation result: add a matched no-touch baseline and a predeclared task-success/drop criterion over all attempts. The timestamps are synthetic.
Self-review checklist
- Uses acquisition and receive times in a common clock.
- Distinguishes stale observations from missing observations.
- Separates prediction quality from manipulation success.
- Defines matched conditions and a complete outcome denominator.
Original Dexhands teaching examples. This is a self-study rubric, not automated grading or certification. Research sources reviewed September 25, 2026; examples do not report experiments run by Dexhands.
Try it yourself
Your practical task
- Change decision time and transport delay in the technical workshop.
- Record the chosen sample, age and stale/missing rule.
- Define the no-touch baseline and one held-out contact condition.
What to produce: A sensor contract and tactile-ablation experiment card.
Check your understanding
Original sources & next steps
Sparsh: tactile representations and downstream tasks ↗Sparsh: code, labeled tasks and released checkpoints ↗Rotating without Seeing: touch-based in-hand rotation ↗Original Dexhands teaching material. Lesson and linked references reviewed 2026-09-25. Research links are not endorsements or evidence of hardware compatibility.