Latency, touch and the loop that actually controls the hand
Understand why a fast model can still react late. Use a timing worksheet to separate computation, sensing and execution.
Calculate an illustrative reaction delay and distinguish sensing changes from action-interface changes.
Measure observation age, not just GPU time
At the moment a command is applied, the scene may have changed since the camera exposure. Trace sensing, preprocessing, inference, communication and queuing as separate stages. Report an end-to-end distribution and tail latency, not only a warmed-up average. Include the timestamps needed to reproduce the calculation and whether clocks share a reference.
Understand the illustrative timing model
The exercise below assumes a serial pipeline and a disturbance occurring uniformly between sensor samples. Mean sample wait is half the sample interval. Add that to preprocessing, inference, transport and queue delays to estimate a mean reaction delay for this simplified case. Changing these numbers does not predict task success, stability or a product’s performance. Real pipelines can overlap stages or trigger asynchronously.
Chunk length and feedback are different decisions
A policy may predict many future actions yet execute only a few before observing again. Prediction horizon, execution horizon and sensor frequency are separate quantities. Holding a long sequence can hide compute latency but delay reaction to an unexpected slip. Async inference and temporal aggregation have their own alignment rules. Compare them at the same task and log when each command was computed and applied.
Touch helps only if it reaches the decision
A tactile sensor may expose images, forces, marker motion or processed features. Identify what is measured and what is inferred. Contact signals must be calibrated and time-aligned with robot state; a vision-trained checkpoint cannot use a new tactile stream just because it exists. Add the processing path and training data, then evaluate whether it reduces the particular failure you care about.
Action representation changes the integration burden
Joint positions require a position-compatible controller. Torques require the appropriate interface and a different control design. Fingertip targets require kinematics and may be infeasible under contact or joint limits. Synergies can reduce action dimensions but also restrict motion. Compare representations with a clear explanation of how they map onto the physical device; dimension count alone is not a measure of dexterity.
Record late and rejected commands
A stale target can still be syntactically valid. Record drops, retries, rejected commands and controller fallbacks alongside successful actions. Test timing first using recorded data or a supported simulation. For physical trials, use the manufacturer’s operating limits and procedures. The browser tool below issues no robot commands and provides no safe operating threshold.
Work through the mechanics
Build a trace that distinguishes model speed from closed-loop responsiveness.
Prerequisites: Sampling frequency, timestamps and action chunks.
Three clocks, three different rates
Sensor acquisition, policy decisions and low-level control can run at different rates. A controller may interpolate targets at high frequency while receiving new action chunks much less often. Quoting the controller rate as the model’s feedback rate obscures how long a policy acts on old information.
At 20 actions per second, executing four actions before taking another policy decision covers 200 ms. A predicted chunk of eight actions covers 400 ms at that rate, but a receding-horizon system might execute only four. Neither number alone tells you the age of the image or the time until a command takes physical effect.
For a serial pipeline and a randomly arriving event independent of sensor phase, average sample wait is half a sample period. At 30 Hz that is 16.7 ms. With 10 ms preprocessing, 80 ms inference, 10 ms transport and 10 ms queueing, the illustrative mean delay to dispatch is 126.7 ms. This excludes actuator response, dynamics and asynchronous overlap.
Mean sample wait = 1000/(2fs) ms
Execution horizon = 1000 × executed_steps/action_rate ms
Serial mean delay = sample wait + processing + inference
+ transport + queueRead the primary work: Diffusion Policy: receding-horizon execution ↗
Measure end-to-end delay on each trial
Capture acquisition time, inference start/end, command dispatch and controller acknowledgement. Where possible log measured response time too. Use a shared monotonic clock or an estimated clock mapping; subtracting unaligned camera and host clocks can yield a precise-looking but meaningless latency.
In an asynchronous system, stages overlap. Build a timeline for each observation-command pair rather than summing service averages. Report the distribution of command age, dropped frames, stale actions, queue length and deadline misses. A fast median can coexist with long stalls.
Do not add per-stage 95th percentiles and call the result an end-to-end 95th percentile. The joint behavior and correlation between stages determine the end-to-end distribution. Pair the actual timestamps first, then calculate the statistic on those paired delays.
# Synthetic aligned timestamps, in milliseconds.
traces = [(0, 110), (100, 235), (200, 320), (300, 470)]
ages = [dispatch - capture for capture, dispatch in traces]
assert all(age >= 0 for age in ages)
print(ages) # [110, 135, 120, 170]
print(sum(ages)/len(ages)) # 133.75 ms
# Four samples illustrate arithmetic; they do not characterize tails.Read the primary work: Real-Time Action Chunking: handling inference delay ↗
Write a stale-action policy, not only a speed target
An expired target may be inappropriate after an object slips or a human moves it. Record which observation produced each command and define an application-specific expiration or revalidation policy. The response to stale data must be designed with the hardware team: holding, stopping and releasing each have different consequences under load.
Chunk blending can reduce discontinuities, but blending incompatible modes can create an invalid command. Apply transformations and constraints in the right representation, and log rejected or modified commands. A watchdog is an engineering mechanism to test, not a proof that a learned controller is safe.
The timing calculator below is for planning a measurement experiment. Replace its assumptions with recorded distributions before making a performance claim. It cannot predict stability, force limits, object damage or successful manipulation.
Read the primary work: MIT Manipulation: force and controller behavior ↗
Explain a misleading 1 kHz claim
A hand servo runs at 1 kHz, but the policy receives 30 Hz images and executes four 20 Hz action targets per update. Does the policy react to a slip within 1 ms? What must the trace show?
Reveal the worked answer
No. The servo frequency does not bound perception-to-policy reaction time. Measure capture, preprocessing, inference, dispatch, queueing and measured response; document action-prefix execution and asynchronous overlap. A separate local tactile reflex would require its own evidence. The settings are a teaching example, not product performance.
Self-review checklist
- Separates sensor, policy and servo rates.
- Calculates the 200 ms executed action horizon.
- Uses paired, synchronized timestamps.
- Defines and tests handling of stale observations and commands.
Original Dexhands teaching examples. This is a self-study rubric, not automated grading or certification. Research sources reviewed September 25, 2026; examples do not report experiments run by Dexhands.
Try it yourself
Explore a timing budget
Interactive concept exercise · no model inference, hardware commands or measured performance.
Illustrative serial pipeline: mean sample wait = 500 ÷ sensor Hz. Reaction delay = sample wait + preprocessing + inference + transport + queue. No stages overlap in this model.
- Mean sample wait: 16.7 ms
- Preprocessing: 10.0 ms
- Inference: 80.0 ms
- Transport: 10.0 ms
- Queue: 10.0 ms
Execution horizon is shown separately, not added to reaction delay. It describes a hold/update interval only under the stated execution assumption. This calculation excludes dynamics and asynchronous overlap.
Sensing and action selections update the integration notes only. They do not change the timing numbers or imply a performance improvement.
Your practical task
- Vary inference delay while holding other inputs fixed.
- Change the sensor sampling rate and execution horizon; explain what each number means.
- Replace assumptions with measured stage timestamps in your own recorded trace.
What to produce: A timing budget with stated assumptions, measurement gaps and proposed sensing/action interfaces.
Check your understanding
Original sources & next steps
Diffusion Policy: prediction and execution horizons ↗ACT: temporal aggregation and action chunks ↗LeRobot: SmolVLA and runtime workflow ↗Original Dexhands teaching material. Lesson and linked references reviewed 2026-09-25. Research links are not endorsements or evidence of hardware compatibility.