DexDeform
DexDeform research team
Dexterous deformable-object manipulation with human demonstrations and differentiable physics.
What you can explore
Explore simulated shaping tasks, demonstration loading and trajectory optimization.
Before you use it
The README includes incomplete release tasks. Do not assume every paper component is turnkey or that simulated deformable materials match real ones.
Original contributors
Sizhe Li, Zhiao Huang and collaborators
License / access: MIT · check model, data and dependency terms separately
DexDeform: load a soft-object scene, replay a demonstration and inspect the score
- Environment
- CUDA simulator + PyTorch / Jupyter
- Embodiment
- Released Shadow-hand soft-object simulation
What you need
Use the project environment.yml in a separate environment and record the commit, GPU, CUDA and PyTorch versions. The reviewed file includes Linux-specific package builds and CUDA dependencies; it is not a portable Mac installation recipe. Install the editable package and the documented shape-distance dependencies before opening a notebook.
First useful check
Load the folding environment and save one initial render before replaying actions or optimizing a trajectory.
A small first experiment
Open the files that actually exist
At the reviewed commit, environment loading is tutorials/2_environment_loading.ipynb and trajectory optimization is tutorials/1_trajectory_optimization.ipynb. The README lists the opposite numbering. Start Jupyter from the repository root and use the file tree links below. Before running cells, replace author-specific /home/... paths with your own data and output locations.
Environment-loading notebook →Inspect the scene before changing its dynamics
Create the folding task, render the initial material shape and inspect the hand from multiple views. The notebook obtains the action width from env.simulator.torch_action_scale; use that value instead of assuming it matches the number of visible joints. Record the task configuration, initial state and random seed. Run random actions only in the simulator.
Folding material and scene configuration →Replay one trusted demonstration
Use tutorials/3_demonstration_loading.ipynb. Set demo_dir to your downloaded task folder and choose an existing file explicitly: the example indexes demo_files[5], which fails with fewer than six files. The loader uses serialized demonstration files; only load files from a source you trust. Check states and actions are non-empty, use the demonstration’s initial state, and match env_name to that demonstration before stepping through its actions.
Demonstration replay notebook →Separate a shape score from task success
The scoring notebook selects CUDA device 4; change this before importing the GPU stack if your machine has a different visible device set. Its random target cloud demonstrates an API, not a task goal. Evaluate against the intended target shape in the same coordinates and units. Keep the Sinkhorn settings, particle sampling and task protocol fixed when comparing runs; also inspect whether the object stays grasped and reaches the intended shape.
Shape-distance example →Optimize only after replay is understood
The trajectory notebook uses lift_box and optimizes an action sequence through differentiable physics. Save the starting state, loss definition and before/after trajectory. Replaying a saved demonstration, optimizing one trajectory and training the full learned method are three different results. For a useful report, name which one you completed and retain failed rollouts as well as the best render.
Trajectory-optimization notebook →Make the result reproducible
Attach your task, code revision, environment, input-file provenance, rendered output and exact changes to paths or devices. Keep setup errors separate from manipulation failures. Use the linked checklist to plan a controlled comparison rather than compare videos with different materials or initial states.
Reproduction checklist →
What to record
- A scene loads and renders before any optimization is attempted.
- A replay starts from the demonstration state and uses the matching task and action shape.
- The target shape, metric settings and failure cases accompany the result.
Common problems and what to check
- A notebook link is missing
- Use the numbered files at the pinned commit; the README numbering is stale.
- No demonstrations or an index error
- Confirm the local folder and available filenames before selecting a demonstration; do not assume a sixth file exists.
- CUDA device ordinal error
- Review the notebook’s device selection and the devices visible to your process before importing CUDA libraries.
- A low loss looks like success
- Check the real task goal and the rendered trajectory. The scoring tutorial’s random target is illustrative, not a benchmark target.
Where it stops
This is a simulation workflow, not a physical Shadow Hand controller. The release still notes unfinished data-loading cleanup and teleoperation synchronization. A rendered example does not reproduce the full paper or establish material fidelity.
Keep exploring
DexMV
Simulation and retargeting tools for learning dexterous manipulation from human videos.
ManiSkill
SAPIEN-based manipulation simulation with GPU-parallel environments and robot-learning tools.
Editorial reference · checked 2026-09-22