DexPoint
DexPoint research team
Point-cloud reinforcement learning for generalizable dexterous manipulation.
What you can explore
Inspect the released environments, point-cloud inputs and training workflow.
Before you use it
Use the authors’ simulator and dependency versions. Reported sim-to-real results do not validate another hand or camera setup.
Original contributors
Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Hao Su and Xiaolong Wang
License / access: MIT · check model, data and dependency terms separately
DexPoint: inspect point-cloud observations before training
- Environment
- Python 3.8 recipe; SAPIEN 2.1.0 / Gym 0.25.2
- Embodiment
- Released Allegro relocation environment; not an arbitrary hand driver
What you need
Use a separate research environment. The released setup.py pins SAPIEN 2.1.0 and Gym 0.25.2; do not substitute current simulator or Gymnasium releases without checking the API. Obtain the scene texture and the appropriate object assets from the authors. Record their download terms separately from the code license.
First useful check
Run the state-only example, then the point-cloud example. A successful reset and a plausible cloud are useful setup evidence; neither is a reproduced policy result.
A small first experiment
Use the actual repository and example paths
Clone the author repository and enter dexpoint-release. The reviewed README contains a cd dexart-release typo and calls the state example state_only_env.py; the released file is example/example_use_state_only_env.py. Follow the upstream installation, then use the actual file below from the repository root.
State-only example at the reviewed revision →cd dexpoint-release python example/example_use_state_only_env.pyCheck the rendering assets before blaming the policy
The scene texture belongs at assets/misc/ktx/day.ktx. ShapeNet assets are an additional requirement for the documented training route. A missing file, graphics-driver problem or failed renderer is an environment issue; changing rewards cannot fix it. Keep a minimal example log before attempting long runs.
Author installation and asset instructions →Inspect the observed cloud and its coordinate frame
After installation, run the visual example below. It prints observation/action spaces and opens a point-cloud viewer, so an interactive display is needed for that viewer even though the simulation sets use_gui=False. The configured cloud uses the robot frame with x forward and workspace cropping. Confirm orientation with the drawn axes instead of treating XYZ as camera coordinates.
Point-cloud input and viewer code →python example/example_use_pc_env.pyAdd a small observation sanity check
Insert this original read-only check immediately after obs = env.reset() in a local copy of the visual example. The reviewed relocate_noise configuration requests 512 observed points. Log the shape and axis bounds; repeated or collapsed points deserve inspection but this check does not establish correct calibration or a successful grasp. No robot is connected by this snippet.
Camera, point-count and observation settings →import numpy as np # Insert after obs = env.reset() in the visual example. cloud = np.asarray(obs["relocate-point_cloud"]) if cloud.ndim != 2 or cloud.shape[1] != 3 or len(cloud) == 0: raise ValueError("Expected a nonempty N-by-3 XYZ cloud") if not np.isfinite(cloud).all(): raise ValueError("Cloud contains NaN or infinity") print("XYZ shape:", cloud.shape) print("Axis minimum:", cloud.min(axis=0)) print("Axis maximum:", cloud.max(axis=0)) print("Unique points:", len(np.unique(cloud, axis=0))) for key in ("imagination_robot", "imagination_goal"): print(key, np.asarray(obs[key]).shape if key in obs else "not enabled")Separate measured geometry from imagined geometry
The imagination demonstration enables both robot and goal samples. The paper-style grasping example enables robot-only imagination. Compare these two configurations before choosing an input tensor: goal geometry can change what information the policy receives. Imagined robot points come from the robot model; they are not extra depth measurements.
Paper-style grasping observation setup →Choose the evaluation question before the PPO run
Follow the training link in the author README for its DexArt-based training implementation. Save the selected task, object split, camera configuration, seed and evaluation criterion. First verify whether your experiment uses robot-only imagination or a custom variant; do not compare a modified input setup with a published result as if the protocols matched.
Build a reproducible policy evaluation →
What to record
- Record the DexPoint commit, simulator versions, object assets and exact example file.
- Save observation keys, cloud shape, coordinate-frame notes and a viewer screenshot.
- Report setup success separately from policy success, with failures and changes to the original protocol.
Common problems and what to check
- The installation directory does not exist
- Use dexpoint-release, not the stray dexart-release path in the README. Read setup.py for the dependencies of this release.
- The cloud viewer fails on a headless server
- The Open3D visualization still needs a display. Separate the renderer/data check from the interactive viewer rather than assuming use_gui=False disables every window.
- The cloud looks mirrored or shifted
- Inspect robot/camera transforms, x-forward convention and workspace cropping. Do not repair the display with an unexplained sign flip.
- A policy expects a different observation dictionary
- Compare the visual and imagination configurations. The example with goal points is not identical to the robot-only setup used by the paper-style grasping script.
Where it stops
This guide reviews the public release; we have not run its GPU simulation or real robot. The paper’s sim-to-real results apply to its reported setup, not every depth camera or dexterous hand. A modern SAPIEN environment may require a deliberate port rather than a package upgrade.
Keep exploring
DexMV
Simulation and retargeting tools for learning dexterous manipulation from human videos.
DexArt
A simulated benchmark for learning to manipulate articulated objects and testing generalization to unseen objects.
DexGraspBench
A MuJoCo benchmark for comparing open-loop grasp poses and trajectories using explicit simulation metrics.
Editorial reference · checked 2026-10-25