DEXTEROUS MANIPULATION

The world of dexterous hands.

Compare hardware. Discover applications. Share what works. An open reference and community for dexterous manipulation.

DexPoint

DexPoint research team

Point-cloud reinforcement learning for generalizable dexterous manipulation.

What you can explore

Inspect the released environments, point-cloud inputs and training workflow.

Before you use it

Use the authors’ simulator and dependency versions. Reported sim-to-real results do not validate another hand or camera setup.

Original contributors

Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Hao Su and Xiaolong Wang

License / access: MIT · check model, data and dependency terms separately

Original code & documentation

FROM REFERENCE TO FIRST EXPERIMENT

DexPoint: inspect point-cloud observations before training

INPUTObject and robot assets + configured simulated camera observations
OUTPUTPoint-cloud observations, simulation rollouts and a separate PPO training path
Environment
Python 3.8 recipe; SAPIEN 2.1.0 / Gym 0.25.2
Embodiment
Released Allegro relocation environment; not an arbitrary hand driver

What you need

Use a separate research environment. The released setup.py pins SAPIEN 2.1.0 and Gym 0.25.2; do not substitute current simulator or Gymnasium releases without checking the API. Obtain the scene texture and the appropriate object assets from the authors. Record their download terms separately from the code license.

First useful check

Run the state-only example, then the point-cloud example. A successful reset and a plausible cloud are useful setup evidence; neither is a reproduced policy result.

A small first experiment

  1. Use the actual repository and example paths

    Clone the author repository and enter dexpoint-release. The reviewed README contains a cd dexart-release typo and calls the state example state_only_env.py; the released file is example/example_use_state_only_env.py. Follow the upstream installation, then use the actual file below from the repository root.

    cd dexpoint-release
    python example/example_use_state_only_env.py
    State-only example at the reviewed revision →
  2. Check the rendering assets before blaming the policy

    The scene texture belongs at assets/misc/ktx/day.ktx. ShapeNet assets are an additional requirement for the documented training route. A missing file, graphics-driver problem or failed renderer is an environment issue; changing rewards cannot fix it. Keep a minimal example log before attempting long runs.

    Author installation and asset instructions →
  3. Inspect the observed cloud and its coordinate frame

    After installation, run the visual example below. It prints observation/action spaces and opens a point-cloud viewer, so an interactive display is needed for that viewer even though the simulation sets use_gui=False. The configured cloud uses the robot frame with x forward and workspace cropping. Confirm orientation with the drawn axes instead of treating XYZ as camera coordinates.

    python example/example_use_pc_env.py
    Point-cloud input and viewer code →
  4. Add a small observation sanity check

    Insert this original read-only check immediately after obs = env.reset() in a local copy of the visual example. The reviewed relocate_noise configuration requests 512 observed points. Log the shape and axis bounds; repeated or collapsed points deserve inspection but this check does not establish correct calibration or a successful grasp. No robot is connected by this snippet.

    import numpy as np
    
    # Insert after obs = env.reset() in the visual example.
    cloud = np.asarray(obs["relocate-point_cloud"])
    if cloud.ndim != 2 or cloud.shape[1] != 3 or len(cloud) == 0:
        raise ValueError("Expected a nonempty N-by-3 XYZ cloud")
    if not np.isfinite(cloud).all():
        raise ValueError("Cloud contains NaN or infinity")
    print("XYZ shape:", cloud.shape)
    print("Axis minimum:", cloud.min(axis=0))
    print("Axis maximum:", cloud.max(axis=0))
    print("Unique points:", len(np.unique(cloud, axis=0)))
    for key in ("imagination_robot", "imagination_goal"):
        print(key, np.asarray(obs[key]).shape if key in obs else "not enabled")
    Camera, point-count and observation settings →
  5. Separate measured geometry from imagined geometry

    The imagination demonstration enables both robot and goal samples. The paper-style grasping example enables robot-only imagination. Compare these two configurations before choosing an input tensor: goal geometry can change what information the policy receives. Imagined robot points come from the robot model; they are not extra depth measurements.

    Paper-style grasping observation setup →
  6. Choose the evaluation question before the PPO run

    Follow the training link in the author README for its DexArt-based training implementation. Save the selected task, object split, camera configuration, seed and evaluation criterion. First verify whether your experiment uses robot-only imagination or a custom variant; do not compare a modified input setup with a published result as if the protocols matched.

    Build a reproducible policy evaluation →

What to record

  • Record the DexPoint commit, simulator versions, object assets and exact example file.
  • Save observation keys, cloud shape, coordinate-frame notes and a viewer screenshot.
  • Report setup success separately from policy success, with failures and changes to the original protocol.
Common problems and what to check
The installation directory does not exist
Use dexpoint-release, not the stray dexart-release path in the README. Read setup.py for the dependencies of this release.
The cloud viewer fails on a headless server
The Open3D visualization still needs a display. Separate the renderer/data check from the interactive viewer rather than assuming use_gui=False disables every window.
The cloud looks mirrored or shifted
Inspect robot/camera transforms, x-forward convention and workspace cropping. Do not repair the display with an unexplained sign flip.
A policy expects a different observation dictionary
Compare the visual and imagination configurations. The example with goal points is not identical to the robot-only setup used by the paper-style grasping script.

Where it stops

This guide reviews the public release; we have not run its GPU simulation or real robot. The paper’s sim-to-real results apply to its reported setup, not every depth camera or dexterous hand. A modern SAPIEN environment may require a deliberate port rather than a package upgrade.

Public source reviewed; observation-check snippet tested with synthetic inputs, not a simulator or hardware reproduction · reviewed 2026-10-25. Record the exact source revision when reproducing it.

Keep exploring

DexMV

Simulation and retargeting tools for learning dexterous manipulation from human videos.

DexArt

A simulated benchmark for learning to manipulate articulated objects and testing generalization to unseen objects.

DexGraspBench

A MuJoCo benchmark for comparing open-loop grasp poses and trajectories using explicit simulation metrics.

Editorial reference · checked 2026-10-25

Start a discussion

Published posts are public. First contributions are reviewed. Only share material you have permission to disclose.

Report a post