Latent shape representation for tactile exploration

We estimate 3D geometry from sparse grasp measurements by training a flow matching model inside the 32-dimensional latent space of an occupancy VAE. Contact points, fingertip poses, or joint angles alone are each sufficient to decode plausible shapes.

Explicit geometric representations such as voxel grids and meshes scale poorly with resolution, which makes them expensive to store, process and generate. Tactile exploration is a particularly awkward case, since it couples a sparse input domain — on the order of ten contact points per grasp — to a dense, high-frequency output domain. We decouple the two with a compact latent representation: a variational autoencoder encodes occupancy at query positions, and a flow matching model maps grasp measurements to shape latents within that space.

Dataset

We use simulated grasps of the DLR Hand II on a subset of ShapeNet classes. Following the Acronym preprocessing, meshes are made watertight with Manifold, yielding 108,014 grasps on 14,507 shapes. Grasps scoring below a success threshold of 10 are discarded. Per grasp we extract hand–mesh collision centres and normals as simulated tactile readings, fingertip positions and orientations, and the hand configuration, consisting of the base transform and joint angles. Per mesh we sample 512 query points uniformly within a surrounding cube and 512 distributed near the surface, storing position and occupancy for each.

Occupancy VAE

Encoder and decoder are MLPs with four hidden layers of width 256 projected to a bottleneck of 32, with max pooling over the feature dimension following PointNet and sinusoidal positional encoding. The objective is binary cross-entropy on occupancy plus a KL term whose weight is ramped linearly to ensure convergence.

Depth dominates the sweep: the four-layer variant reduces BCE from 0.161 to 0.122 and prediction entropy from 0.0166 to 0.0097. Positional encoding variants — sinusoidal, Fourier, learnable, and raw coordinates — differ only marginally. A reprojection loss inspired by LeJEPA and a cross-view scheme that encodes and decodes different point sets both increased training time substantially without improving results. Linear interpolation between validation latents decodes to plausible intermediate shapes, a property the conditional model depends on.

Unconditional latent denoising

To select a generative model for this latent space we compared MLP, ResMLP, 1D UNet and DiT with per-dimension and per-latent tokenization, trained under DDPM with ε- and v-prediction, EDM, and flow matching, and evaluated with maximum mean discrepancy, coverage and density.

Architectures with spatial inductive bias underperform. DiT with per-latent tokenization does not reach a usable model, and the 1D UNet trails a plain ResMLP. The VAE latent is continuous and compact, with none of the grid or sequence structure that transformers and UNets are designed to exploit. The best configuration is flow matching on a ResMLP of hidden dimension 512 with FiLM timestep conditioning.

Conditioning

FiLM conditioning is extended by a linear projection of the encoded grasp information. We evaluate three signals of decreasing information content.

• Contacts, with optional normals, encoded either by a PointNet-style encoder trained jointly with the flow model, or by the frozen VAE encoder with a learned linear map to the conditioning dimension. The VAE encoder expects a fixed point count, so remaining slots are filled with random positions labelled unoccupied; normals are represented by an additional empty point placed at a fixed offset along each normal.
• Fingertips, with optional orientation. The point count is fixed here, so a direct linear mapping is available in addition to the PointNet and VAE encoders.
• Hand configuration: base position and orientation plus three joint angles per finger, an 18-dimensional vector mapped by an MLP. This variant requires no knowledge of the forward kinematics.

Results

All conditioning variants decode plausible shapes, and similarity to ground truth orders with information content: contacts outperform fingertips, and normals improve on bare positions. Conditioned on hand configuration alone the model still reaches an IoU of 0.382, indicating that it implicitly learns the forward kinematics. Contacts with normals encoded by PointNet gives the strongest joint performance across generative and surface metrics.

Limitations and future work

The reported accuracy depends heavily on the shape prior learned by the VAE, which limits generalization: on geometry outside the training distribution, reconstructions collapse toward familiar shapes and the tactile evidence does not override the prior. Our dataset covers few ShapeNet classes, most of them approximately rotationally symmetric, which narrows that prior further. A more diverse dataset would allow higher shape fidelity and variance. On the architecture side, conditioning mechanisms other than FiLM and constrained latent spaces such as quantized VAEs are promising directions for improving latent space structure.

This page provides a high-level overview of the project. For detailed information about the dataset, training setup, experimental evaluation, results, and future work, please refer to the full report.