CAAT: Contact-Aware Attention Scaling and Tactile Masking for
Data-Efficient Contact-Rich Manipulation

Jiaming Jiang1,3*, Yuzhe Huang2,3*, Hao Liang3, Pei Lin1,3, Shengcheng Luo1,3,
Fanrong Dong4, Jiaping Wu5, Chenxi Xiao1†, Wanlin Li3†, Ziyuan Jiao2,3†

1ShanghaiTech University   2Beihang University   3Beijing Institute for General Artificial Intelligence
4Zhejiang University   5Beijing University of Posts and Telecommunications

* Equal contribution    † Corresponding authors

CAAT overview and performance comparison

CAAT improves data-efficient visuo-tactile policy learning by explicitly reasoning about when to see and when to feel.

Abstract

In contact-rich manipulation, visual observations primarily guide motion in free space, whereas tactile observations become particularly informative during contact. However, standard Transformer-based visuo-tactile policies typically rely on either token concatenation or learnable gating. These approaches lack explicit contact-aware priors, making it difficult to efficiently learn effective cross-modal representations from demonstrations.

To address this limitation, we propose CAAT, a lightweight contact-aware framework that explicitly incorporates contact priors through attention scaling and dynamic tactile masking. Specifically, CAAT emphasizes visual information before contact and tactile information during contact. It also suppresses static background tokens by comparing the current tactile observation with a non-contact reference.

CAAT can be integrated into commonly used Transformer-based policies without modifying their action decoders. In simulation, integrating CAAT with ACT improves the average success rate by 18.0 percentage points over direct visuo-tactile fusion and by 10.0 percentage points over gated fusion. In real-world experiments using a visuo-tactile UMI platform, CAAT achieves an average success rate of 60.0% across ACT, Diffusion Policy, and π0.

Method Overview

CAAT method pipeline

Visual observations and robot states are encoded as modality-specific tokens, while tactile images are converted into patch tokens using a pretrained tactile encoder. CAAT first applies Dynamic Tactile Masking, which compares current tactile tokens with a non-contact reference and suppresses unchanged background regions. The visual, masked tactile, and state tokens are then processed by Transformer layers.

For action decoding, CAAT computes modality-specific attention readouts and scales the visual and tactile contributions according to the estimated contact state: visual features are emphasized before contact, whereas tactile features are emphasized during contact.

Videos

Real-world Experiments

Draw Power

Lift Bottle

Open Box

Simulation Experiments

Lift Bottle

Lift Can

Hardware Design

Visuo-tactile UMI hardware design

Our visuo-tactile UMI device is designed for both human demonstration collection and robot policy deployment. The two-finger gripper integrates tactile sensors at the fingertips, enabling contact-rich manipulation with an isomorphic hardware setup.

BibTeX

@misc{jiang2026caat,
  title  = {CAAT: Contact-Aware Attention Scaling and Tactile Masking
            for Data-Efficient Contact-Rich Manipulation},
  author = {Jiaming Jiang and Yuzhe Huang and Hao Liang and Pei Lin
            and Shengcheng Luo and Fanrong Dong and Jiaping Wu
            and Chenxi Xiao and Wanlin Li and Ziyuan Jiao},
  year   = {2026},
  note   = {arXiv preprint}
}