TouchWorld: A Predictive and Reactive Tactile Foundation Model for Dexterous Manipulation

Jianyi Zhou^1,2*, Feiyang Hong^1,2*, Yunhao Li^1,2*, Yicheng Zhao^1,2, Yongjue Cen^1,2, Zirui Liu^1,2, Jiakang Huang^1,2, Zirui Chen^1,2, Ruiyang Zhang^1,2, Weizhuo Zhu^1,2, Xuhua Song^1,2, Shuo Yang^1,2†

¹Harbin Institute of Technology, Shenzhen²PHANES AI^*Equal contribution^†Corresponding author

Paper

Abstract

Dexterous manipulation in everyday environments requires both anticipation and reaction: a robot must predict how contact should evolve while rapidly correcting local errors caused by slip, misalignment, unstable grasping, or force mismatch. Vision and language provide semantic and geometric guidance, but they cannot reliably reveal hidden contact states such as force, slip, and contact stability. Although tactile sensing exposes these physical cues, most existing policies treat touch as a low-frequency observation stream within a monolithic action model, coupling slow task reasoning, action generation, and fast contact feedback in a single loop.We introduce TouchWorld, a predictive-and-reactive tactile foundation model for dexterous manipulation. TouchWorld uses a hierarchical policy that separates vision-language subtask planning, tactile world-model prediction, visuo-tactile goal-conditioned action generation, and high-frequency tactile residual refinement. A High-Level Planning Layer produces executable subtasks and predicts tactile subgoals; a Visuo-Tactile Goal-Conditioned Policy generates nominal action chunks; and a Tactile-Conditioned Refinement Policy performs online residual correction using recent tactile and proprioceptive feedback. By using touch as both a predictive contact reference and a fast feedback signal, TouchWorld preserves the semantic generalization of vision-language-action policies while improving local contact adaptation. Across six long-horizon and contact-rich dexterous manipulation tasks, TouchWorld achieves 65.0% success in the clean setting and 53.7% success under human perturbations, outperforming the strongest baseline by 15.7 and 18.5 percentage points, respectively.

Predictive and Reactive Tactile Policy

TouchWorld uses touch in two complementary ways: a predictive pathway anticipates future contact-aware goals, and a reactive pathway corrects local execution errors online. This keeps semantic reasoning, predictive goal generation, nominal action generation, and tactile feedback correction on separate time scales.

Figure 1: Conceptual overview of TouchWorld. The high-level planning layer predicts executable subtasks and visual-tactile subgoals, while the downstream policies generate nominal actions and high-frequency tactile refinements.

TouchWorld System

Figure 2: TouchWorld architecture. The subtask planner, tactile world model, goal-conditioned policy, and tactile refinement policy operate at separate semantic, action, and control-loop time scales.

Robot Hardware and Tactile Interface

TouchWorld is evaluated on a humanoid platform equipped with Wuji dexterous hands and a JQ-Industries tactile glove. The teleoperation side uses a Meta Quest headset, Meta Quest Touch Plus controllers, and a Wuji Glove to collect synchronized visual, proprioceptive, action, and tactile demonstrations.

Figure 3: Hardware platform for TouchWorld. The human teleoperation stack collects visual and hand-motion inputs, while the robot platform executes dexterous manipulation with tactile feedback.

Tactile World Model Prediction

The Tactile World Model predicts future visual-tactile subgoals that describe the expected contact outcome of the current subtask. These predictions serve as contact-aware references for downstream action generation.

Tactile-Conditioned Refinement

The Tactile-Conditioned Refinement Policy operates faster than the nominal VLA policy. At each refinement step, it reads a sliding nominal-action lookahead window, recent tactile histories, and proprioception, then predicts a residual action correction.

Experiments

We evaluate TouchWorld on six real-robot tasks: Water Flower, Tabletop Clearing, Cup Insertion, Power Plug Insertion, Pot Wiping, and Tissue Pulling. Each task is evaluated in both a clean setting and a human perturbation setting.

Figure 4: Real-robot task suite for evaluating TouchWorld. The tasks cover long-horizon planning, precision insertion, continuous contact regulation, soft-object handling, and recovery from human perturbations.

Loading trajectory samples...

Figure 5: TouchWorld trajectory sample explorer.

We also show representative teleoperated collection rollouts from the dataset. These clips are human-collected demonstrations rather than model inference outputs.

Figure 6: Representative dataset collection rollouts. The clips are teleoperated demonstrations used for data collection, not model inference results.

Benchmark Results

TouchWorld consistently improves over Pi-0.5, FTP-1, and GR00T N1.7 across both clean and human-perturbed rollouts. The gains are especially clear on Power Plug Insertion, Pot Wiping, and Tissue Pulling, where tactile prediction and fast local correction are most important.

Table 1Per-task manipulation success rates (%)

Method	Water Flower	Tabletop Clearing	Cup Insertion	Power Plug Insertion	Pot Wiping	Tissue Pulling	Avg.
Clean Setting
Pi-0.5	52	66	36	12	39	39	40.7
FTP-1	56	60	48	32	57	43	49.3
GR00T N1.7	50	58	33	18	36	41	39.3
TouchWorld	72	76	66	45	70	61	65.0
Human Perturbation Setting
Pi-0.5	34	44	24	6	28	30	27.7
FTP-1	39	42	34	20	42	34	35.2
GR00T N1.7	32	36	21	9	26	32	26.0
TouchWorld	60	62	52	35	57	56	53.7

Best results are shown with filled emphasis; second-best results are underlined. TouchWorld improves the strongest baseline by 15.7 points in the clean setting and 18.5 points under human perturbations.

Figure 7: Stacked ablation results. Each bar reports average success with task-level contributions, comparing clean rollouts against human-perturbed rollouts.

Tactile World Model Evaluation

We visualize held-out Tactile World Model predictions by comparing the generated future visual-tactile subgoal with the corresponding ground-truth subgoal over the same subtask segment.

Loading Tactile World Model prediction demos...

Figure 8: Tactile World Model prediction comparison.

Table 2Tactile World Model prediction accuracy

Method	Temporal Contact Acc.	Contact IoU	Volumetric IoU
Current tactile copy	70.4	31.8	24.6
Nearest-neighbor subgoal	77.5	39.2	31.0
Tactile World Model	86.3	52.7	43.8

Prediction metrics are evaluated on held-out subgoal segments; all values are percentages and higher is better.

Vision-Language Subtask Planner Analysis

The Subtask Planner receives the task instruction, current visual observations, and high-level memory, then emits an executable subtask for downstream policy conditioning. The memory-augmented planner improves subtask accuracy, execution success, and transition consistency.

Table 3Vision-Language Subtask Planner evaluation

Planner	Subtask Acc.	Execution Success	Transition F1
Zero-shot Qwen3-VL-4B	43	34	62
Zero-shot Qwen3-VL-32B	69	54	71
SFT Qwen3-VL-4B w/o Memory	73	60	76
Memory-Augmented SFT Qwen3-VL-4B	88	65	82

Memory-augmented planning improves subtask correctness, downstream execution success, and phase-transition consistency.

Qualitative Inference Analysis

TouchWorld decomposes each instruction into executable intermediate subtasks and predicts tactile subgoals that provide contact-aware references for downstream action generation.

The predictive pathway supplies contact-aware subgoals for stable long-horizon execution.
The reactive pathway corrects local contact errors caused by slip, perturbation, or misalignment.

Figure 9: Qualitative inference demonstration of TouchWorld. For each task, the model progresses from the initial scene through executable subtasks and predicts tactile subgoals for contact-aware manipulation.

Limitations & Future Directions

Task diversity

Our real-robot evaluation focuses on six representative contact-rich tasks. These tasks cover planning, insertion, wiping, and soft-object handling, but they do not yet exhaust the diversity of household manipulation or deformable-object interactions.

Sensor and embodiment transfer

TouchWorld is tied to the sensing layout used in our robot platform. Transferring to a different tactile sensor or hand morphology still requires calibration, normalization, and likely a small amount of adaptation data.

Acknowledgements

We thank the Harbin Institute of Technology, Shenzhen and PHANES AI teams for their support with the robot platform, data collection, and experiments.