Tactile interaction through virtual finger extensions in spatial computing environments

Team:

Mathias Hradecsni
Andreas Muxel

This research project explores how haptic feedback and body extension can enhance interaction in mixed reality environments. Using the Apple Vision Pro and the Hapticlabs Toolkit, Phantom Hands allows users to control extended virtual finger shapes that interact with both digital content and the physical world. Each virtual finger can take on simple geometric shapes—such as cones or cubes—and collisions with virtual or real-world objects are translated into vibrotactile feedback on the user’s fingertips.

The aim is to investigate how tactile feedback affects perception, embodiment, and interaction in spatial computing, and how new forms of non-visual communication can be designed for immersive systems.

Interaction Experiments

Maze

A narrow 3D labyrinth floats in space. Users must guide their virtual finger through without touching the walls. If contact occurs, a sharp vibration is triggered, accompanied by a visual flash—producing an immediate “error” sensation through touch.

Frame

A square frame hovers in mid-air. As the virtual finger approaches the center of the frame, vibrations intensify, enabling users to sense spatial depth and distance without relying on vision. This interaction demonstrates how tactile gradients can make abstract spatial data feel tangible.

Strings

An array of floating lines can be plucked or brushed with the virtual finger extensions. When touched, each string emits a short vibration and visual feedback, evoking the sensation of interacting with physical tensioned materials. The experience bridges the gap between touch and gesture.

Phantom Hands is built with visionOS (SwiftUI + RealityKit), using native hand tracking and BLE communication with custom ERM motors attached to the fingers. Vibration patterns are designed and deployed using the Hapticlabs Studio. The project proposes playful and perceptual pathways toward more embodied, intuitive, and responsive AR interfaces.

Special thanks to the Hapticlabs team for their support and for providing the tools that made precise and customizable haptic experimentation possible.