Archive 2020: VR and Paper UI: Bridging Virtual and Physical Instruments

When building musical instruments in VR, one problem becomes immediately apparent: there is no tactile feedback. Pressing a virtual button, turning a virtual knob — the hand passes through everything. The visual cue says "contact", but the body feels nothing. This disconnect is one of the biggest missing pieces in making VR instruments feel playable.

This experiment attempts to bridge that gap between the virtual and the physical world. Instead of recreating an entire instrument inside VR, the setup keeps the real hardware — a Nord Lead synthesizer and a Focusrite audio interface — on the desk, and uses only the HTC Vive hand controllers as the interface between performer and sound.

Cardboard as interface

Simple cardboard cards are placed on the desk surface in front of the synthesizer. Each card acts as a physical trigger zone — a tangible surface the controller can strike or hover over. Hitting a card triggers a drum sound; moving a controller above another card controls the filter cutoff of the Nord Lead in real time. The cardboard provides what VR cannot: a surface that pushes back. The hand knows it has arrived somewhere.

Signal chain

The Vive controllers send positional and button data into Unity, which translates gestures into OSC messages. These are converted to MIDI and routed to the Nord Lead for real-time filter control, while drum triggers are sent to Logic Pro. The result is a hybrid instrument — physical keys for melody, cardboard surfaces for percussion and parameter control, all connected through the VR tracking system without ever putting on a headset.