Prototype 2025: Real-Time Synth Controller with A.R. on Quest 3

Three prototypes, built incrementally, each one pushing the Meta Quest 3 camera passthrough API one step further. The goal: control synthesiser sounds in real time through a mixed reality interface — nothing physical between hands and music, just spatial gesture mapped directly to sound.

The signal chain runs through Open Sound Control (OSC), a protocol designed for real-time network communication between music software and hardware. Spatial values captured from the Quest's hand-tracking feed are sent as OSC messages, converted to MIDI via a bridging tool, and received in Logic Pro — where they trigger and sculpt synthesiser responses live. What floats in space is, underneath, a tightly chained control path.

The three prototypes

Each prototype extended the interface in a meaningful way: from a spatial matrix controller, to a step sequencer, to a full dual-zone sequencer with real-time pitch shaping — all built in Unity and deployed to the Quest 3 headset. Every iteration revealed something the previous one couldn't have predicted.

Video 1: 4-Point Matrix Interface — ASM Hydrasynth

Exploring a four-point matrix interface to shape the sound of the ASM Hydrasynth in real time. The matrix captures spatial hand position and maps it across two axes simultaneously, each driving a different synthesis parameter on the Hydrasynth's highly modular architecture.

Captured in two passes: first laying down the melody, then sculpting the tone live in mixed reality. Separating the two recording passes made it possible to give each layer full attention — something a physical controller setup rarely allows.

Video 2: Step-Sequencing the Bassline

A new mixed reality layout, split in two: the left side controls a 16-step rhythmic sequence, the right side shapes the pitch of each step in real time. Separating rhythm and pitch into distinct spatial zones makes editing intuitive — each hand has a clear, unambiguous role.

The test piece is "Let the Music Play" by Shannon (1983) — one of the earliest records built entirely around a drum machine and synthesiser bassline, and frequently cited as a founding moment of freestyle music. A lesser-known detail: the iconic chorus features uncredited backing vocals by Jimi Tunnell, adding a raw, soulful edge the producers wisely kept in. It made it back into this montage too.

More than a dance hit, it was the sound of the early eighties urban landscape — echoing from cassette decks and boomboxes before anyone had a name for what it was.

Video 3: Step Sequencing — Sequential Prophet 6

Where spatial design meets synthesiser obsession. Every sound in this test comes from the Sequential Prophet 6 — a six-voice polyphonic synthesiser that traces its lineage directly back to the original Prophet-5 of 1978.

The VR environment is built on a sketch I first made in 2018. Scroll back to MIDI Sequencing in Virtual Reality to see where it started — the idea of step-sequencing inside a virtual space has been running as a thread through several prototypes since then.

This project connects to a longer line of experiments in spatial instrument design. Earlier iterations are documented in VR Without Headset, Bassline Synth Modulation in Virtual Reality, and MIDI Sequencing in Virtual Reality.